Deep Learning with PyTorch · Course book
Chapter 4: Feedforward Networks on GPUs
Logistic regression hit a ceiling because a single linear layer can only draw straight boundaries. This chapter breaks through that ceiling by adding hidden layers with non-linear activation functions, turning the model into a true neural network. It explains why non-linearity is essential, shows by hand how one hidden layer solves a problem no linear model can, and introduces the universal approximation theorem, which says how far the idea can go. The second half deals with speed. Neural networks are dominated by large matrix multiplications, which graphics processing units perform far faster than ordinary processors, so you will see how to get access to a free cloud GPU and how to move models and data onto it with a few small helper functions.
Hidden layers and activation functions
A feedforward neural network passes its input through a sequence of layers, each feeding only the next. For digit images, a simple design takes the 784 pixel values, passes them through a linear layer to a hidden layer of, say, 32 values, applies an activation function, and passes the result through a second linear layer to 10 outputs, one per digit. The first linear layer has 784 × 32 + 32 = 25,120 parameters and the second has 32 × 10 + 10 = 330, for a total of 25,450.
The activation function is the crucial piece. Without it, the two linear layers would collapse into one: multiplying by one matrix and then by another is the same as multiplying by their product, so the network could still only draw straight boundaries. The most common choice is the rectified linear unit, ReLU, which outputs its input when it is positive and zero otherwise. It is cheap to compute, and its slope is exactly 1 for every positive input, so gradients pass through it without shrinking.
In PyTorch, the network is a class extending nn.Module. Its __init__ creates the two linear layers, and its forward method flattens the batch of images, applies the first layer, applies F.relu and applies the second layer. The training and validation steps, the cross-entropy loss and the accuracy function are exactly the same as for logistic regression, which shows the value of organising code well: only the model changed.
Choosing the size of the hidden layer is a trade-off. A very small layer, say 4 units, squeezes all the information about an image through 4 numbers, too narrow to separate ten digits well. A very large layer, say 4,096 units, has more than three million parameters in its first layer alone, trains more slowly and can memorise the training images. Sizes between about 32 and 512 are common starting points for a problem of this scale, and powers of two are traditional mainly for convenience. Adding more hidden layers, each with its own activation, lets the network build features in stages, with early layers detecting simple strokes and later layers combining them.
PyTorch also chooses sensible starting weights for each linear layer automatically, small random values scaled according to the number of inputs. Starting every weight at zero would be a mistake: every hidden unit would then compute the same thing and receive the same gradient, so they would stay identical for ever. Randomness breaks this symmetry and lets each unit learn something different.
The universal approximation theorem
How powerful can a network with hidden layers be? The universal approximation theorem gives a striking answer: a feedforward network with a single hidden layer and a suitable non-linear activation can approximate any continuous function on a bounded range as closely as you like, provided the hidden layer is wide enough. The idea is easy to see with ReLU. Each ReLU unit contributes a "hinge", a function that is flat and then rises in a straight line from a chosen point. Adding and subtracting hinges with different starting points and slopes builds up any shape made of straight pieces, and enough small pieces can follow any smooth curve.
Check the bump at a few points. At x = 0.5 only the first hinge is active, giving 0.5. At x = 1.5 the first two are active: 1.5 − 2 × 0.5 = 0.5. At x = 3 all three are active: 3 − 2 × 2 + 1 = 0. The theorem guarantees that a good approximation exists; it does not promise that training will find it, or say how many units are needed. In practice, deeper networks with several narrower layers usually learn more efficiently than one enormously wide layer, which is one reason deep learning is deep.
Why GPUs, and where to find one
Training consists mostly of multiplying large matrices: a batch of 128 images times a 784 × 32 weight matrix is 128 × 784 × 32 = 3,211,264 multiply-and-add operations for one layer of one step. A graphics processing unit has thousands of simple cores that perform such operations in parallel, so it can be tens of times faster than a processor for this work. That difference decides whether an experiment takes minutes or hours.
| Option | Cost | Typical use | Watch out for |
|---|---|---|---|
| Free notebook platforms with GPUs | Free, with weekly or session limits | Learning and small projects | Sessions end after a time limit; save your work often |
| Your own computer with a supported GPU | Cost of the hardware | Regular, long experiments | Matching driver and library versions |
| Rented cloud machines | Paid by the hour | Large models and datasets | Remember to stop machines when finished |
Free notebook platforms let you switch on a GPU in the notebook settings. Inside PyTorch, torch.cuda.is_available() reports whether one can be used.
A GPU does not speed up everything. Loading images from disk, decoding them and applying transforms happen on the processor, and for a very small model the time spent copying each batch to the card can outweigh the time saved on arithmetic. The gains grow with the size of the model and of each batch, which is why larger batches are often used on GPUs, limited by the memory on the card.
Moving models and data to the GPU
Every tensor lives on a particular device, and an operation can only combine tensors on the same device. The model's parameters and each batch of data must therefore be moved to the GPU before they meet. Three small helpers keep this tidy. A get_default_device function returns the GPU if one is available and the processor otherwise, so the same notebook runs anywhere. A to_device function moves a tensor, or a list or tuple of tensors, to a chosen device. And a DeviceDataLoader class wraps an ordinary data loader so that every batch it yields is moved to the device automatically.
class DeviceDataLoader():
def __init__(self, dl, device):
self.dl = dl
self.device = device
def __iter__(self):
for batch in self.dl:
yield to_device(batch, self.device)
def __len__(self):
return len(self.dl)
The model is moved once, with to_device(model, device), which moves all its parameters. From then on, the training loop is unchanged. Moving batches one at a time, rather than the whole dataset at once, matters because GPU memory is limited: the 50,000 training images take about 157 megabytes as 32-bit floats, which fits easily, but real image datasets are often far larger than the memory on the card.
With the deeper network on a GPU, training for a few epochs takes seconds, and validation accuracy climbs well past the logistic regression ceiling, typically to around 96 to 97 percent on handwritten digits. Experiments that follow naturally include widening the hidden layer, adding a second hidden layer and adjusting the learning rate, each compared on the validation set. Keeping a record of each run's settings and results makes it easy to see what helped.
Check your understanding
- Count the parameters of a network with 784 inputs, hidden layers of 128 and 64 units, and 10 outputs.
- Explain why two linear layers with no activation between them are no more powerful than one.
- Using the exclusive-or network from the worked example, compute the output for the input (2, 0). Is it still a sensible answer?
- Evaluate the triangular bump ReLU(x) − 2 ReLU(x − 1) + ReLU(x − 2) at x = 0.25, 1 and 1.75.
- How many multiply-and-add operations does a batch of 64 images need in a 784 × 128 layer?
- Describe what the
DeviceDataLoaderclass does and why the whole dataset is not moved to the GPU at once.
This course book is available to read online only at tech-academy.chrysolites.org.