Chrysolites Tech Academy

Deep Learning with PyTorch · Course book

Chapter 6: Training a State-of-the-Art Image Classifier

Pairs with Lesson 6: ResNets, Regularisation and Data Augmentation

The convolutional network in the previous chapter reached a respectable accuracy and then started to overfit. Closing the gap to state-of-the-art results does not need a radically new idea, but a set of well-understood improvements applied together: better-prepared data, an architecture that trains reliably when it is deep, and a training procedure that makes the most of every epoch. This chapter explains each improvement, works the key calculations by hand, and shows how combining them lets a residual network reach over 90 percent accuracy on CIFAR-10 within minutes on a single GPU, far beyond the plain network, while training for fewer epochs.

Better data: normalisation and augmentation

Normalisation rescales each colour channel so that, across the training set, it has a mean of 0 and a standard deviation of 1. The mean and standard deviation are computed once per channel from the training images and then applied to every image, training, validation and test alike. For a red value of 0.8 in a channel whose mean is 0.5 and standard deviation 0.25, the normalised value is (0.8 − 0.5) / 0.25 = 1.2. Normalised inputs keep the early layers' activations in a sensible range and stop one channel from dominating the gradients simply because its values are larger.

Data augmentation applies random transformations to training images each time they are loaded, so that the network rarely sees exactly the same picture twice. Two transformations work especially well for small photos. A random crop pads the 32 × 32 image by 4 pixels on every side, to 40 × 40, using reflected pixels rather than black borders, and then cuts out a random 32 × 32 window, which shifts the content by up to 4 pixels in each direction. A random horizontal flip mirrors the image left to right half the time. A car facing left is still a car, so the label stays correct. Augmentation is applied only to training data; validation and test images are left unchanged so that the scores measure real performance.

Augmentations must respect the meaning of the data. Vertical flips would turn ships and horses upside down, a view the model will never meet in real photos, so they are usually left out for this dataset, while they might be perfectly sensible for satellite or microscope images that have no natural "up". Strong colour changes, rotations and zooms can help larger datasets but may distort tiny 32 × 32 images too much. Because augmented images differ every epoch, the training loss of an augmented model stays higher than that of an unaugmented one; that is expected, and the validation score is the one that shows whether augmentation is helping.

Residual connections and batch normalisation

Stacking more layers should, in principle, never make a network worse, because the extra layers could just learn to pass their input through unchanged. In practice very deep plain networks are hard to train and often do worse than shallower ones. Residual networks, or ResNets, solve this with a simple change. A residual block adds its input to its output: instead of computing y = F(x), it computes y = F(x) + x. The layers inside the block only need to learn the change to make, the residual, and if no change helps they can learn F(x) close to zero and pass x straight through.

x conv, batch norm,ReLU conv, batch norm,ReLU + F(x) + x shortcut: x is carried around unchanged the layers learn F(x), only the change to make
Figure 6.1 — A residual block. The shortcut lets information and gradients flow around the layers, which is what makes very deep networks trainable.

The shortcut also helps gradients. By the chain rule, the derivative of F(x) + x with respect to x is F′(x) + 1. Even if the layers' own derivative F′(x) is tiny, the gradient flowing back through the block is at least about 1, so it does not vanish as it passes through many blocks.

Batch normalisation normalises the activations inside the network. For each channel it computes the mean and variance over the current batch, rescales the values to mean 0 and variance 1, then applies a learned scale γ and shift β. For four batch values 2, 4, 6 and 8, the mean is 5 and the variance is (9 + 1 + 1 + 9) / 4 = 5, so the standard deviation is 2.236 and the normalised values are −1.342, −0.447, 0.447 and 1.342. With γ = 2 and β = 1 the outputs become −1.683, 0.106, 1.894 and 3.683. Keeping activations in a steady range lets the network train with higher learning rates. During evaluation, running averages collected during training replace the batch statistics, which is one reason to call model.eval() before testing.

Better training: one-cycle schedules, weight decay and clipping

A fixed learning rate is a compromise: high enough to make fast progress early, yet low enough to settle precisely at the end. A learning rate schedule removes the compromise by changing the rate during training. The one-cycle policy starts low, rises to a maximum over roughly the first 30 percent of steps, then falls smoothly to almost zero. The warm-up avoids wild early steps while the weights are random; the high middle phase covers ground quickly; the long decline lets the model settle into a good solution.

peak 0.01 at 30% 0.010.0050 0%30%50%100% progress through traininglearning rate
Figure 6.2 — A one-cycle schedule with a maximum of 0.01, a starting rate of 0.01 / 25 = 0.0004 and cosine-shaped rise and fall. Halfway through, the rate is 0.005.

Weight decay is a regularisation technique that adds a small penalty on large weights, nudging every weight a little towards zero at each step. With the plain update rule w ← w − lr × (gradient + λw), a weight of 0.5 with a gradient of 0.2, a learning rate of 0.01 and λ = 0.0001 moves to 0.5 − 0.01 × (0.2 + 0.00005) = 0.4979995, slightly further than the gradient alone would take it. Over thousands of steps this keeps the weights modest, which tends to make the network generalise better.

Gradient clipping limits how large any gradient value can be, for example to the range −0.1 to 0.1, before the update is applied. An occasional unusual batch can produce a huge gradient that would throw the weights far off course; clipping turns that into a modest step. A gradient of 3.5 becomes 0.1, while one of 0.04 is unchanged. Together with a schedule and weight decay, it makes training at high learning rates safe.

Putting it together

A compact residual network for CIFAR-10 stacks a few convolutional blocks, each with convolution, batch normalisation and ReLU, inserts two residual blocks and some max pooling, and ends with pooling to a single value per channel and a linear layer to 10 outputs. The training function grows a few lines longer: it creates a one-cycle scheduler and steps it after every batch, records the learning rate for plotting, clips gradients after backward and before step, and passes the weight decay to the Adam optimiser.

Adam is a popular optimiser that keeps two running averages for each weight: one of recent gradients, which acts like momentum, and one of recent squared gradients, which it uses to scale each weight's step so that weights with consistently large gradients take smaller steps and rarely updated weights take larger ones. It is forgiving of the choice of learning rate, which is why it pairs well with an aggressive schedule. The main settings left to choose are the maximum learning rate, the number of epochs, the weight decay and the clipping limit. Sensible first values for a problem of this size are a maximum rate of 0.01, about eight epochs, a weight decay of 0.0001 and a clipping limit of 0.1, which can then be adjusted one at a time while watching the validation accuracy.

TechniqueWhat it improvesWhere it goes
Channel normalisationInput scale and stabilityData transforms, all sets
Random crop and flipVariety, less overfittingData transforms, training set only
Residual connectionsTrainability of deep networksModel architecture
Batch normalisationStable activations, higher learning ratesAfter convolutions in the model
One-cycle scheduleFaster and better convergenceTraining loop, every batch
Weight decayGeneralisationOptimiser setting
Gradient clippingSafety at high learning ratesTraining loop, between backward and step

Trained this way for about eight epochs, a network like this can pass 90 percent validation accuracy on CIFAR-10 in a few minutes on a GPU. Plotting the validation accuracy, the training and validation losses and the learning rate per batch shows how the pieces interact: the losses fall fastest during the high-rate middle of the cycle and settle as the rate declines. The final steps are the usual ones: evaluate once on the test images, try the model on individual pictures and save its weights.

Check your understanding

  1. Normalise a green value of 0.3 in a channel with mean 0.45 and standard deviation 0.25.
  2. After padding a 32 × 32 image by 4 pixels per side and taking a random 32 × 32 crop, by how many pixels can the content shift, at most, in each direction?
  3. Apply batch normalisation to the values 1, 3, 5 and 7, then apply γ = 0.5 and β = 2.
  4. Explain why the gradient through a residual block cannot vanish as easily as through a plain stack of layers.
  5. In the one-cycle schedule shown, roughly what is the learning rate at 15 percent and at 80 percent of training?
  6. Clip the gradients 0.25, −0.07 and −1.3 to the range −0.1 to 0.1.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.