Deep Learning with PyTorch · Course book
Chapter 5: Convolutional Networks for Colour Images
Fully connected networks treat an image as a long list of unrelated numbers. They have no idea that neighbouring pixels belong together, or that a cat in the top-left corner is the same cat when it appears bottom-right. Convolutional neural networks build those facts into their structure, and that is why they dominate computer vision. This chapter applies them to CIFAR-10, a dataset of small colour photographs. It covers how colour images are represented, how convolutions, kernels and feature maps work, worked through by hand, and how a typical network shrinks the image while growing the number of channels. It finishes by reading training curves for signs of underfitting and overfitting, and by surveying the ways to improve a model, which the next chapter puts into practice.
Colour images and the CIFAR-10 dataset
CIFAR-10 contains 60,000 colour images of 32 × 32 pixels in 10 classes: aeroplane, car, bird, cat, deer, dog, frog, horse, ship and lorry. There are 50,000 training images and 10,000 test images, 6,000 per class in total. A common practice is to hold back 5,000 of the training images as a validation set, leaving 45,000 for training.
A colour image has three channels, red, green and blue, each a 32 × 32 grid of intensities, so as a PyTorch tensor it has shape (3, 32, 32), channels first. A batch of 128 such images has shape (128, 3, 32, 32). Plotting libraries expect channels last, so displaying an image needs the dimensions reordered to (32, 32, 3) with permute(1, 2, 0). When images are stored in the standard layout of one folder per class, torchvision's ImageFolder loads them and assigns labels from the folder names. Looking through a grid of training images quickly shows how hard the task is: the photos are tiny, objects appear at many angles, and backgrounds vary widely.
Convolutions, kernels and feature maps
A convolution slides a small grid of weights, the kernel or filter, across the image. At each position it multiplies every kernel weight by the pixel underneath and adds up the products, producing one number. Doing this at every position produces a new grid called a feature map, which is bright wherever the image locally matches the pattern in the kernel.
Three settings control the size of the output. The kernel size k, the padding p, extra zeros added around the border, and the stride s, how many pixels the kernel moves each step. For an n × n input, the output width is (n + 2p − k) / s + 1, rounded down. A 32-pixel input with k = 3, p = 1 and s = 1 stays at 32; with no padding it becomes 30; with p = 1 and s = 2 it becomes (32 + 2 − 3) / 2 + 1 = 16.5, rounded down to 16.
Convolutions have two great advantages over fully connected layers. Weight sharing: the same small kernel is used at every position, so a layer needs very few parameters and a pattern learned in one place is recognised everywhere. Locality: each output depends only on a small neighbourhood, matching the way visual features are built from nearby pixels. A layer usually learns many kernels at once, each producing its own feature map, and the number of kernels becomes the number of output channels.
A max-pooling layer then reduces each feature map by keeping the largest value in each 2 × 2 block, halving the height and width. Pooling cuts computation, makes the network tolerant of small shifts, and lets later layers see a larger part of the image.
The region of the original image that can influence one value in a feature map is that value's receptive field, and it grows layer by layer. One 3 × 3 convolution sees a 3 × 3 patch. A second 3 × 3 convolution on top of it sees 5 × 5 pixels of the original, because each of its nine inputs already summarises a 3 × 3 patch. A 2 × 2 pooling step then doubles the reach of everything after it. By the last block of a network like the one below, each value is influenced by most of the 32 × 32 image, which is what allows it to respond to whole objects rather than fragments.
This growth explains what trained networks are seen to learn when their kernels and feature maps are visualised. The first layer's kernels respond to simple features such as edges at particular angles and patches of particular colours. Middle layers respond to combinations of those, such as corners, stripes and textures like fur or feathers. The last layers respond to parts and arrangements that are specific to the classes, such as wheels, eyes or wings. Nobody programs these detectors: they emerge from gradient descent because they help reduce the loss.
A convolutional network for CIFAR-10
A typical design repeats a block of two convolutional layers with ReLU activations followed by max pooling. Each block halves the spatial size and the number of channels grows, so the network trades "where" information for "what" information as it goes deeper.
| Stage | Layers | Output shape | Parameters |
|---|---|---|---|
| Input | – | 3 × 32 × 32 | – |
| Block 1 | Conv 3→32, conv 32→64, pool | 64 × 16 × 16 | 896 + 18,496 = 19,392 |
| Block 2 | Conv 64→128, conv 128→128, pool | 128 × 8 × 8 | 73,856 + 147,584 = 221,440 |
| Block 3 | Conv 128→256, conv 256→256, pool | 256 × 4 × 4 | 295,168 + 590,080 = 885,248 |
| Classifier | Flatten to 4,096, linear to 1,024, 512 and 10 | 10 | 4,195,328 + 524,800 + 5,130 = 4,725,258 |
| Total | – | – | 5,851,338 |
Each convolution count follows the rule 3 × 3 × input channels × output channels + output channels; the first is 3 × 3 × 3 × 32 + 32 = 896. The total is revealing: the six convolutional layers together hold about 1.1 million parameters, while the first linear layer alone holds 4.2 million, 72 percent of the whole model. Convolutions are cheap in parameters; dense layers are not.
Building the model with nn.Sequential keeps it readable, and passing a single batch through it before training, then printing the output shape, catches shape errors early: the output should be (128, 10) for a batch of 128. Training uses the same loop, cross-entropy loss and GPU helpers as before, often with the Adam optimiser.
Underfitting, overfitting and improving the model
Plotting training and validation loss after every epoch is the best diagnostic. Three patterns are common. If both losses are high and falling slowly, the model is underfitting: it is too small, has trained too briefly, or the learning rate is poor. If the training loss keeps falling but the validation loss bottoms out and starts rising, the model is overfitting: it has begun memorising the training photos. If both fall together and level off close to each other, the model is about as good as its design allows.
A network of this kind, trained for about ten epochs, usually reaches validation accuracy in the region of 75 percent on CIFAR-10, and soon shows the overfitting pattern because it has almost six million parameters and only 45,000 training images. The main ways to improve are:
- More and better data: collect more images or create variations of existing ones with data augmentation.
- Normalisation: scale each channel to have mean 0 and standard deviation 1, and normalise the activations inside the network.
- Regularisation: weight decay, dropout and early stopping, which discourage memorising.
- Better architectures: residual connections, which make deeper networks train reliably.
- Better training schedules: changing the learning rate during training and clipping very large gradients.
Each of these is examined, and combined, in the next chapter. When comparing changes, change one thing at a time and judge it on the validation set, so that you know which change actually helped.
Practice tasks
- A 64 × 64 input passes through a convolution with k = 5, p = 2 and s = 1, then a 2 × 2 max pool. What is the final width?
- Compute the output of the plus-shaped kernel on the patch with rows (2, 2, 2), (2, 2, 2) and (2, 2, 2). Why is the result the same as on a blank patch scaled by 2?
- How many parameters does a convolutional layer with 3 × 3 kernels, 64 input channels and 32 output channels have?
- If the classifier's first linear layer took 256 × 2 × 2 inputs instead of 256 × 4 × 4, how many parameters would that layer have with 1,024 outputs?
- Describe the training and validation loss curves you would expect from an underfitting model and from an overfitting one.
- Explain weight sharing and why it makes convolutional layers so economical.
This course book is available to read online only at tech-academy.chrysolites.org.