Deep Learning with PyTorch · Course book
Chapter 3: Classifying Digits with Logistic Regression
Linear regression predicts numbers; most interesting problems ask for a category instead. This chapter takes the next step by classifying images of handwritten digits, the classic first problem in computer vision. Along the way it introduces the tools that every image project in PyTorch uses: datasets and transforms from torchvision, a proper split into training and validation sets, and data loaders that serve batches. The model itself is logistic regression, a single linear layer whose outputs are turned into probabilities by the softmax function and trained with cross-entropy loss. It is simple, but it introduces the complete pattern of training, evaluating, predicting and saving a classifier, which later chapters keep while swapping in more powerful models.
Working with image data
The MNIST dataset contains 70,000 greyscale images of handwritten digits from 0 to 9, each 28 × 28 pixels: a training set of 60,000 and a test set of 10,000. The torchvision library downloads it with one line and returns each sample as a pair: an image and its label. Raw images arrive as picture objects, so a transform is passed in to convert them. ToTensor turns each image into a tensor of shape (1, 28, 28), meaning one colour channel, 28 rows and 28 columns, with pixel values scaled from the range 0 to 255 down to 0 to 1.
Looking at the data first is always worthwhile. Plotting a few images with their labels confirms that the labels match, shows how varied handwriting is, and reveals ambiguous cases, a 4 that resembles a 9 or a 1 with a serif that looks like a 7, which no model will ever get perfectly right.
Training and validation sets
The 10,000 test images are kept aside to report final performance. To make decisions during development, such as how long to train or what learning rate to use, we need a separate validation set carved out of the training data. A random split of the 60,000 training images into 50,000 for training and 10,000 for validation does this. The split must be random, because datasets are sometimes ordered by label, and a validation set that contained only 8s and 9s would be useless.
Data loaders then serve each set in batches, for example of 128 images. The training loader shuffles the data every epoch so that each batch is a fresh mix; the validation loader does not need to. With 50,000 training images and batches of 128, an epoch has 390 full batches and one final batch of 80 images, 391 steps in total.
The model: a linear layer and softmax
Logistic regression for ten classes is a single nn.Linear layer with 784 inputs, one per pixel, and 10 outputs, one per digit. Its weight matrix has 784 × 10 = 7,840 entries and its bias 10, so the whole model has 7,850 parameters. Each image tensor of shape (1, 28, 28) must first be flattened into a vector of 784 values; inside the model's forward method, reshape(-1, 784) does this for a whole batch at once, the −1 letting PyTorch work out the batch size.
Writing the model as a class that extends nn.Module keeps everything together: the layers are created in __init__, the computation is described in forward, and helper methods for a training step and a validation step can live alongside them. The raw outputs of the linear layer, called logits, can be any real numbers. The softmax function turns them into probabilities: raise e to the power of each logit, then divide each result by their sum.
Accuracy, loss and the training loop
The natural measure of a classifier is accuracy, the fraction of images whose most probable class matches the label. If 103 of a batch of 128 are right, the batch accuracy is 103 / 128 = 0.805. Accuracy cannot be used to train the model, though, for two reasons. It changes in jumps, because a small change to the weights usually changes no predictions at all, so its gradient is zero almost everywhere. And it ignores confidence, treating a correct answer with 51 percent probability the same as one with 99 percent. Cross-entropy is smooth and rewards confidence in the right answer, so it serves as the loss, while accuracy is reported for people to read.
| Quantity | Used for | Smooth? | Typical start for 10 classes |
|---|---|---|---|
| Cross-entropy loss | Training, with gradients | Yes | About 2.3 |
| Accuracy | Reporting progress to people | No, it changes in steps | About 0.1 |
PyTorch's F.cross_entropy takes the raw logits and applies softmax internally, which is more numerically stable than doing the two separately. The training function loops over epochs. In each, it runs through the training batches, computing the loss, calling backward, stepping the optimiser and zeroing the gradients. After each epoch it runs through the validation batches without tracking gradients, computes the average validation loss and accuracy, prints them and appends them to a history list.
Plotting the history shows validation accuracy rising quickly over the first few epochs and then levelling off, typically somewhere between about 85 and 92 percent for this simple model, depending on the learning rate and how long it trains. Training longer brings little further gain, because the model's power, not the amount of training, has become the limit: a single linear layer can only draw straight boundaries between classes in the 784-dimensional pixel space, and handwriting is too varied for that.
The training function takes the number of epochs and the learning rate as arguments, which makes experiments easy. A common approach is to train for a few epochs at one learning rate, look at the history, then continue for a few more epochs, perhaps at a lower rate, since the model keeps its weights between calls. If the validation loss jumps around or rises, the learning rate is probably too high; if it creeps down very slowly, it may be too low. Recording each run's settings next to its final validation accuracy turns this trial and error into a proper log of experiments, which becomes essential once models take hours rather than seconds to train.
It also helps to measure the untrained model before the first epoch. With random weights, the validation loss should be close to 2.3 and the accuracy close to 10 percent, as calculated above. If the numbers are very different, something is wrong with the data or the loss, and it is far better to find that out before spending time on training.
Testing, predicting and saving
Once development is finished, the model is evaluated once on the 10,000 test images, which it has never seen and which played no part in any decision. If its test accuracy is close to the validation accuracy, the validation set did its job. Large gaps suggest that repeated tuning has fitted the validation set too closely.
Predicting a single image takes a little care: models expect a batch, so the single image of shape (1, 28, 28) is given an extra batch dimension with unsqueeze(0), making it (1, 1, 28, 28). The output is one row of ten logits, and the position of the largest is the predicted digit. Looking at the images the model gets wrong is instructive; many are genuinely ambiguous, and some reveal systematic weaknesses, such as digits written unusually small or off-centre.
Finally, torch.save(model.state_dict(), "mnist-logistic.pth") writes the learned weights and biases to a file. To use them later, you create a fresh instance of the same model class and call load_state_dict with the loaded dictionary. Checking that the reloaded model gives the same test accuracy confirms the save worked. Saving only the parameters, rather than the whole Python object, is the recommended practice because it does not depend on the exact code layout at the time of saving.
Practice tasks
- How many parameters does logistic regression have for colour images of 32 × 32 pixels with 3 channels and 10 classes?
- Compute the softmax probabilities for logits 2, 2 and 0, and the cross-entropy loss if the true class is the third.
- With 50,000 training images and a batch size of 256, how many batches make one epoch?
- Explain in your own words why accuracy is a poor training loss even though it is the number we care about.
- What shape must a single 28 × 28 greyscale image have before it is passed to the model, and why?
- Your validation accuracy is 91 percent and your test accuracy is 84 percent. Suggest what might have happened.
This course book is available to read online only at tech-academy.chrysolites.org.