Deep Learning with PyTorch · Course book
Chapter 2: Tensors, Gradients and Linear Regression
Every deep learning model, however large, is built from the same three ingredients: tensors that hold the data and parameters, gradients that say how to change the parameters, and a loop that applies those changes over and over. This chapter builds all three from first principles in PyTorch, using linear regression as the running example because it is small enough to compute by hand. You will create tensors and see how PyTorch tracks the operations performed on them, let automatic differentiation compute gradients and check them against calculus, then train a linear regression model from scratch with nothing but tensor operations. Finally, you will rebuild the same model using PyTorch's built-in layers, losses, datasets and optimisers, the tools every later chapter relies on.
Notebooks, tensors and NumPy
Deep learning work is usually done in Jupyter notebooks, documents that mix runnable code cells, their outputs and written explanations. Notebooks can run on your own computer or on free cloud platforms that also offer graphics cards, and they make it easy to experiment one step at a time, inspecting every intermediate result.
A PyTorch tensor is a number, vector, matrix or higher-dimensional array. Creating one from a Python number or list is a single call, and every element shares one data type. A tensor created from the list [[5., 6], [7, 8], [9, 10]] has shape (3, 2): three rows and two columns. Tensors support arithmetic element by element, matrix multiplication with the @ operator, and many mathematical functions.
Two details save a lot of confusion later. First, the data type matters: a tensor made from whole numbers holds integers, while neural network weights need floating-point numbers, so writing 5. rather than 5, or asking for torch.float32 explicitly, avoids type errors when the two meet. Second, PyTorch supports broadcasting: when two tensors of different shapes are combined, the smaller one is stretched to match where that makes sense. Adding a tensor of shape (2,) holding 1 and 10 to the (3, 2) tensor above adds 1 to every value in the first column and 10 to every value in the second, giving [[6, 16], [8, 18], [10, 20]]. Broadcasting is exactly what lets a single bias vector be added to the predictions for every sample in a batch.
PyTorch tensors are deliberately similar to NumPy arrays, the standard numerical arrays in Python, and convert in both directions with torch.from_numpy and .numpy(). The conversion shares memory where possible, so it is fast. Why not just use NumPy? Two reasons: PyTorch tensors can live on a graphics card, where large operations run many times faster, and PyTorch can automatically compute gradients of any calculation made with tensors.
Gradients with autograd
A gradient is the rate at which an output changes as one input changes, the derivative from calculus. When a tensor is created with requires_grad=True, PyTorch records every operation that uses it. Calling .backward() on a final result works back through that record and stores, in each tracked tensor's .grad attribute, the derivative of the result with respect to it. This is autograd, automatic differentiation.
y.backward(), w.grad holds 2 and b.grad holds 1, exactly as expected. Because x was created without gradient tracking, x.grad is empty. If y had been w2 × x + b instead, the derivative with respect to w would be 2wx = 12.One behaviour surprises everyone once: gradients accumulate. Calling backward again adds the new gradients to those already stored, rather than replacing them. That is useful in some advanced situations, but in an ordinary training loop the gradients must be reset to zero after each update, or every step would use a growing sum of all previous gradients.
Linear regression from scratch
Linear regression predicts each output as a weighted sum of the inputs plus a bias. With inputs arranged as a matrix X, one row per sample, weights W and biases b, the predictions for every sample at once are X @ WT + b. The model starts with random weights; training adjusts them until the predictions match the targets.
The quality of the predictions is measured by the mean squared error (MSE): subtract target from prediction for every sample, square the differences, and average them. The gradient of the MSE with respect to each weight has a simple form: two divided by the number of samples, times the sum over samples of the error multiplied by the matching input. Autograd computes this automatically, but working it once by hand makes it concrete.
Repeating this is gradient descent. In PyTorch, each step computes the predictions, computes the loss, calls loss.backward(), then, inside a torch.no_grad() block so the update itself is not tracked, subtracts the learning rate times each gradient and resets the gradients to zero. An epoch is one pass over the training data. On the three-sample example, the loss falls quickly at first and then ever more slowly as the weights approach the exact answer, weights 1 and 2 and a bias of 1.
After 200 epochs the weights are about 1.06 and 2.04 and the bias about 0.79, with a loss near 0.002. They are still creeping towards the exact values, which shows a typical pattern: most of the improvement comes early, and the last small gains take many more steps. A larger learning rate would converge faster, up to the point where steps overshoot and the loss starts to grow.
The same model with PyTorch built-ins
Writing everything by hand is the best way to understand it, but PyTorch provides tested building blocks that make the same model shorter and less error-prone.
| From scratch | Built-in replacement | What it adds |
|---|---|---|
| Weights and bias tensors with gradients | nn.Linear(2, 1) | Creates and initialises the parameters for you |
| Hand-written MSE | F.mse_loss | A standard, efficient loss function |
| Manual subtraction of gradients | torch.optim.SGD | Updates every parameter with step() and clears with zero_grad() |
| Indexing input and target tensors | TensorDataset | Pairs inputs with targets as one dataset |
| Looping over all data at once | DataLoader | Serves shuffled mini-batches |
With these, a training function takes the number of epochs, the model, the loss function, the optimiser and the data loader. For each epoch it loops over the batches: compute predictions, compute the loss, call backward, call step, call zero_grad. Printing the loss every ten epochs confirms progress.
model = nn.Linear(2, 1)
opt = torch.optim.SGD(model.parameters(), lr=1e-2)
loader = DataLoader(TensorDataset(inputs, targets), batch_size=5, shuffle=True)
for epoch in range(100):
for xb, yb in loader:
loss = F.mse_loss(model(xb), yb)
loss.backward()
opt.step()
opt.zero_grad()
Using mini-batches rather than the whole dataset at each step is called stochastic gradient descent. Each batch gives a noisy estimate of the true gradient, but the steps are much cheaper, and the noise can even help the model avoid poor solutions. Shuffling the data each epoch ensures every batch is a fair mixture. With 15 samples and a batch size of 5, each epoch makes 3 updates instead of 1.
Once trained, the model is used by simply calling it on new inputs: model(new_inputs) returns the predictions. Comparing predictions with targets for the training samples is a quick sanity check, and printing model.weight and model.bias shows what was learned. Because those parameters were created by nn.Linear with gradient tracking switched on, predictions made purely for inspection are best wrapped in torch.no_grad(), which saves memory and makes it clear that no training is happening.
The same pattern, a model, a loss, an optimiser and a loop over batches, carries through every chapter that follows. Only the model and the loss will change as the problems get harder.
Practice tasks
- With x = 4, w = 2 and b = 3, compute y = w2x + b and the gradients autograd would report for w and b.
- Continue the worked example for one more step from weights 0.28 and 0.30 and bias 0.14, and check that the loss falls to about 22.64.
- Explain what would go wrong if you forgot to zero the gradients after each update.
- A dataset has 1,000 samples and a batch size of 64. How many updates happen per epoch, counting a final partial batch?
- What shape is the weight tensor of
nn.Linear(3, 2), and how many parameters does the layer have in total? - Rewrite the from-scratch training step using
nn.Linear,F.mse_lossandtorch.optim.SGD.
This course book is available to read online only at tech-academy.chrysolites.org.