Chrysolites Tech Academy

How Neural Networks Work · Course book

Chapter 1: Inside a Deep Neural Network

Pairs with Lesson 1: How Deep Neural Networks Work

Deep neural networks power image recognition, speech transcription, translation and much more, yet the core machinery is simple enough to work through with a calculator. This chapter builds that machinery piece by piece. It starts with a single artificial neuron, shows how layers of neurons combine simple pattern detectors into complex ones, and then follows a real gradient calculation through one training step so that backpropagation stops being a mystery. The second half explains the two architectures that made deep learning famous: convolutional networks, which find patterns in images wherever they appear, and recurrent networks with LSTM memory units, which handle sequences such as sentences. It closes with a sober look at what these networks can and cannot do, and why making machines that act in the physical world is harder still.

From one neuron to a network

An artificial neuron does three things. It multiplies each input by a weight, which says how much that input matters and in which direction; it adds the results together with a bias; and it passes the total through a squashing or activation function that keeps the output in a fixed range. A positive weight means "this input pushes me towards on", a negative weight means "this input pushes me towards off", and a weight near zero means "ignore this input".

Imagine a tiny camera with only four pixels, arranged in a 2 × 2 square, each pixel reading +1 for white and −1 for black. A neuron with weights +1 on the top two pixels and −1 on the bottom two responds strongly, with a sum of +4, to an image that is white on top and black below, responds with −4 to the reverse, and gives 0 to an all-white image. That neuron is a detector for one particular pattern. A second layer of neurons can then combine detectors: one that fires when either the horizontal-edge detector or the vertical-edge detector is on, another that fires only when both are. Each layer builds more abstract patterns out of the layer below, which is exactly what "deep" means: many layers, each learning a more complex combination.

1−1 −4−224 ReLU tanh sigmoid input z
Figure 1.1 — Three common activation functions. Tanh squashes to the range −1 to 1, the sigmoid to 0 to 1, and ReLU passes positive values through unchanged and sets negative values to zero.

The squashing function matters because without it, stacking layers achieves nothing: a weighted sum of weighted sums is just another weighted sum, so a hundred linear layers are no more powerful than one. The bend in the activation function is what lets each extra layer add new expressive power. Tanh and the sigmoid were the classic choices; the rectified linear unit, or ReLU, is now the default in hidden layers because it is cheap to compute and its slope does not fade to almost nothing for large inputs.

How a network learns: gradient descent and backpropagation

A network starts with random weights and is useless. Learning means adjusting the weights so that the outputs move towards the correct answers. The error, or loss, measures the gap; a common choice is the squared difference between output and target. For any weight we ask: if I nudged this weight slightly, how much would the error change? That rate of change is the gradient. Stepping each weight a little in the direction that lowers the error, again and again, is gradient descent.

Computing gradients for millions of weights one by one would be hopeless. Backpropagation uses the chain rule from calculus: the effect of a weight on the error equals the effect of the weight on the neuron's sum, times the effect of the sum on the neuron's output, times the effect of the output on the error. Each of those pieces is simple and local, and the results for later layers are reused for earlier ones, so a single backward pass delivers every gradient.

Worked example. One tanh neuron has inputs 1 and 0.5, weights 0.4 and −0.2, and bias 0.1. Its sum is 0.4 − 0.1 + 0.1 = 0.4 and its output is tanh(0.4) = 0.380. The target is 1, so the squared error is (0.380 − 1)2 = 0.385. Now chain the pieces backwards. The slope of the error with respect to the output is 2 × (0.380 − 1) = −1.240. The slope of tanh at this point is 1 − 0.3802 = 0.856. Their product, −1.061, is the slope of the error with respect to the sum. Each weight's gradient is that times its input: −1.061 for the first weight and −0.531 for the second. With a learning rate of 0.1, the weights become 0.506 and −0.147 and the bias 0.206. The new output is 0.564 and the error drops to 0.190; a further step brings it to 0.124.

Real networks repeat this over many examples and many layers, but nothing else is fundamentally new. Practical additions make it work better: processing examples in small batches to average out noise, techniques such as batch normalisation that keep the numbers flowing between layers in a healthy range, and careful choice of the learning rate so that steps are neither timid nor reckless.

Convolutional networks for images

A detector neuron like the four-pixel one only looks at fixed positions. In a real photo, a cat's ear might appear anywhere, so a convolutional neural network slides each small detector, called a filter, across the whole image and records how strongly it matches at every position. The result is a feature map, a new image that is bright wherever the pattern was found. A convolutional layer runs many filters at once, producing a stack of feature maps.

Two more operations complete the standard building block. A ReLU layer sets every negative value in the maps to zero, keeping only positive evidence. A pooling layer shrinks each map, most often by keeping the largest value in each 2 × 2 block. For the 4 × 4 map with rows (1, 3, 2, 0), (4, 6, 1, 1), (0, 2, 5, 7) and (1, 1, 3, 2), max pooling gives the 2 × 2 map (6, 2) over (2, 7). The map is a quarter of the size, but the strongest signals survive, and small shifts of the pattern no longer change the answer.

LayerWhat it doesWhy it helps
ConvolutionSlides learned filters across the inputFinds the same pattern anywhere in the image
ReLUReplaces negative values with zeroAdds non-linearity cheaply
Max poolingKeeps the largest value in each small blockShrinks the data and tolerates small shifts
Fully connectedEvery input connects to every outputTurns detected features into a final vote for each class

Stacking several rounds of convolution, ReLU and pooling lets early layers find edges and spots, middle layers combine them into textures and parts, and late layers recognise whole objects. The filters themselves are learned by backpropagation, exactly like any other weights. The same idea works on any data where neighbouring values are related, such as sound, where nearby moments in time belong together, or the grid of a board game. It fails where position carries no meaning, as in a spreadsheet whose columns could be shuffled without changing the data.

Recurrent networks and LSTM memory

Some data comes as a sequence, where order matters: words in a sentence, notes in a melody, readings from a sensor. A recurrent neural network (RNN) handles this by feeding part of its output back into itself at the next step, so its prediction for word five can depend on words one to four. Unrolled in time, it is a very deep network with shared weights, one copy per step.

Simple RNNs struggle to remember anything for long, because during backpropagation the gradient is multiplied over and over as it travels back through the steps and either shrinks to nothing or blows up. The long short-term memory unit, or LSTM, fixes this with an explicit memory and three gates, small sigmoid-based switches that output values between 0 and 1 and control the flow of information.

  • The forget gate decides how much of the existing memory to keep.
  • The input gate decides which new information to write into memory.
  • The output gate decides how much of the memory to reveal as the unit's output at this step.

Because the memory can pass from step to step mostly unchanged when the forget gate stays open, useful information, such as the subject of a sentence, can survive across many words. LSTMs were behind major advances in speech recognition and translation before attention-based transformer models largely replaced them for language.

What deep learning can and cannot do

Deep networks excel at finding patterns in large amounts of labelled data, and often match or beat people on narrow perception tasks. Their limits are just as important. They need far more examples than a person does. They can fail badly on inputs that differ from their training data, even in ways a human would not notice. Their internal reasoning is hard to inspect. And they learn correlations, not causes, so a model can latch onto an accidental clue, such as a watermark that happened to appear on most photos of one class.

Robotics shows the gap clearly. A robot must perceive, decide and act in a world that keeps changing, where mistakes have physical consequences and collecting millions of labelled examples is slow and expensive. Progress there needs networks that learn from fewer trials, adapt to new situations and combine learned perception with planning. Thinking in these terms keeps expectations realistic: deep learning is a remarkably general way to fit functions to data, not a general intelligence.

Check your understanding

  1. Using the four-pixel camera, design weights for a neuron that detects a white left column and a black right column. What sum does it give for that image and for its reverse?
  2. Explain why a network built only from weighted sums, with no activation function, cannot gain power by adding layers.
  3. Continue the worked example for one step from weights 0.566 and −0.117 and bias 0.266. What sum and output do you get?
  4. Apply 2 × 2 max pooling to the map with rows (2, 0, 1, 4), (1, 3, 0, 2), (5, 1, 1, 0) and (0, 2, 6, 3).
  5. Give one kind of data where convolutional networks work well and one where they do not, and explain the difference.
  6. Describe the job of each of the three LSTM gates in one sentence each.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.