Chrysolites Tech Academy

Deep Learning with TensorFlow and Keras · Course book

Chapter 4: Classifying Text with Neural Networks

Pairs with Lesson 4: Text Classification with TensorFlow

Text is everywhere: product reviews, support tickets, emails, social posts. Sorting it automatically, into positive and negative, urgent and routine, or one topic among many, is called text classification, and it is one of the most useful things a small neural network can do. This chapter builds up to it in stages. It first sets out the anatomy of a dataset and the ways a model's performance is measured, with each loss function worked by hand. It then builds a feedforward network in TensorFlow on ordinary tabular data, covering the splitting, scaling and balancing steps that real data needs. Finally it introduces recurrent networks and uses them, together with learned and pre-trained word embeddings, to classify written reviews.

The anatomy of a dataset

A supervised dataset is a table. Every row is a sample. The columns used as inputs are the features, collected into a matrix usually called X, with one row per sample and one column per feature. The column to be predicted is the label or target, collected into a vector called y. Features may be qualitative, meaning categories such as country or grape variety, which are converted to numbers with one-hot encoding, or quantitative, meaning measured numbers such as price.

The output determines the kind of task. Predicting one of two categories is binary classification; one of several is multi-class classification; a continuous number is regression. A reviews dataset that records each review's text, the reviewer's score out of 100, the price and the country of origin supports all three: predict the country from the text, predict the exact score, or predict whether the score is above a chosen cut-off.

The data is divided into three parts before training. The training set is used to fit the weights, the validation set to compare models and decide when to stop, and the test set, used once at the end, to report final performance. A common split is 60, 20 and 20 percent: from 10,000 reviews, 6,000 for training and 2,000 each for validation and testing. Shuffling first ensures each part is a fair sample of the whole.

Assessing performance

A loss function measures how far predictions are from the truth, and training tries to make it small. Different tasks need different losses.

LossFormula for one sampleUsed forBehaviour
L1, mean absolute error|y − ŷ|RegressionGrows steadily with the error; tolerant of outliers
L2, mean squared error(y − ŷ)2RegressionSmall errors count little, large errors count a lot
Binary cross-entropy−[y ln p + (1 − y) ln(1 − p)]Binary classificationPunishes confident wrong answers very heavily
Categorical cross-entropy−ln of the probability of the true classMulti-class classificationThe same idea extended to many classes

Take three regression predictions with errors of 3, 0.5 and −1. The mean absolute error is (3 + 0.5 + 1) / 3 = 1.5. The mean squared error is (9 + 0.25 + 1) / 3 = 3.417; squaring has made the single large error dominate. For classification, suppose the true label is 1. A predicted probability of 0.9 costs −ln 0.9 = 0.105, a hesitant 0.6 costs 0.511, and a confident wrong 0.2 costs 1.609. The curve below shows how sharply the cost rises as the probability given to the true class falls.

0.2 costs 1.6090.6 costs 0.5110.9: 0.105 0123 00.51 predicted probability of the true classloss
Figure 4.1 — Binary cross-entropy for a sample whose true label is 1. Being confidently wrong costs far more than being unsure.

Loss guides training, but people usually report accuracy, the share of correct predictions, because it is easier to interpret. Both should be tracked on the validation set as well as the training set: a training loss that keeps falling while the validation loss rises means the model is overfitting.

A feedforward network on tabular data

Before tackling text, the workflow is easiest to see on a table of numbers, such as a few measurements per sample and a yes-or-no label. Three preparation steps come first. Splitting into training, validation and test sets, as above. Scaling each feature to mean 0 and standard deviation 1, using statistics computed on the training set only. And balancing: if only 30 percent of samples are positive, a model can score 70 percent by always answering no, so the rare class is oversampled, its rows randomly repeated in the training set until both classes are equally common. Balancing is applied only to the training set; the validation and test sets keep their real proportions so that the scores reflect reality.

model = tf.keras.Sequential([
    tf.keras.layers.Dense(16, activation="relu"),
    tf.keras.layers.Dense(16, activation="relu"),
    tf.keras.layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss=tf.keras.losses.BinaryCrossentropy(),
              metrics=["accuracy"])
model.fit(X_train, y_train, batch_size=16, epochs=20,
          validation_data=(X_valid, y_valid))

The final layer has one sigmoid neuron, which outputs the probability of the positive class, and the loss is binary cross-entropy to match. Calling model.evaluate on the test set at the very end gives the loss and accuracy to report. If the training accuracy is far above the validation accuracy, adding a dropout layer, fewer neurons or fewer epochs usually helps.

The object returned by fit keeps a history of the loss and accuracy for every epoch, on both the training and validation sets. Plotting those four curves is the single most useful diagnostic in deep learning. Curves that are still falling at the last epoch suggest training for longer. A validation loss that bottoms out and then climbs marks the point to stop, and Keras can do this automatically with an early stopping callback that watches the validation loss and restores the best weights it has seen. Trying a few combinations of layer size, dropout rate, learning rate and batch size, and keeping the one with the lowest validation loss, is a simple and effective way to tune a small network.

Recurrent neural networks

A feedforward network sees its whole input at once and has no notion of order. Text, speech and other sequences are different: the meaning of a word depends on what came before. A recurrent neural network (RNN) reads a sequence one item at a time and keeps a hidden state, a small vector that summarises everything read so far. At each step the new state is computed from the current input and the previous state, using the same weights at every step.

Worked example. A one-number RNN uses ht = tanh(0.5 xt + 0.8 ht−1), starting from h0 = 0. Feed it the sequence 1, 0, −1. Step 1: tanh(0.5 × 1 + 0) = tanh(0.5) = 0.462. Step 2: tanh(0 + 0.8 × 0.462) = tanh(0.370) = 0.354. Step 3: tanh(−0.5 + 0.8 × 0.354) = tanh(−0.217) = −0.214. The state after step 2 still carries the first input even though the second input was zero: that carried-over memory is what makes the network recurrent.

Training an RNN unrolls it through every step and backpropagates through the whole chain. Over long sequences the gradient is multiplied by similar factors many times and tends to shrink towards zero, the vanishing gradient problem, so early words stop influencing learning. The LSTM (long short-term memory) and GRU (gated recurrent unit) cells solve this with gates that decide what to keep, what to forget and what to output, letting information travel across many steps largely unchanged. In practice you use these cells rather than the simple version.

Classifying text with embeddings and LSTMs

To classify reviews, the text must become numbers. One approach uses a pre-trained embedding: a ready-made model, downloaded from TensorFlow's model hub, that maps a whole sentence to a fixed-length vector, trained beforehand on a huge amount of text. It is used as the first layer of the network, followed by a couple of dense layers and a sigmoid output. Because the embedding already captures a lot about word meaning, this gives strong results with little training.

The second approach learns everything from the reviews themselves. A text vectorisation layer builds a vocabulary from the training text, for example the 2,000 most common words, and turns each review into a list of integer codes, with 0 reserved for padding and 1 for unknown words. If the vocabulary assigns "rich" code 41, "and" code 3 and "smooth" code 87, the phrase "rich and smooth" becomes [41, 3, 87]. An embedding layer then replaces each code with a learned vector, and an LSTM layer reads the vectors in order and passes its final state to a dense layer and the output neuron.

encoder = tf.keras.layers.TextVectorization(max_tokens=2000)
encoder.adapt(train_text)

model = tf.keras.Sequential([
    encoder,
    tf.keras.layers.Embedding(input_dim=len(encoder.get_vocabulary()),
                              output_dim=32, mask_zero=True),
    tf.keras.layers.LSTM(32),
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(1, activation="sigmoid"),
])

Setting mask_zero=True tells the later layers to ignore padding, so short reviews are not diluted by zeros. Feeding data through a tf.data pipeline, which batches, shuffles and prefetches the reviews, keeps training efficient. Comparing the two approaches on the same validation set is a good exercise: the pre-trained embedding usually wins with little data, while the learned LSTM can catch up when there is plenty.

Practice tasks

  1. Compute the mean absolute error and mean squared error for prediction errors of 2, −2 and 4. Which loss is more affected by the error of 4?
  2. Compute the binary cross-entropy for a true label of 0 and a predicted probability of 0.3.
  3. A dataset of 15,000 rows is split 60/20/20. How many rows go to each set? If 20 percent of the training rows are positive, how many rows are added by oversampling to balance it?
  4. Run the worked RNN example on the sequence 0, 1, 1 and give the three hidden states.
  5. Explain why oversampling is applied only to the training set.
  6. Using the example vocabulary, encode "smooth and rich" and explain why the order of codes matters to an LSTM but not to an average-pooling model.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.