Machine Learning Fundamentals · Course book
Chapter 4: The Scikit-learn Workflow
Scikit-learn is the most widely used library for classic machine learning in Python. Its great strength is consistency: almost every model, from a simple nearest-neighbour classifier to a neural network, is used in exactly the same way, so once you have learned the pattern you can try a new algorithm by changing one line. This chapter teaches that pattern and the habits around it. It covers how data is arranged into features and labels, how to split it, how to train, evaluate, save and reload a model, and how the main algorithms in the library compare. The second half moves from classic models to neural networks, explaining overfitting, backpropagation and gradient descent, and ends with convolutional networks and a handwritten digit recogniser, the classic first project in image recognition.
The scikit-learn pattern
Scikit-learn is installed with Python's package manager, together with NumPy for arrays and Matplotlib for plotting. It expects data in two pieces. X is a two-dimensional array of features, one row per example and one column per feature. y is a one-dimensional array of labels, one per row of X. Plotting one feature against another, or a feature against the label, before modelling is a quick way to spot outliers and to see whether a straight line or a curve is likely to fit.
Every model in the library is an estimator, an object with the same small set of methods. You create it, choosing its settings, then call fit(X, y) to train it, predict(X) to get predictions for new rows and score(X, y) to measure it, which returns accuracy for classifiers and a measure of fit called R2 for regressors.
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42)
model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)
print(model.score(X_test, y_test))
The iris dataset used here has 150 flowers of three species, each described by four measurements. With test_size=0.2, the split keeps 120 rows for training and 30 for testing. Setting random_state fixes the shuffle, so the same rows land in the same split every time the code runs, which makes results reproducible. The model never sees the 30 test rows during training, so its score on them estimates how it will do on new flowers.
Training can take minutes or hours on large data, so a trained model is usually saved to a file and loaded later. The joblib or pickle modules write the whole estimator, learned parameters included, to disk; loading it gives back an object that can call predict immediately. Only load model files you created yourself or received from a trusted source, because loading a pickle file can run code stored inside it.
Choosing a classic algorithm
Because every estimator shares one interface, it is common to try several on the same split and compare scores. The table summarises the classic algorithms the video covers and how each one thinks about the data.
| Algorithm | Scikit-learn class | Task | Key idea | Main setting |
|---|---|---|---|---|
| K-nearest neighbours | KNeighborsClassifier | Classification | Vote among the k closest training points | n_neighbors |
| Support vector machine | SVC | Classification | Widest margin between classes, with kernels for curves | C, kernel |
| Linear regression | LinearRegression | Regression | Best straight line by least squares | None needed |
| Logistic regression | LogisticRegression | Classification | Sigmoid of a weighted sum gives a probability | C (inverse regularisation) |
| K-means | KMeans | Clustering | Assign points to the nearest of k centres, then move the centres | n_clusters |
| Neural network | MLPClassifier | Classification | Layers of weighted sums and activations | hidden_layer_sizes |
Linear and logistic regression are easily confused because they share a name and a weighted sum. The difference shows clearly when both are fitted to a yes-or-no label. Suppose eight students studied for 1 to 8 hours, and the result, pass as 1 or fail as 0, was 0, 0, 0, 1, 0, 1, 1, 1. The least-squares line through these points has a slope of 7 / 42 = 0.167 and an intercept of 0.5 − 0.167 × 4.5 = −0.25. At 1 hour it predicts −0.08 and at 10 hours it predicts 1.42, neither of which makes sense as a probability. A logistic curve, by contrast, always stays between 0 and 1.
K-means is the odd one out because it is unsupervised: you call fit(X) with no labels, and the model groups the rows into the number of clusters you ask for. Its labels_ attribute then records which cluster each row joined. Since the cluster numbers are arbitrary, judging the result means looking at the groups or comparing them with known categories if any exist.
Neural networks and how they learn
A neural network stacks layers of neurons. Each neuron multiplies its inputs by weights, adds a bias and applies an activation function. A network for 28 × 28 pixel images has 784 inputs; with one hidden layer of 128 neurons and an output layer of 10, one per digit, it has 784 × 128 + 128 = 100,480 parameters in the first layer and 128 × 10 + 10 = 1,290 in the second, 101,770 in total, all of which are learned.
Learning follows the loop from earlier chapters. A cost function measures how wrong the network's outputs are. Gradient descent moves every weight a little in the direction that lowers the cost. The difficulty is computing that direction for weights buried deep in the network, and backpropagation solves it: it starts at the output, works out how much each output neuron contributed to the error, and passes that blame backwards layer by layer using the chain rule of calculus, so that every weight receives its own gradient in a single backward pass.
Networks have so many parameters that overfitting is a constant risk. The symptom is a training score that keeps improving while the validation score stalls or gets worse. Underfitting is the opposite: both scores are poor because the model is too small or trained too briefly. The table shows how to read the two scores together.
| Training score | Validation score | Diagnosis | What to try |
|---|---|---|---|
| Low | Low | Underfitting | A bigger model, better features, longer training |
| High | Much lower | Overfitting | More data, regularisation, a smaller model, stopping earlier |
| High | Close to training | A good fit | Confirm on the test set, then stop |
Convolutional networks and recognising digits
Feeding every pixel into a fully connected layer ignores the fact that nearby pixels belong together. A convolutional neural network (CNN) respects that structure. Its basic operation slides a small grid of weights, a filter or kernel, across the image; at each position it multiplies the overlapping values and adds them up, producing one number in an output map. A filter that responds to vertical edges lights up wherever brightness changes from left to right, wherever that happens in the image.
A filter's weights are learned just like any other weights, and because one small filter is reused at every position, a convolutional layer has very few parameters. Thirty-two 3 × 3 filters on a greyscale image need only 32 × 9 + 32 = 320 parameters. A typical CNN alternates convolution layers with pooling layers, which shrink each map by keeping, for example, the largest value in every 2 × 2 block, and ends with a fully connected layer that makes the final decision.
The handwritten digit recogniser brings everything together. Scikit-learn ships a small digits dataset of 1,797 images, each 8 × 8 pixels with values from 0 to 16, which flatten into 64 features and train in seconds. The larger MNIST dataset has 60,000 training images and 10,000 test images of 28 × 28 pixels. The workflow is the one from the start of the chapter: load and scale the pixels, split the data, fit a model, check the test accuracy and inspect the confusion matrix to see which digits, such as 4 and 9, are confused most often. Finally, you can draw a digit of your own, scale it the same way as the training images and ask the model to read it.
Practice tasks
- Rewrite the iris example to use
SVCinstead ofKNeighborsClassifier. Which lines change? - With 1,797 digit images and
test_size=0.25, about how many images go to training and testing? Check the rounding rule in the scikit-learn documentation. - Using the linear model from the study-hours example, what does it predict for 6 hours? Explain why this is not a valid probability for some inputs.
- Count the parameters in a network with 64 inputs, a hidden layer of 32 neurons and 10 outputs.
- Apply the same edge filter to a 5 × 5 image. What size is the output map?
- Your training accuracy is 99 percent and your validation accuracy is 82 percent. Diagnose the problem and suggest two fixes.
This course book is available to read online only at tech-academy.chrysolites.org.