Machine Learning Fundamentals · Course book
Chapter 5: Pipelines, Metrics and Meta Estimators
Knowing the algorithms is only half of practical machine learning. The other half is the engineering around them: transforming raw columns into something a model can use, chaining the steps so they cannot leak information, searching for good settings, and deciding what "good" actually means for the problem in front of you. Scikit-learn has a carefully designed set of tools for all of this. This chapter explains how pipelines and grid search fit together, which preprocessing tools to reach for and why, how to measure models with built-in and custom metrics, and how meta estimators wrap and extend ordinary models. The central message is that the score you optimise and the way you compute it deserve as much thought as the model itself.
Pipelines and grid search
A typical model needs several steps before the learning algorithm runs: fill missing values, scale numbers, encode categories. Doing each step by hand on the whole dataset invites a subtle mistake called data leakage. If you compute the mean and standard deviation for scaling from all the data and only then split it, the training rows have quietly seen information about the test rows, and the test score is too optimistic.
A pipeline solves this by chaining the steps into a single estimator. When you call fit, each transformer learns its parameters from the training data only and passes the transformed data to the next step; when you call predict, the same learned transformations are applied to the new rows. The pipeline behaves exactly like a model, so it can be scored, saved and searched.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import GridSearchCV
pipe = Pipeline([
("scale", StandardScaler()),
("model", KNeighborsRegressor()),
])
grid = GridSearchCV(
pipe,
param_grid={"model__n_neighbors": [1, 5, 10, 20]},
cv=5,
)
grid.fit(X, y)
print(grid.best_params_)
GridSearchCV tries every combination of settings in the grid and scores each one with cross-validation. With cv=5 the data is cut into five folds; the model is trained on four and scored on the fifth, five times over, so every row is used for scoring exactly once. The double underscore in model__n_neighbors tells the search which step of the pipeline the setting belongs to.
The cost adds up quickly. Four values of one setting with five folds means 4 × 5 = 20 model fits, plus one final refit of the best setting on all the data, 21 in total. Adding a second setting with three values makes it 4 × 3 × 5 + 1 = 61. The results are stored in cv_results_, which loads neatly into a table so you can see not only the best setting but how sensitive the score is to each choice.
Reading that table is a skill in its own right. Each row lists the settings tried, the mean score across the folds and the standard deviation of those fold scores. If two settings differ by less than one standard deviation, the difference is probably noise, and the simpler or faster option is the better choice. A score that changes smoothly as a setting grows suggests you have found a sensible range; a best value sitting at the very edge of the grid suggests the grid should be extended in that direction. When the grid becomes too large to search exhaustively, RandomizedSearchCV samples a fixed number of random combinations instead, which often finds a nearly-as-good setting in a fraction of the time, because only a few of the settings usually matter much.
Preprocessing tools
Preprocessing can change a model's performance more than switching algorithms. The most common transformers each address a specific problem.
| Transformer | What it does | When to use it |
|---|---|---|
| StandardScaler | Subtracts the mean and divides by the standard deviation | Distance-based and gradient-based models |
| MinMaxScaler | Maps each feature to the range 0 to 1 | When a bounded range is needed |
| QuantileTransformer | Replaces each value by its rank, spread evenly or into a bell shape | Features with outliers or very skewed distributions |
| PolynomialFeatures | Adds squares and products of features | Letting a linear model fit curves and interactions |
| OneHotEncoder | One 0-or-1 column per category | Text categories such as colour or country |
| SimpleImputer | Fills missing values with the mean, median or a constant | Data with gaps |
A quick calculation shows what standard scaling does. For the values 2, 4, 6 and 8 the mean is 5 and the squared deviations are 9, 1, 1 and 9, averaging 5, so the standard deviation is √5 = 2.236. The scaled values are −1.342, −0.447, 0.447 and 1.342. Outliers distort this: a single value of 1,000 would drag the mean and inflate the deviation so much that all the ordinary values crowd together near zero. The quantile transformer avoids the problem because it only cares about order, so an extreme value simply becomes the largest rank.
Polynomial features grow fast. Three features at degree 2 produce a constant term, the three originals, three squares and three pairwise products: 1 + 3 + 3 + 3 = 10 columns. Ten features at degree 2 produce 66 columns, which is why the degree is kept low and usually paired with regularisation. When a dataset has both numbers and categories, a column transformer applies a different transformer to each group of columns and joins the results, all inside the pipeline.
Metrics, thresholds and custom scorers
By default, a classifier's score is accuracy, but the right metric depends on the costs of different mistakes. Precision asks what share of the flagged cases are real; recall asks what share of the real cases were flagged. Most classifiers produce a probability, and the cut-off that turns it into a yes or no is a choice, not a law of nature. Moving it trades one metric against the other.
Grid search can optimise any metric through its scoring argument, and it can record several at once, with refit naming the one used to pick the winner. When no built-in metric matches the problem, make_scorer turns an ordinary Python function into a scorer. A function might, for example, compute the share of errors that are expensive ones, or the minimum of precision and recall so that neither is allowed to collapse. Because scikit-learn always treats larger scores as better, a function that measures an error is wrapped with greater_is_better=False, which flips its sign.
Meta estimators
A meta estimator is an estimator that wraps one or more other estimators and changes what they do. Pipelines and grid search are themselves meta estimators, and the library has many more.
- VotingClassifier combines several different models, either by majority vote or by averaging their probabilities, which often beats any single member.
- Threshold tuning wraps a classifier and changes the probability cut-off, so the threshold from the worked example becomes a setting that grid search can tune like any other.
- Class weights, available on many classifiers, make mistakes on a rare class cost more during training, which is a simple fix for imbalanced data.
- Multi-output wrappers fit one model per target column when there are several labels to predict.
The common thread is composition. Because every piece speaks the same fit and predict language, a scaler, a polynomial expansion, a voting ensemble and a threshold adjuster can be nested inside one another and searched as a single object. That design also makes scikit-learn easy to extend, and a whole ecosystem of separate libraries adds new transformers and models that slot into pipelines alongside the built-in ones.
A sensible routine for a new problem follows from all this. Start with a simple pipeline and a baseline model. Choose the metric that reflects the real costs of mistakes before tuning anything. Use cross-validated grid search to compare preprocessing choices and settings together. Look at the full results table, not only the best score, and finally confirm the chosen model once on a held-out test set.
Practice tasks
- A grid has three values for one setting, four for another and two for a third, with ten-fold cross-validation. How many fits does the search make, including the final refit?
- Scale the values 10, 20, 30, 40 and 50 with standard scaling, showing the mean and standard deviation.
- Using the ten-case worked example, compute precision and recall at a threshold of 0.6 and check your answer against Figure 5.2.
- How many columns does PolynomialFeatures produce at degree 2 for four input features, including the constant?
- Explain in your own words why scaling outside a pipeline, before cross-validation, can leak information.
- Write the outline of a custom scoring function that returns the smaller of precision and recall, and say why it might be useful.
This course book is available to read online only at tech-academy.chrysolites.org.