DevOps Foundations: Linux, Pipelines and Delivery · Course book
Chapter 3: Building a Pipeline in Jenkins
Chapter 2 described continuous integration and deployment in principle. This chapter builds them with Jenkins, one of the oldest and most widely used automation servers. Jenkins is free, runs on your own machines and can be extended with well over a thousand plugins, which is why it is found in so many organisations. The chapter explains the vocabulary Jenkins uses, how a typical setup is arranged, how a pipeline is written as code in a Jenkinsfile, and how Docker fits into the picture to package and ship the result.
What Jenkins does, and the words it uses
Jenkins is a program that waits for something to happen, typically a new commit pushed to a code repository, and then carries out a defined series of steps: fetch the code, install its dependencies, run the tests, build the application and deploy it. It records the result of every run, shows the output of every step and can notify the right people as soon as something fails.
| Term | Meaning |
|---|---|
| Job or project | A task that Jenkins knows how to run, such as "build and test the website" |
| Build | One run of a job, numbered in sequence: build #1, build #2 and so on |
| Pipeline | A job described as a series of stages, written as code |
| Stage | A named group of steps within a pipeline, such as Build, Test or Deploy |
| Step | A single action, such as running a shell command |
| Agent or node | A machine on which Jenkins runs the steps |
| Trigger | What starts a build: a code push, a schedule, or someone pressing a button |
| Plugin | An add-on that teaches Jenkins a new ability, such as talking to GitHub or Docker |
Jenkins began life as a tool configured entirely through web forms. Those "freestyle" jobs still exist, but they have a serious weakness: their configuration lives inside Jenkins, invisible to anyone reading the code, and is easily lost or changed without trace. Modern practice is to describe the pipeline in a file kept alongside the code, as explained below.
Arranging the pieces
A small but realistic setup has four parts, shown in Figure 3.1. Developers push code to a repository on a hosting service such as GitHub. A Jenkins server, here a rented Linux virtual machine in the cloud, notices the push and runs the pipeline. The pipeline builds a Docker image of the application and uploads it to a registry, a storage service for images such as Docker Hub. Finally a separate server, which runs the application for users, pulls the new image and starts it.
Setting up the Jenkins server itself follows a familiar Linux pattern: install Java, which Jenkins needs, install Jenkins from its package repository, start it as a service, and open its web interface in a browser. On first launch Jenkins asks for an initial administrator password stored in a file on the server, offers to install a recommended set of plugins and asks you to create an administrator account. Plugins worth adding early include Git support, Docker support and Blue Ocean, a modern interface that draws pipelines as a clear sequence of stages with green and red markers.
Pipelines as code: the Jenkinsfile
A Jenkinsfile is a text file, stored in the root of the repository, that describes the pipeline. Jenkins reads it on every run, so the pipeline is versioned, reviewed and shared like any other code. A simple declarative Jenkinsfile for a Node.js website might read:
pipeline {
agent any
stages {
stage('Install') {
steps { sh 'npm install' }
}
stage('Test') {
steps { sh 'npm test' }
}
stage('Build image') {
steps { sh 'docker build -t myname/site:latest .' }
}
}
post {
failure { echo 'Build failed: check the test output.' }
}
}
The structure is easy to read. agent any lets Jenkins run the pipeline on any available machine. Each stage has a name and steps; sh runs a shell command. If any step fails, by finishing with a non-zero exit code, the pipeline stops at that stage and later stages are skipped, so a failing test prevents an image from being built. The post section runs afterwards, here printing a message only when the build failed.
Updating the pipeline is now just a matter of editing this file and pushing the change. Adding a Lint stage between Install and Test, for example, takes three lines, and the next build shows the new stage automatically.
Pipelines can do more than run stages in a line. Independent checks, such as unit tests and the linter, can run in parallel stages on different agents, cutting the total time. A stage can be limited with a when condition, so that, for example, the Deploy stage runs only for the main branch while other branches stop after testing. And an input step can pause the pipeline until a person approves, which suits teams that want a human decision before anything reaches production.
Every build is kept with its full console output, its duration and its result, so the history doubles as a record of the project's health. Blue Ocean shows at a glance which stage failed and why, and the trend of build times reveals when a pipeline is slowly getting longer and needs attention. Notifications by email or chat ensure that a failure on the main branch is noticed within minutes, by the person whose change caused it, while the change is still fresh in their mind.
Builds should be triggered automatically. The usual method is a webhook: the repository host is told the address of the Jenkins server and sends it a short message whenever code is pushed. The alternative, having Jenkins check the repository for changes every few minutes, works but is slower and wasteful.
Packaging and shipping with Docker
A test that passes on the Jenkins server is useful only if the application then runs the same way in production. Docker provides that guarantee. A Dockerfile in the repository lists how to assemble an image: start from a base image with Node.js installed, copy in the code, install dependencies and state the command that starts the site. docker build produces the image, and the image includes everything the application needs.
The pipeline then logs in to Docker Hub and pushes the image. Credentials such as the Docker Hub password must never be written into the Jenkinsfile, which anyone with access to the repository can read. Jenkins has a credentials store for this purpose: secrets are saved once in Jenkins, given a name, and referred to by that name in the pipeline, so their values never appear in code or in build logs.
Images are identified by tags. Using only latest makes it hard to know which version is running or to roll back. A better habit is to tag each image with the build number or the commit identifier as well, for example myname/site:57, so that every image can be traced to the exact code it contains. Rolling back then simply means running the previous tag again.
Images also benefit from being small. Starting from a slim base image, copying in only the files the application needs and listing the rest in a .dockerignore file keep images quick to build, push and pull, which shortens every run of the pipeline.
The final stage deploys. The simplest approach connects to the application server over SSH and runs commands to pull the new image, stop the old container and start a new one. Larger systems hand this step to an orchestrator such as Kubernetes, which is the subject of the next course.
Taken together, the timings show why this is worth the effort. A manual release that takes 30 minutes of careful work, done twice a week, costs about 52 hours a year and carries the risk of a forgotten step each time. An automated pipeline might take 5 minutes of machine time per run and no human time at all, so the team can release ten times a day if it wants to.
Check your understanding
- What is the difference between a job, a build and a pipeline in Jenkins?
- Why is a Jenkinsfile better than configuring a job through web forms?
- In the Jenkinsfile above, what happens if npm test fails? (Answer: the Test stage fails, the Build image stage is skipped, and the failure message is printed.)
- What is a webhook, and why is it preferred to checking the repository on a schedule?
- Where should a Docker Hub password be kept, and why not in the Jenkinsfile?
- Why tag images with a build number as well as latest?
- A manual release takes 45 minutes and happens three times a week. How many hours a year is that? (Answer: 2.25 hours a week, about 117 hours a year.)
This course book is available to read online only at tech-academy.chrysolites.org.