Chrysolites Tech Academy

Infrastructure as Code: Pulumi, Terraform and the AWS CDK · Course book

Chapter 4: Infrastructure on Demand

Pairs with Lesson 4: Build a Heroku-Style Platform with Python and Pulumi

Platforms such as Heroku made deployment feel effortless: a developer picks what they want, fills in a short form or types one command, and a running website appears, with no knowledge of the servers underneath. Behind such platforms is infrastructure code run not by a person at a terminal but by another program. This chapter shows how that works by building a small self-service platform of your own: a Python web application that creates, lists, updates and deletes real cloud resources, static websites and virtual machines, whenever a user submits a form. It introduces the idea of an automation API and the design questions that arise when infrastructure is created on demand.

Infrastructure code inside an application

In the previous chapters, a person ran the IaC tool: they edited files, ran a preview and confirmed the change. That suits a team managing its own environments. A platform is different. Its users never see the infrastructure code; they ask for a website, and the platform must create it, reliably and without anyone typing a command.

An automation API makes this possible by exposing the IaC engine as a library that your own program calls. Pulumi's Automation API, used here, lets a Python program create a stack, set its configuration, run the equivalent of pulumi up, read the stack's outputs and destroy it, all through ordinary function calls. The infrastructure itself is described by a Python function, called the program, which the API runs whenever a stack is deployed.

TaskUsing the command lineUsing the automation API
Create an environmentpulumi stack init, then pulumi upCall a function to create or select a stack, then call its up method
Change a settingpulumi config set, then pulumi upSet a configuration value in code, then call up again
Read resultspulumi stack outputRead the outputs returned by up, or query the stack
Remove everythingpulumi destroyCall the stack's destroy method, then remove the stack
Who triggers itA person at a terminalAny event in an application, such as a submitted form

The underlying engine is identical; only the way it is driven changes. That means everything learned about state, previews and idempotence still applies, which is reassuring when real resources are being created on behalf of users.

Setting up the project

The project needs two command-line tools installed and configured: the Pulumi CLI, whose engine the Automation API uses, and the AWS CLI, which stores the credentials that the program will use to talk to AWS. As before, those credentials should belong to a dedicated user with only the permissions the platform needs.

A new Pulumi project is created from a Python template for AWS, which sets up a virtual environment and installs the AWS provider. Flask, the small web framework met in the CS50 course, is installed alongside it, together with a library for making web requests. The application follows Flask's usual factory pattern: a function in app.py creates the Flask app, registers the parts of the application and returns it.

The user interface is built from Jinja templates. A base template provides the shared layout and navigation; an index page offers the choice between websites and virtual machines; and separate pages list existing resources and show forms for creating and editing them. Keeping the templates simple matters less than keeping the logic clear, because the interesting work happens on the server.

Feature 1: static websites on demand

The first feature lets a user create a website by entering a name and some HTML content. When the form is submitted, the server's route function does four things.

  1. It defines the Pulumi program for one site: an S3 bucket configured for website hosting, an object holding the submitted HTML as index.html, a policy allowing public reading, and an exported output holding the site's address.
  2. It creates a stack named after the site, so that each website has its own separate stack and state.
  3. It calls up to deploy the stack and waits for it to finish.
  4. It redirects the user to the list of sites, now including the new one.
Browsersubmits a form Flask approute functions Automation APIone stack per item S3 sitestatic website EC2 VMvirtual machine outputs such as the site address flow back to be listed
Figure 4.1 — The platform's flow. Each form submission becomes a stack operation, and each stack owns exactly one user's website or machine.

The program is just a Python function defined inside the route, which is what makes the approach so flexible. Because it is created at request time, it can use the values from the form directly: the site's name becomes part of the bucket name, and the submitted HTML becomes the content of the index object. Passing values through the stack's configuration instead is a cleaner alternative when the same program serves many stacks, since the configuration is stored with each stack and can be read back later.

Stack names need care. They must be unique within the project, so the platform should check that a requested name is not already taken and contains only allowed characters before creating anything. Telling the user "that name is in use" is far better than failing half-way through a deployment.

Listing sites is the reverse. The program asks the Automation API for every stack in the project and reads each one's exported outputs, so the list page can show every site's name and address without the platform keeping a separate database. The state already records everything.

Editing a site reuses the same machinery. The edit form shows the current content; on submission the program selects the existing stack and calls up again with the new HTML. Because IaC is idempotent and works by comparing desired and actual state, only the changed object is uploaded, and the bucket itself is left alone. Deleting calls destroy, which removes the bucket and its contents, and then removes the stack.

One stack per thing. Giving every website or machine its own stack keeps them independent: updating one cannot accidentally change another, and deleting one removes exactly its own resources.

Feature 2: virtual machines on demand

The second feature creates EC2 virtual machines, with a form asking for a name and an instance size. The program for a VM creates a security group allowing SSH access and web traffic, looks up a current machine image with a data source, uploads the user's public SSH key as a key pair, and launches the instance, exporting its public IP address and DNS name.

Testing it needs an SSH key pair, generated on your own computer with ssh-keygen. The platform reads the public key and passes it to the stack's configuration, and once the machine is running, the private key on your computer lets you log in to it at the exported address. The list page for virtual machines shows each machine's name, size and address, gathered from the stacks' outputs as before. Deleting a machine from the list calls destroy on its stack, which terminates the instance and removes its security group and key pair together, leaving nothing behind to be billed.

Changing the instance size from the edit page shows something important about IaC. Some properties can be updated in place, while others force a replacement. Changing the size of a running EC2 instance requires it to be stopped and restarted, and other changes would destroy it and create a new one with a different address. A careful platform would show the user a preview of the change, using the API's preview method, before applying anything that replaces a machine.

Lessons for real platforms

A demonstration like this omits much that a production platform needs, and listing those gaps is instructive. Deployments take from seconds to minutes, so a real platform runs them as background jobs and shows progress instead of making the web request wait. Users must be authenticated, and each must see and change only their own stacks. Inputs such as names must be validated, since they become resource names with strict rules. Costs need limits, so that no user can create a fleet of large machines. And state must live in a durable, shared backend, not on the web server's disk.

Even so, the core idea is the same one used by large commercial platforms: infrastructure code, driven by a program, turning a request into running resources and keeping a precise record of what exists. Combine it with the pipelines, containers and modules of the previous chapters, and you have the building blocks of an internal developer platform, the self-service portal many organisations now give their engineers.

Check your understanding

  • What is an automation API, and how does it differ from running the IaC tool at a terminal?
  • Why does the platform create one stack for each website?
  • How can the platform list every site without keeping its own database?
  • When a user edits a site's HTML, why is the bucket not recreated?
  • Which half of an SSH key pair is uploaded to AWS, and which stays on your computer?
  • Why might changing a virtual machine's settings replace it, and how could the platform warn the user first?
  • Name three things a production platform would need that this demonstration leaves out.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.