Inside AI Hardware: Chips and Data Centers · Course book
Chapter 2: Power, Cooling and Connections
A large AI data center is sometimes called an AI factory: electricity and data go in, and trained models and answers come out. Its design is governed less by computing than by physics. Enormous amounts of power must be delivered without interruption, the same amount of heat must be carried away, and thousands of processors must exchange data as if they were one machine. This chapter follows each of those flows and works out the numbers involved.
Power: from the grid to the rack
Electrical power is measured in watts. A kettle uses about 2 kilowatts (2,000 watts). A megawatt is a thousand kilowatts and a gigawatt is a thousand megawatts. Traditional data centers used tens of megawatts; the largest AI sites are planned in gigawatts, comparable to a large power station's entire output.
Power equals voltage multiplied by current. Delivering a lot of power therefore needs a high voltage, a high current or both, and there is a strong reason to prefer voltage: the energy lost as heat in a cable grows with the square of the current. Consider delivering 1 megawatt.
- At 400 volts, the current is 1,000,000 ÷ 400 = 2,500 amperes.
- At 40,000 volts, the current is 1,000,000 ÷ 40,000 = 25 amperes.
The second case carries the same power with 100 times less current, so for the same cable the loss is 100 × 100 = 10,000 times smaller. This is why electricity travels long distances at very high voltage and is stepped down in stages by transformers as it nears the equipment.
Training a large model can occupy thousands of processors for weeks, so an interruption is expensive. Reliability comes from layers of backup. An uninterruptible power supply, or UPS, uses batteries to take over instantly when the grid falters, but only for a short time. That is long enough for backup generators to start and carry the load for hours or days. Finally, power distribution units, or PDUs, share the supply out among the racks. Critical parts are duplicated so that no single failure switches anything off.
How much power, and how many racks
A rack is a tall cabinet holding servers. In a traditional data center a rack draws perhaps 5 to 10 kilowatts. A rack filled with AI processors can draw 100 kilowatts or more, as much as dozens of homes, in the floor space of a wardrobe. This density is what forces everything else in the building to change.
Not all the electricity a site buys reaches the computers. Some runs the cooling and some is lost in conversion. The standard measure is power usage effectiveness, or PUE: total power drawn by the facility divided by the power that reaches the computing equipment. A perfect score is 1.0, and modern sites reach about 1.1 to 1.3.
Take a facility whose computing equipment draws 100 megawatts, with a PUE of 1.25 and racks of 100 kilowatts each.
- Racks: 100 megawatts is 100,000 kilowatts, and 100,000 ÷ 100 = 1,000 racks.
- Total power: 100 × 1.25 = 125 megawatts, of which 25 megawatts is overhead.
- Energy in a year: a year has 8,760 hours, so 125 × 8,760 = 1,095,000 megawatt-hours.
If a household uses 10 megawatt-hours a year, that is the consumption of about 110,000 homes. A site ten times larger, at a gigawatt, matches a city. This is why new facilities are sited wherever large amounts of power can be had, and why the wait for a grid connection, often measured in years, has become the main limit on how quickly they can be built.
Cooling: moving heat out
Almost every watt of electricity that enters a computer leaves it as heat. A 100 kilowatt rack is therefore a 100 kilowatt heater, and the heat must be removed as fast as it is produced or the chips shut down within seconds.
Older data centers blow chilled air through the racks. Air is a poor carrier of heat, and beyond a few tens of kilowatts per rack no practical quantity of it suffices. Water is far better: a given volume of water absorbs roughly 3,500 times as much heat as the same volume of air for each degree of warming. AI racks therefore use liquid cooling. In the common direct-to-chip method, a metal cold plate sits on each processor and coolant flows through it, taking the heat away at its source.
The flow needed follows from a simple formula: heat removed each second equals the mass of water flowing each second, multiplied by water's heat capacity (about 4,186 joules per kilogram per degree), multiplied by the temperature rise. For a 100 kilowatt rack, with the water allowed to warm by 10 degrees:
flow = 100,000 / (4,186 x 10) = about 2.4 kilograms per second
A kilogram of water is a litre, so the rack needs about 2.4 litres a second, or 143 litres a minute. The 1,000 racks in the example need about 2,400 litres every second, circulating round a closed loop.
The heat carried away must finally be released outdoors, a step called heat rejection. Table 2.1 compares the main approaches.
| Method | How it works | Strength | Drawback |
|---|---|---|---|
| Air cooling in the hall | Chilled air is blown through the racks | Simple and well understood | Cannot cope with dense AI racks |
| Direct-to-chip liquid | Coolant flows through cold plates on the processors | Removes heat at the source | Needs plumbing to every server |
| Cooling towers | Warm water is cooled by evaporating some of it outdoors | Very efficient in electricity | Consumes large volumes of water |
| Dry coolers | Fans blow outside air over pipes of warm liquid | Uses almost no water | Less effective in hot weather and uses more electricity |
| Chillers | A refrigeration cycle cools the liquid, like a giant refrigerator | Works in any climate | The most electricity of all |
The choice depends on the local climate and on whether water or electricity is scarcer. Cooler regions can reject heat to the outside air for most of the year, which is one reason they attract data centers.
Networking: thousands of chips as one
Training a large model is not thousands of separate jobs. The processors work on one problem together and must share results constantly, so the connections between them matter as much as the processors. A slow link leaves expensive chips waiting.
Two kinds of connection are used. Within a rack, where distances are a metre or two, processors are joined by copper cables into a tightly coupled group; this is called scaling up. Between racks and across the building, copper cannot carry such high data rates far enough, so signals are converted to pulses of light and sent through optical fibre; joining many racks this way is called scaling out. The small devices that convert between electrical signals and light are called transceivers, and a large site contains hundreds of thousands of them, joined by enough fibre to stretch across a country.
Building it and paying for it
Constructing such a facility is a vast project. Besides the building, it requires equipment with long delivery times, particularly large transformers, generators and switchgear, which may need to be ordered years ahead. Because the processors lose value quickly, every month of delay is costly, and construction is organised to bring sections of the site into use one at a time.
The economics turn on utilisation, the share of the time that the hardware is doing useful work. Take an illustration with round numbers. Suppose a processor and its share of the building cost 30,000 units of currency and are written off over 4 years. Four years is 4 × 8,760 = 35,040 hours, so the hardware costs 30,000 ÷ 35,040 = about 0.86 for each hour, whether or not it is used. If it is busy only 60 percent of the time, each useful hour must cover 0.86 ÷ 0.6 = about 1.43.
Electricity is a smaller part of the bill than many expect. If the processor and its share of cooling draw 1 kilowatt and electricity costs 0.08 per kilowatt-hour, power adds 0.08 an hour, against 0.86 for the hardware. This is why operators care so much about keeping machines busy, and why much capacity is sold in advance under long contracts in which a customer commits to pay for a fixed amount of computing over several years. Such contracts give builders the certainty they need to raise the money, and give customers assured access to scarce hardware.
Check your understanding
- What current is needed to deliver 2 megawatts at 10,000 volts? (Answer: 2,000,000 ÷ 10,000 = 200 amperes.)
- Why is electricity sent over long distances at high voltage?
- What is the job of the UPS, and why are generators needed as well?
- A facility's computing equipment draws 50 megawatts and its PUE is 1.2. What is its total power, and how many 125 kilowatt racks does it hold? (Answer: 50 × 1.2 = 60 megawatts, and 50,000 ÷ 125 = 400 racks.)
- How much water flow does a 50 kilowatt rack need if the water warms by 10 degrees? (Answer: 50,000 ÷ 41,860 = about 1.2 litres a second.)
- Hardware costs 1.00 an hour to own. What does each useful hour cost at 50 percent utilisation? (Answer: 1.00 ÷ 0.5 = 2.00.)
- Why is optical fibre used between racks instead of copper cable?
This course book is available to read online only at tech-academy.chrysolites.org.