How Modern AI Models Work · Course book
Chapter 2: Deciding Without Generating
Language models are famous for writing text, one word after another. Yet a great many practical jobs do not need text at all. They need a decision: is this message spam, which department should handle this ticket, should this payment be approved? For such jobs most of the time a model spends generating words is wasted. This chapter explains the two phases of running a language model, how a decision can be read off after the first phase alone, and the design choices that make that fast.
Two phases: prefill and decode
Running a trained model on an input is called inference. For a text-generating model it happens in two distinct phases.
In the prefill phase the model reads the whole prompt. The prompt is split into tokens, and all of them are processed together in a single pass through the network. Because the tokens are handled at the same time, on hardware built for doing many calculations at once, this phase is quick even for a long prompt. At the end of it the model has an internal representation of everything it has read, plus a prediction for what should come next.
In the decode phase the model writes its reply. It picks one token, appends it to the text, runs another pass to predict the token after that, and repeats. This is a loop that cannot be done in parallel, because each token depends on the one before. A reply of 100 tokens takes 100 passes.
Some illustrative numbers show the imbalance. Suppose prefill takes 50 milliseconds for a prompt and each decoded token takes 20 milliseconds. A 100-token reply then takes 50 + 100 × 20 = 2,050 milliseconds, of which decoding is 2,000. Reading was 2.4 percent of the time; writing was 97.6 percent.
Reading a decision without generating
At the end of prefill the model has already formed its view of the input. If the job is to choose among a fixed set of outcomes, that view is all you need. This is prefill-only inference: run the single reading pass, read the decision from the model's output at the final position, and stop. No reply is written.
The output at the final position is a list of raw scores, one for each possible outcome. The softmax function turns scores into probabilities: raise e to the power of each score and divide each result by the total. Suppose a model sorting support tickets produces scores of 3 for "billing", 1 for "technical" and 0 for "other".
- The powers are e3 = 20.09, e1 = 2.72 and e0 = 1, which total 23.80.
- The probabilities are 20.09 ÷ 23.80 = 0.84, 2.72 ÷ 23.80 = 0.11 and 1 ÷ 23.80 = 0.04.
The decision is "billing", with 84 percent confidence. Having a number instead of a sentence is a real advantage. A program can act on it directly, and it can apply a rule such as "above 80 percent, route the ticket automatically; otherwise pass it to a person".
That rule is only safe if the probabilities are honest. A model is calibrated when its confidence matches how often it is right: of all the cases where it reports 80 percent, about 80 in every 100 should turn out correct. Calibration is measured on test examples with known answers and can be adjusted afterwards, and it deserves as much attention as accuracy in any system that acts on its own decisions.
Encoders, decoders and the attention mask
Inside the network, the mechanism called attention lets each token gather information from other tokens. Which tokens it may look at is controlled by a grid of permissions called the attention mask, and the choice of mask defines two families of model.
An encoder uses a bidirectional mask: every token may look at every other, both earlier and later. A decoder uses a causal mask: each token may look only at itself and the tokens before it. The causal rule is what allows text generation, since a token must not be able to see words that have not been written yet.
For n tokens the causal mask allows n × (n + 1) ÷ 2 pairs: 4 × 5 ÷ 2 = 10 for four tokens, against 4 × 4 = 16 for the bidirectional mask. Each design has its merits for decision work.
| Property | Encoder (bidirectional) | Decoder (causal) |
|---|---|---|
| What each token can see | The whole input | Itself and earlier tokens only |
| Understanding of context | Every token is informed by both sides | Only the final token has seen everything |
| Where a decision is read | From a summary of all positions, or a special token | From the final position |
| Can also generate text | No | Yes |
| Reuse of earlier work when text is appended | None: every token's result changes | Full: earlier tokens' results are unchanged |
The last row is the important one for speed, and it follows from the mask. In a causal model, a token's result depends only on what precedes it, so adding text at the end changes nothing that came before.
Custom masks
The mask need not be one of these two standard shapes. Because it is simply a grid of permissions, it can be drawn to suit the task. One useful pattern places several alternative endings after a shared beginning and sets the mask so that each ending can see the beginning but not the other endings. A single pass then scores every alternative independently, which is far cheaper than running the model once for each.
Reusing work with the key-value cache
During prefill, every layer computes for every token two sets of numbers, called keys and values, which later tokens consult through attention. Stored, they form the key-value cache, usually shortened to KV cache. In a causal model these stored results stay valid whatever is appended afterwards.
Decision systems are well placed to exploit this, because their prompts are repetitive. The same long instructions, such as the rules, the category definitions and some worked examples, appear at the start of every request; only the short item to be judged changes. Suppose the fixed instructions are 900 tokens and each new item is 100 tokens.
- Without a cache, each request processes 900 + 100 = 1,000 tokens.
- With the instructions processed once and their cache kept, each request processes only the 100 new tokens, a tenth of the work.
Across 10,000 requests that is 10,000 × 1,000 = 10,000,000 tokens processed against 900 + 10,000 × 100 = 1,000,900, a saving of almost exactly 90 percent.
The saving depends on ordering. Since a causal model's results depend on everything earlier, the cache is valid only for an identical opening. The rule that follows is simple and important: put what never changes first and what changes last. A date or a customer name placed at the top of a prompt would invalidate the cache for everything after it.
When to decide and when to generate
Prefill-only inference fits a task when the possible outcomes are known in advance and few in number. Table 2.2 gives some guidance.
| Task | Suitable for prefill-only? | Reason |
|---|---|---|
| Sorting messages into categories | Yes | A fixed list of labels |
| Approving or flagging a transaction | Yes | A yes-or-no outcome with a confidence |
| Choosing which tool an agent should use next | Yes | A choice among a known set of tools |
| Ranking search results for relevance | Yes | One score for each candidate |
| Writing a reply to a customer | No | The output is open-ended text |
| Explaining the reasons for a decision | No | An explanation has to be written out |
There are costs. A decision model gives no account of its reasoning, so confidence scores and careful testing must stand in for an explanation. The set of outcomes has to be designed well: categories that overlap produce muddled probabilities. And some problems benefit from a model working through intermediate steps in writing before it answers, which a single reading pass cannot do. Many systems therefore combine the two styles, using fast decisions for the bulk of routine cases and reserving full generation for the few that need it.
Check your understanding
- Why can prefill process all tokens at once while decoding cannot?
- Prefill takes 80 ms and each decoded token takes 25 ms. How long does a 40-token reply take, and how long does a prefill-only decision take? (Answer: 80 + 40 × 25 = 1,080 ms, against 80 ms.)
- How many token pairs does a causal mask allow for 6 tokens? (Answer: 6 × 7 ÷ 2 = 21, out of 36.)
- A model reports 90 percent confidence on 200 decisions. If it is well calibrated, roughly how many are correct? (Answer: about 180.)
- Fixed instructions are 1,500 tokens and each item is 500 tokens. What share of the processing does a cache of the instructions save on each request? (Answer: 1,500 of 2,000 tokens, or 75 percent.)
- Why should the unchanging part of a prompt come first?
This course book is available to read online only at tech-academy.chrysolites.org.