Chrysolites Tech Academy

Building with AI Coding Agents · Course book

Chapter 4: From a Model to an Application

Pairs with Lesson 4: Models, APIs and Agent Harnesses with Meta's Muse

Coding agents and chat assistants are finished products. Underneath each one is a model reached through an API, wrapped in a program that manages the conversation, runs tools and enforces limits. Once you understand those layers you can build AI features of your own, switch between providers without starting again and predict what a feature will cost before the bill arrives. This chapter explains how to choose a model, what an API call contains, how cost is counted, and what a harness adds to turn a model into an agent.

Choosing a model

Providers usually offer a family of models and not a single one, and the members differ along three lines that pull against each other: capability, speed and price. The largest model reasons best and costs the most for each token. The smallest answers quickly and cheaply and manages simpler work perfectly well.

There is a further distinction in how a model is supplied. A hosted model runs on the provider's computers, and you pay for each use. An open-weights model can be downloaded and run on your own hardware; the weights are the learned numbers that make up the model. Running your own gives you control and privacy, in exchange for the cost and effort of operating the machines.

Kind of modelGood atTrade-offTypical use
Large hosted modelHard reasoning, long tasks, writing complex codeHighest price and slowest repliesPlanning, difficult debugging, agent work
Small hosted modelQuick, simple tasks in great numbersLess capable on difficult problemsSorting, extracting fields, short replies
Open-weights model run yourselfPrivacy, control, no charge per tokenYou pay for and maintain the hardwareSensitive data, offline use, experiments

Published benchmarks, which are standard test sets with scores, help to compare models but deserve some caution. A benchmark measures particular tasks that may not resemble yours, and models can be tuned to do well on famous ones. The dependable method is to collect twenty or thirty examples of your own task, run each candidate on them and compare the results. A sound default is to start with the smallest model that passes your examples and move up only where it falls short.

What an API call contains

An API is the doorway through which your program talks to the model. A request is a short structured message, and it has three essential parts: which model to use, the conversation so far and some settings.

{
  "model": "example-model",
  "max_tokens": 500,
  "messages": [
    { "role": "system", "content": "You are a helpful assistant for a bakery." },
    { "role": "user", "content": "What time do you open on Saturday?" }
  ]
}

The conversation is a list of messages, each with a role. The system message sets the model's standing instructions. User messages are what the person said. Assistant messages are the model's earlier replies. The response carries the new assistant message and a count of the tokens used.

Two facts about this arrangement surprise newcomers. First, the model is stateless: it remembers nothing between calls. The appearance of memory in a chat is produced by your program sending the whole conversation again every time. Second, the request format has become something of a common language. Many providers accept requests in the same shape as the most widely used client libraries, so moving to a different provider can be as small a change as pointing the library at a new address and changing the model's name. Writing your code against that common shape keeps your options open.

Counting the cost

Hosted models are billed by the token, with separate prices for input (what you send) and output (what the model writes), each quoted per million tokens. Output usually costs several times more than input. Prices vary between models and change over time, so the figures below are round illustrative numbers: 3 units of currency per million input tokens and 15 per million output tokens.

Take one call with 2,000 input tokens and 500 output tokens.

  • Input: 2,000 ÷ 1,000,000 × 3 = 0.006
  • Output: 500 ÷ 1,000,000 × 15 = 0.0075
  • Total: 0.0135 for the call

That looks negligible, and for one call it is. A feature used 10,000 times a day costs 10,000 × 0.0135 = 135 a day. Notice also that the 500 output tokens cost more than the 2,000 input tokens.

Statelessness makes long conversations dearer than they seem. Suppose the system message is 1,000 tokens, each turn brings 500 new input tokens, and each reply is 300 tokens. Because everything is sent again, the input for turn t is 700 + 800 × t tokens.

  • Turn 1: 1,500 input tokens
  • Turn 10: 8,700 input tokens
  • Turn 20: 16,700 input tokens

Across all 20 turns the input adds up to 182,000 tokens. Had every turn been as short as the first, it would have been 20 × 1,500 = 30,000. Resending the history multiplied the input roughly sixfold.

1,5008,70016,7003,00010,200 history resent in fullafter compaction at turn 10 010,00020,000 15101520turn in the conversation input tokens sent on that turn
Figure 4.1 — Input grows on every turn because the history is sent again. Compaction, shown by the dashed line, replaces old turns with a short summary and resets the growth.

This is why long-running tools use context compaction. When the history grows large, the older turns are replaced by a brief summary. In the figure, compaction after turn 10 swaps 8,000 tokens of history for a 1,500-token summary, so turn 11 sends 3,000 tokens instead of 9,500. Detail is lost, which is the price, so anything that must survive, such as a decision already made, should be written somewhere permanent.

From a model to an agent

A model on its own can only produce text. It becomes an agent through tool calling. Along with the messages, your program sends a list of tools, each with a name, a description and the inputs it accepts. The model cannot run anything itself. What it can do is reply with a structured request: call this tool with these inputs. Your program runs the tool and sends the result back as a new message, and the model carries on.

Your program (harness) Model 1. question + list of available tools2. "call get_weather with city = Lagos"3. tool result: "31 degrees, clear"4. final answer in plain language program runs the tool itself
Figure 4.2 — One round of tool calling. The model only asks; your program decides whether to run the tool and does the running.

Repeat that exchange in a loop and you have an agent. The program that runs the loop is called a harness. It holds the conversation, offers the tools, carries out the calls, applies the rules and decides when to stop. A coding agent in a terminal is a harness whose tools read files, edit files and run commands. You can write a harness yourself in a few dozen lines, or use an agent framework, a library that supplies the loop and common tools so that you write only what is particular to your task.

Harnesses share a set of controls that are worth recognising, whatever they are called in a given product.

  • A project instruction file read at the start of every session, holding the conventions of the project.
  • Skills, packaged instructions loaded only when a task needs them.
  • Task breakdown, in which a large goal is split into a checklist of smaller steps that are worked through and ticked off.
  • Reasoning effort, a setting for how much the model thinks before replying. More effort improves hard answers and costs more time and tokens, so it is wasted on easy ones.

Keeping an agent within bounds

A loop that runs commands needs limits that do not depend on the model behaving well. Four are standard.

Sandboxing runs the agent's commands in a restricted environment. The agent sees only the project folder, and access to the rest of the machine and to the network is blocked unless granted. If the agent is misled or mistaken, the damage is confined.

Approval rules state which actions run freely and which wait for a person. Reading is normally free; deleting, installing and sending are normally not.

Budgets cap the number of steps, the tokens or the time a task may use. Without one, an agent stuck repeating a failing command will continue until someone notices. With one, it stops and reports.

Connection standards. The Model Context Protocol gives agents a uniform way to reach outside systems such as databases and issue trackers. Each server adds tools, and each tool's description takes up room in the context window, so connecting many servers "in case they are useful" makes an agent both slower and less focused. Connect what the task needs.

The model proposes; the harness disposes. Every safety property of an agent is enforced by ordinary code in the harness, not by the model. When judging an agent product, ask what its harness will refuse to do.

Check your understanding

  • Why is it sensible to begin with the smallest model that passes your own test examples?
  • What does it mean that a model is stateless, and how does a chat appear to remember earlier turns?
  • At 3 per million input tokens and 15 per million output tokens, what does a call with 10,000 input and 1,000 output tokens cost? (Answer: 0.03 + 0.015 = 0.045.)
  • Using the formula 700 + 800 × t, how many input tokens does turn 15 send? (Answer: 700 + 12,000 = 12,700.)
  • In tool calling, who actually runs the tool: the model or your program?
  • What does context compaction gain, and what does it lose?
  • Name two limits a harness can enforce that do not rely on the model's good behaviour.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.