Most AI bills grow with context length, and context length is a design choice. Teams rarely treat it as one. They pick a model, wire it in, and discover the cost at the end of the month, by which point it is spread across thousands of sessions and no single decision looks responsible.

The cost of an answer is set much earlier than that. It is set by which model handles which task, by how much the session carries from turn to turn, and by whether anyone is watching the spend while it happens.

Where the money goes

Everything in the context window is re-sent on each turn. Cost grows with session length, and an overloaded context also degrades the output, so the same discipline pays twice. Once you look at a month of sessions by task, the spend tends to gather in 4 places:

  • The largest model doing work a small one would do as well, such as classification and extraction
  • Whole repositories or document sets loaded to answer one question
  • Long sessions that drift across unrelated tasks and carry all of them along
  • Loops, where a session retries the same failing step and keeps growing its context with each attempt

None of these is a bug in the usual sense. Each one produces a correct answer eventually. The waste is in what the answer cost.

Routing by task class

The first lever is routing. Tasks fall into classes, and the classes have very different needs. Extracting a date from an invoice, deciding which queue a ticket belongs to, or summarising a short document needs a fast, small model. Planning a change across a codebase, or reasoning about an ambiguous requirement, needs the large one.

A router decides per task, using rules the team writes down. The rules are simple at first: this kind of request goes here, that kind goes there. They get refined by looking at where the small model's answers were wrong. Routing done this way is cheap to build and easy to reason about, because each decision can be traced to a rule.

The large model is worth its price where reasoning pays for itself. Elsewhere it is an expensive way to be slightly more polite.

One task per session

The second lever is session discipline. A session that starts with a bug fix, moves on to a refactor and ends with a question about deployment carries all 3 contexts to the end. The deployment question pays for the bug fix's files.

The rule is one task per session, and a clean context between unrelated tasks. It costs nothing to follow and it is easy to check: sessions that cover several tickets show up immediately in the logs.

The third lever is how the agent finds things. An agent that reads files to discover how a system fits together pays for that reading on each turn. One that queries an index of relationships builds it once and carries a 400-token answer instead of 10,000. An architectural question costs around 400 tokens through an index, against 10,000 or more by reading.400tokens for an architectural question, through an index

Repository sweeps are the extreme case. A sweep across an unfamiliar codebase runs to between 25,000 and 50,000 tokens. On our projects the agent is not allowed to sweep. It searches, follows the index, and reads what the index points to.25,000 to 50,000tokens for a repository sweep

Cache what repeats

The fourth lever is the order of the prompt. Cache what repeats. Stable instructions and project context go first and stay unchanged, so they are served from the provider's cache on each turn. A prompt that reshuffles its opening on every call pays full price every time.

A cap on every session

A loop is caught inside the session, not in a report. Each session runs under a turn and spend cap, and stops after the same failure repeats. A session looping on a failing test hits its cap and stops, with the reason logged.

A weekly baseline

The fifth lever is attention. Spend per task class is tracked weekly against the week before. The baseline catches what a cap cannot: a task class that is quietly costing 3 times what it did last month, named with the repositories it runs on.

The baseline is a table, and a short one. Task class, sessions, tokens, cost, change against last week. Someone reads it every Monday. When a line moves, they find out why before it becomes the month's surprise.

What a budget does and does not do

A budget with an alert threshold is useful, and teams should set one. It is also the weakest of the 6 levers, because it acts after the money is spent. A budget tells you the month went badly. Routing, session discipline, navigation and caching decide whether it does.

The order of work follows from that. Measure first, so the baseline exists. Route next, because it is the largest saving for the least change. Then tighten sessions, keep the opening of each prompt stable, and add the index where the agents read most. Set the budget last, at a level the earlier changes make realistic.

Choices that stay with you

Some of these decisions belong to the client, and they should stay there. Which providers and models are acceptable is a contract and a regulator question as much as a cost one. The budget and its alert threshold are a business decision. Which tasks are important enough to deserve the large model is a product judgement. A delivery partner can measure and propose. The client decides.

Last month's invoice

Start with last month's invoice and a week of session logs. Split the spend by task class. Most teams find that a small number of classes account for most of the cost, and that at least one of them is running on a model larger than it needs.

Move that class to a smaller model behind an evaluation suite, so the change is measured rather than hoped for. Then read the baseline the following Monday. The first saving is usually large enough to fund the rest of the work.