Multi-turn agents: where the tokens actually go
September 9, 2026
A single LLM call costs what its prompt costs. An agent that loops — observe, call a tool, read the result, decide the next call — resends the whole history every turn. A conversation's cumulative cost isn't the sum of N independent calls: every turn pays again for all the ones before it.
Why the growth is quadratic
If each turn adds roughly T tokens to the history (question, answer, tool result), turn N sends roughly N×T tokens of context. Over a 10-turn loop, the last call pays ten times the first turn's cost just to re-read the history — before even counting the generation itself.
This is a structural problem, not an implementation accident: a stateless language model has no way to "remember" a previous turn other than reading it back in full on every call.
History compression, and its real constraint
The standard approach: past a token threshold, summarize the oldest messages via a dedicated LLM call (usually a lighter model than the one driving the agent), and inject that summary in place of the raw history. The most recent messages stay untouched — a summary should never reach into the current turn, where detail matters most.
The real risk isn't the cost of the summarization call (marginal next to what it saves), but information loss: an overly aggressive summary can make the agent forget a constraint set three turns earlier. Hence keeping a fixed number of the very last messages unsummarized, and only triggering compression past a real threshold rather than every turn.
An adjacent trap: provider-side caching
Providers exposing prompt caching (see the cost-breakdown guide) generally index it on a stable prefix of the context. Rewriting the start of the conversation — even to compress it — invalidates that cache for the next turn. There's a genuine tension between "compress to shrink what's sent" and "preserve the prefix to keep the cache warm": the two optimizations aren't free relative to each other, and the trade-off depends on how much of the conversation is left.
Grouping turns so they can be measured
A looping agent produces one HTTP call per turn. Without an explicit conversation id propagated by the caller, every turn shows up in logs as an isolated request with no link to the others — there's no way to answer "what did this conversation cost end to end?". A session id, supplied by the client rather than guessed server-side (a message-prefix hash breaks the moment an agent truncates its history in a sliding window), is what makes a loop's real cost visible instead of each of its calls taken in isolation.