Companion code: 23-caching

TL;DR

• An agent sends its whole context on every step, so most of what it pays for is input it has sent before. Caching turns those repeats into cheap reads: on Claude Opus 5.5 a cache read costs 5% of the normal input price.
• The cache is a prefix: tools, then the system prompt, then the messages, in that order. Change one byte and everything after it is reprocessed. Keep the front frozen and the history append-only.
• The common breakers are silent: a timestamp in the system prompt, tools in a different order, a reminder injected into the latest turn and removed later. Measure the hit rate from the usage fields, and let cache diagnostics tell you what changed.

Why agents live or die by the cache

A chatbot sends a question and gets an answer. An agent sends its instructions, its tool definitions and everything that has happened so far, gets back a short tool call, runs it, and sends the whole lot again with one more result appended. Manus, which builds a general-purpose agent, reported in July 2025 that its average input-to-output ratio was about 100:1, and called the cache hit rate “the single most important metric for a production-stage AI agent”.

Post #3 measured what this means in practice. An agent that kept its history intact read 87% of its input from the cache and cost $0.51. A version that cleared old tool results to save space kept rewriting its cache and cost $0.96, nearly twice as much, despite sending fewer tokens. Caching beat cleverness.

The prices make the case on their own. On Claude Opus 5.5, input is $4 per million tokens. Writing to the cache costs 1.25× that ($5), and reading from it costs $0.20, a twentieth of the normal price. A cache entry pays for itself the first time it’s read.

How the cache works

The rules fit in a paragraph. The API assembles every request in a fixed order: tools, then the system prompt, then the messages. You mark breakpoints with cache_control. At each breakpoint the API stores the prefix up to that point, and a later request that starts with exactly the same bytes reads it back instead of reprocessing it. Entries last five minutes by default, refreshed every time they’re read, or an hour if you pay double to write them. Prefixes shorter than the model’s minimum (512 tokens on Claude Opus 5.5) silently aren’t cached.

Everything else follows from prefix. The cache can’t skip a changed section and pick up after it. One different byte, anywhere, and every breakpoint after that byte misses.

The setup that suits agent loops

Anthropic’s documentation shows a combination of two breakpoints that suits agent loops well, and it’s what the companion code uses:

client.messages.create(
    model="claude-opus-5-5",
    tools=TOOLS,                                     # fixed list, fixed order
    system=[{"type": "text", "text": SYSTEM,         # byte-identical on every request
             "cache_control": {"type": "ephemeral"}}],  # 1. explicit breakpoint after the stable part
    cache_control={"type": "ephemeral"},             # 2. automatic breakpoint at the end of the conversation
    messages=history,                                # append-only
    ...
)

The explicit breakpoint guarantees the expensive, stable part (tools and system prompt) can always be read back, whatever happens to the conversation. The automatic one moves forward each turn, so each request reads everything up to the previous turn and writes only what’s new. In a healthy loop, cache_read_input_tokens grows every turn while cache_creation_input_tokens stays small.

Three silent breakers

None of these raises an error. The agent works exactly as before; it just costs more.

1. A timestamp in the system prompt. “Current time: 2026-10-30T09:14:02Z” at the top of the system prompt looks harmless and helpful. It makes the very start of every request unique, so nothing after it is ever read back. Worse, each request still writes the cache at 1.25× the price. A cache that’s written and never read costs more than no cache at all. If the agent needs the date, put it in a message, after the breakpoints, and only to the precision it needs.

2. Tools in a different order. Tools come first in the prefix, so any change there invalidates everything. Reordering counts, and it’s easy to do by accident: tools collected from a set, plugins that register in whatever order they load, a schema serialized without sorted keys. Build the list once, in a fixed order, and keep it fixed for the conversation.

3. Editing history. The classic is a reminder injected into the newest message (“remember to check the incident log”) and removed from the older ones on the next request, so it only ever appears once. Each request then rewrites a message the previous request cached. The system prompt still reads from cache, but the conversation is reprocessed every step. Truncating old tool results, re-serializing earlier turns, or “cleaning up” history do the same. Treat the history as append-only: leave old reminders in place, or use a mechanism designed for this, such as Anthropic’s mid-conversation system messages or its server-side context editing (post #3).

Two more are worth knowing. Switching models mid-conversation starts from an empty cache, since caches are per model, so routing (post #16) should happen between tasks, not within one. And parallel requests with the same prefix all pay full price, because none can read what the others are still writing: send one, wait for its first token, then send the rest.

On current models, these mistakes can also break requests

There’s a second reason to get this right. On Claude Opus 5.5, a thinking block is tied to the exact conversation that produced it: the system prompt, the tools and every earlier message. For accounts created since August 31, 2026, a request that replays thinking blocks after the system prompt, the tools or earlier turns changed is rejected with an error, rather than just missing the cache. The same append-only discipline keeps both working.

Measure it

Every response reports three numbers in usage: input_tokens (processed at full price, after the last breakpoint), cache_creation_input_tokens (written to the cache) and cache_read_input_tokens (read from it). The total prompt is all three added together, which catches people out: a long agent run can show a tiny input_tokens because almost everything was read from cache.

The hit rate is reads divided by that total. Track it per agent, and alert when it drops: caching usually breaks through a later change to how prompts are assembled, not in the first version, and nothing else will tell you.

When it does drop, cache diagnostics finds the cause. Pass the previous response’s id with each request, and the response says where the two requests first diverged: system_changed, tools_changed, messages_changed or model_changed, with an estimate of the tokens lost. It’s available on the Claude API only.

r = client.messages.create(..., diagnostics={"previous_message_id": previous.id})
r.diagnostics.cache_miss_reason.type   # e.g. "system_changed"

The experiment

The companion code (23-caching) runs a small operations agent once, finding which warehouse’s on-time shipping got worse and why, and records every request it made. It then replays exactly that sequence of requests under five ways of assembling the prompt: stable (the setup above), timestamp, tool order, history edit and no caching. Replaying keeps the comparison fair: the conversations are identical, so only the assembly differs. The replays use max_tokens: 0, which processes the prompt and updates the cache without generating anything, so they cost only their input.

The agent took four requests: list the warehouses, fetch all eight monthly figures in one parallel batch, check Columbus’s incident log, and answer (Columbus, late rate 2.2% → 7.4%, most likely the carrier pickup moved from 18:00 to 15:00). Replayed under each assembly on Claude Opus 5.5, twice, with near-identical results:

Assembly	Hit rate	Input cost vs. no caching	Cache diagnostics reported
Stable	69%	43%	nothing
Timestamp in system prompt	0%	126%	system_changed
Tools reordered	20%	101%	tools_changed
History edited	56%	59%	messages_changed
No caching	0%	100%	–

Four things stand out.

The timestamp costs more than no cache at all. Every request wrote a fresh cache entry at 1.25× the input price, and none was ever read: 26% more than sending everything uncached. It’s the worst of the five, and it’s the one most often added with good intentions.

Reordered tools lost almost everything. The only reads came on the fourth request, when the rotation happened to put the tools back in their original order and matched the first request’s entry. A real nondeterministic ordering would hit by chance about as often.

The history edit kept the system prompt. Because of the explicit breakpoint after the system prompt, every request still read the 2,238 tokens of tools and system prompt from cache, and lost only the conversation. Without that breakpoint, the edit would have cost nearly as much as the timestamp.

Diagnostics named every breaker on the first request where it happened, which is the point of the feature. One caveat: in the no-caching variant it also reported messages_changed once, between the first and second requests, although nothing had been edited. It’s a hint to read alongside the usage numbers, not a verdict on its own.

This conversation is short. Stable caching reads a larger share with every turn, so a 20-step agent would show a higher hit rate, and the breakers would cost proportionally more.

A checklist

• Freeze the front. Tools and system prompt byte-identical for the whole conversation: no timestamps, request IDs, user names or feature flags in them. Put per-request facts in messages.
• Fix the tool list. Same tools, same order, deterministic serialization. Change behaviour through messages, not by adding and removing tools mid-task.
• Append, never edit. Old reminders stay; old tool results stay, or are cleared by the API’s own context editing.
• One model per conversation. Route between tasks, not within them.
• Put the breakpoints where stability changes: after the system prompt, and at the end of the conversation.
• Measure in production. Hit rate per agent from the usage fields, with an alert, and cache diagnostics switched on.

Trade-offs

• Five minutes is short. Entries expire after five minutes without a read. An agent waiting on a slow tool or a person can lose its cache. The one-hour option costs twice as much to write and pays off only when gaps of 5–60 minutes are common.
• Stable prefixes constrain design. Some good ideas, such as personalizing the system prompt or swapping tools per phase, fight the cache. Usually there’s a cache-friendly way to do the same thing; occasionally the better prompt is worth the misses. Measure both.
• Caching doesn’t shrink the context. It makes re-reading cheap, not short. A context that grows without limit still slows the model and eventually overflows (post #3).

Next in the series: Agent identity: who is this agent, and what may it do?

Sources & further reading

• Anthropic, Prompt caching and Cache diagnostics documentation.
• Yichao “Peak” Ji, Context Engineering for AI Agents: Lessons from Building Manus (Jul 2025).
• Post #3: Context is a budget, for the measured cost of clearing versus caching.

#cost #context #harness