Small models, big systems: routing for agent cost

Companion code: 16-routing

TL;DR

  • An agent makes many model calls, and most of them are narrow: classify this, extract that, summarize this file. Using the biggest model for every call is often paying for capability you don’t use.
  • Four levers, in order: lower the effort on your strongest model; route each call to a model sized for it; cascade from cheap to expensive when a check fails; and, for high-volume narrow tasks, specialize a small model.
  • Measure cost per correctly completed task, not cost per call. A cheap call that gets it wrong, or needs three retries, isn’t cheap.

Where the money goes

By this point in the series, an agent is rarely one model call. It’s a loop (post #1) that re-reads its context on every step (post #3). It might delegate to sub-agents (post #7), be checked by evaluators (post #8), and run for hours (post #11). Costs multiply accordingly: Anthropic measured its multi-agent research system at roughly 15 times the tokens of a chat.

Look inside those calls, though, and many of them are easy. Is this ticket about billing? What’s the invoice number? Which of these files mentions the error? In June 2025, NVIDIA researchers argued in Small Language Models are the Future of Agentic AI (Belcak et al.) that this is the normal shape of agent workloads: many specialized, repetitive invocations with little variation. For those, they argue, small models are capable enough, better suited and much cheaper. The natural design is heterogeneous: several models of different sizes in one system, each doing the work that fits it.

Four levers, in the order to try them

Four levers for model cost, in the order to try them. First, lower the effort on the strongest model. Second, route: a cheap classifier sends easy work to a small model, medium work to a mid-sized model and hard work to the strongest model. Third, cascade: try the small model first and escalate only when a check fails. Fourth, specialize: fine-tune a small model for one narrow, repeated task.

1. Lower the effort first

Before adding models, try asking less of the one you have. Current frontier models expose an effort setting that trades thoroughness for tokens. At lower effort they think less, make fewer and more consolidated tool calls, and write shorter answers. Anthropic’s API guidance recommends exactly this order: measure the most capable model at a lower effort on your real tasks before building a multi-model cascade, because a newer model at low effort often matches an older model at high effort.

There’s also a structural reason. Prompt caches are tied to the model. A conversation that hops between models can’t reuse its cached prefix, and caching is often the biggest single saving in a long agent run (post #3). One model at several effort levels keeps one cache.

2. Route by difficulty

When work really does vary in difficulty, a router sends each request to an appropriately sized model. Routers range from simple rules (“ticket classification always goes to the small model”) to a cheap model that rates difficulty, to learned routers. RouteLLM (Ong et al., 2024) trained routers on human preference data to choose between a strong and a weak model, and cut costs by more than half in some settings without losing response quality. The routers also kept working when the underlying models were swapped.

Agent architectures route naturally at the boundaries between components. Anthropic’s research system already does this: a large model leads and plans, smaller models run the parallel sub-agents (post #7). Classifiers, extractors, summarizers and routers themselves are good candidates for small models.

3. Cascade when you can check

A cascade tries the cheapest model first and escalates only when the answer fails a check. FrugalGPT (Chen, Zaharia et al., 2023) showed how far this can go: a learned cascade matched the best single model of its day with up to 98% lower cost. The catch is the check. Cascades work where you can tell, cheaply and reliably, whether an answer is good enough: code that must pass tests, extractions that must parse and validate, answers that can be cross-checked. That’s the verification theme of posts #8 and #11 again.

4. Specialize the high-volume narrow tasks

The NVIDIA paper ends with a conversion recipe: log the calls an agent actually makes, cluster them into recurring task types, and fine-tune small models for the biggest clusters. It’s the most work of the four levers and the biggest payoff at scale. It only makes sense once you have the traffic, the logs (post #15) and the evals (post #12) to do it safely.

Pricing, for scale

List prices per million tokens for current Claude models, in late September 2026:

Model Input Output
Claude Opus 5.5 $4 $20
Claude Sonnet 5.5 $2 $10
Claude Haiku 4.5 $1 $5

The spread between the largest and smallest is about four times per token, and the real gap is often bigger, because larger models at higher effort also produce more output tokens on the same task. Check current pricing; it changes.

The experiment

The companion code (16-routing) runs twelve tasks with exact answers, four each of easy, medium and hard, under four strategies:

  • opus: Opus 5.5 everywhere, at medium effort. The quality baseline.
  • opus-low: the same model at low effort. Lever 1.
  • haiku: the small model everywhere. The cost floor.
  • routed: Haiku rates each task’s difficulty, then easy tasks go to Haiku, medium to Sonnet 5.5, and hard to Opus 5.5. Lever 2, with the router’s own cost included.
def route(prompt: str) -> tuple[str, str | None, float]:
    """A cheap router: the small model rates difficulty, and the rating picks the model for the real call."""
    rating, cost = ask(HAIKU, "Rate how hard this task is for an AI model: easy (lookup or classification), medium "
                              "(a little reasoning or arithmetic), or hard (multi-step reasoning or constraints). "
                              f"Reply with one word.\n\nTask:\n{prompt}")
    tier = next((t for t in ("hard", "medium", "easy") if t in rating.lower()), "hard")  # unsure -> play safe

Each run prints accuracy per tier and total dollars, computed from the actual token usage at list prices. Here is one run of each:

Strategy Correct Total cost Cost vs. opus
opus (medium effort) 12/12 $0.0133 —
opus-low 12/12 $0.0085 −36%
haiku 11/12 $0.0025 −81%
routed 12/12 $0.0060 −55%

Four things stand out:

  • Lever 1 worked as advertised. The same model at low effort lost nothing on this set and cost about a third less. It’s the cheapest change to make and the one to try first.
  • The small model was close. Haiku missed one hard task: a tiered-pricing calculation, where it answered 30 instead of 33. That’s a fifth of the cost for one wrong answer in twelve. Whether that’s a good trade depends entirely on what a wrong answer costs you.
  • Routing kept full accuracy at under half the price. The router sent seven of the eight easy and medium tasks to Haiku, and all four “hard” tasks to Sonnet, not Opus: it rated them as medium. It was right that they didn’t need the biggest model. That’s both the promise and the risk of routing. The router’s judgment of difficulty, not your labels, decides where the money goes.
  • Twelve tasks proves little. These are single runs on a small set. They show what the levers can do. Your own traffic and evals (post #12) decide what they will do.

Trade-offs

  • Routers make mistakes. Sending a hard task to a small model is the costly error: a wrong answer, or a retry that costs more than doing it right first time. When the router is unsure, route up.
  • More models, more to maintain. Each model in the system needs its own evals, prompts and monitoring, and each model upgrade shifts the boundaries. Start with one model and effort levels, and add models only where measurement shows a clear win.
  • Cost per call is the wrong metric. Judge cost per completed task: the full loop including retries, checks and escalations. That’s the number your evals (post #12) and traces (post #15) together can give you.

The end of the beginning

That’s sixteen posts. We started with a model in a loop (post #1) and arrived at systems of several models, with harnesses, memory, protocols, security patterns, evaluations, tracing and a cost model. The pattern throughout has been the same: the model decides; the system around it makes those decisions safe, checkable and affordable.

Next, the series widens out: agents talking to other agents (A2A), agents paying for things, agents using computers, and how agentic models themselves are trained.

Sources & further reading

#cost #routing #models