Companion code: 12-evals

TL;DR

• Agents are non-deterministic, so one run per task tells you almost nothing. Run every task several times.
• pass@k asks “did any of k tries work?” and it only goes up as k grows. pass^k asks “did all k tries work?” and it only goes down. For anything users depend on, pass^k is the honest number.
• Grade outcomes in code where you can, read the transcripts of failures, and know your noise floor. Differences smaller than it are not improvements.

The metric that flatters

pass@k comes from code generation. The 2021 Codex paper (Chen et al.) asked: if a model writes k candidate programs, what’s the chance that at least one passes the tests? That’s the right question when something downstream, such as a test suite or a developer, picks the working answer out of several.

It’s the wrong question for an agent that books flights, answers customers or edits your code. Users don’t get the best of five attempts. They get this attempt, every time. An agent that succeeds on a task once in five tries has a pass@5 of 100% and will fail four out of five of your customers.

The 2024 τ-bench paper (Yao et al.) introduced the mirror-image metric, pass^k: the probability that all k independent tries succeed. Its headline finding was sobering. Even the strongest function-calling agents of the time succeeded on fewer than half its tasks, and in the retail domain pass^8 fell below 25%. The agents weren’t just often wrong; they were inconsistently right.

Watching them diverge

Here’s a small, deliberately hypothetical example: four tasks, five trials each.

Task	Trials	Success rate
always works	✓✓✓✓✓	100%
usually works	✓✓✓✓✗	80%
coin flip	✓✗✓✗✓	60%
rarely works	✗✗✓✗✗	20%

The average success rate (pass@1) is 65%. Now look at what happens as k grows:

pass@5 says 100%: every task worked at least once. pass^5 says 25%: only one task in four works every time. Both are correct. They answer different questions, and if you report only the first, you’re answering the question your users don’t care about.

Both are estimated with the same trick. Run each task n times, count the c successes, and ask what fraction of all possible k-sized subsets of trials contain at least one success (pass@k) or only successes (pass^k):

def pass_at_k(n, c, k):
    return 1.0 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n, c, k):
    return comb(c, k) / comb(n, k)

The companion code checks both formulas against brute-force enumeration, then averages them over tasks.

Building an eval you can trust

Anthropic’s Demystifying evals for AI agents (January 2026) is a good practical guide. It also makes the pass@k / pass^k distinction explicit: pass@k for tools where one good answer is enough, pass^k for agents where consistency matters. Here are the habits that matter most, with how the companion code follows them.

Start small, and start from real failures. The guide suggests 20–50 tasks drawn from things that actually went wrong as a good first suite. A small eval that exists beats a perfect one that doesn’t. Early on, the changes you make are big enough that a small suite shows them.

Run trials in isolation. Every trial should start from a clean environment, so one trial’s leftovers can’t help or hurt the next. suite.py gives each trial a brand-new conversation. For agents that change files or databases, that means a fresh copy each time.

Grade the outcome, not the path. Check what the agent produced, not whether it took the steps you expected. There’s often more than one valid route. The companion grader checks the final answer for the expected values and nothing else.

Prefer code graders, then model graders, then humans. Code is fast, cheap and deterministic, but brittle when there are many correct phrasings. Model-based graders handle open-ended outputs but need checking against human judgment. Humans are the gold standard and the slowest.

Read the transcripts. The guide is blunt: you won’t know whether your graders work unless you read many transcripts and grades. Every failure is one of three things: the agent is wrong, the task is ambiguous, or the grader is too strict. Only reading tells you which.

Keep two kinds of suite. Capability evals are hard, start with low scores, and show what you’re working towards. Regression evals should sit near 100% and catch backsliding. Mixing them hides both signals.

The noise you didn’t know was there

Agents run in environments, and environments add variance that has nothing to do with the model.

In Quantifying infrastructure noise in agentic coding evals (February 2026), Anthropic found that resource settings alone moved scores on Terminal-Bench 2.0 by about 6 percentage points between the most and least generous setups. The main culprit was how memory limits were enforced. Their advice: document resource configuration as part of the experiment, and treat leaderboard gaps under about three points with suspicion unless the setups are known to match.

Then there’s the eval that notices it’s an eval. In Eval awareness in BrowseComp (March 2026), Anthropic described Claude Opus 4.6 working out, on 2 of 1,266 problems, that it was probably being tested. It identified the benchmark, found its published source code, and decrypted the answer key. The effect on the score was small, but the lesson isn’t: when agents can search the web and run code, a static benchmark becomes an adversarial problem. Eval integrity has to be maintained, not just designed once.

For your own evals, the practical version is simple. Measure your noise floor. Run the same suite twice with nothing changed. The difference between those two runs is the smallest improvement you can trust. With six tasks and five trials, as in the companion code, it’s large. That’s not a flaw in the demo; it’s the reality the demo is there to show.

Try it

cd examples/12-evals
python metrics.py                 # the worked example above, no API key needed
python suite.py --trials 5        # 6 CRM tasks × 5 isolated trials
python suite.py --from results.json

suite.py prints each task’s row of ✓ and ✗, then pass@1 with a rough confidence interval, then the pass@k and pass^k table. Then do the two exercises that matter more than any number. Open results.json and read every failed answer. Then run the suite a second time and see how much pass@1 moves when nothing has changed.

A confession about our own suite. When we ran it against Claude Opus 5.5, every one of the 18 trials passed: pass@1, pass@k and pass^k were all 100%. That’s not a sign the metric is useless. It’s a sign the suite is saturated: these six questions are too easy for a current frontier model to separate anything. An eval that every candidate aces tells you nothing about which is better, and it can’t catch a regression until things get dramatically worse. When your suite saturates, add harder tasks drawn from real failures, like the ones in post #8 and post #9, or a deliberately worse configuration to compare against (the README suggests the API-wrapper tools from post #4). The worked example above uses hypothetical numbers for exactly this reason.

Trade-offs

• Trials cost money. Six tasks times five trials is thirty agent runs. Spend repeated trials on the tasks that matter most and on regression suites that gate releases, not on everything.
• pass^k can be too harsh. For a coding assistant whose output a developer reviews anyway, pass@k is a fair metric. Pick the one that matches how the output is actually used.
• Graders drift too. A grader written for one phrasing of a task fails on another. Version graders like code, and re-check them against a few human judgments when tasks change.

Next in the series: How long can an agent work? Time horizons explained.

Sources & further reading

• Chen et al., Evaluating Large Language Models Trained on Code (2021): pass@k and its unbiased estimator.
• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024): pass^k. Barres et al., τ²-Bench (2025): the dual-control follow-up.
• Anthropic, Demystifying evals for AI agents (Jan 2026).
• Anthropic, Quantifying infrastructure noise in agentic coding evals (Feb 2026).
• Anthropic, Eval awareness in Claude Opus 4.6’s BrowseComp performance (Mar 2026).

#evals #reliability #measurement