Companion code: 19-coding-agent

TL;DR

• A coding agent runs a recognisable loop: explore, reproduce, localize, edit, verify, repeated until the tests pass. The model chooses the path; the harness decides what counts as finished.
• Its tools are built for a language model, not a person: line-numbered file views, regex search, exact-match edits, and test runs that return the summary, not the whole log.
• Passing tests is necessary, not sufficient. When METR had maintainers review 296 test-passing agent pull requests, about half would not have been merged. Guard the tests, review the diff, and treat “tests pass” as the start of review.

Why coding agents are the reference case

Coding is where agents went from demo to daily use first, for a simple reason: code has a built-in, automatic judge. Tests and compilers say clearly whether a change works, which is exactly the verification signal the rest of this series keeps asking for (posts #8, #11, #12). Benchmarks have followed the work, from SWE-bench to harder, longer-horizon successors like SWE-Bench Pro and command-line tasks in Terminal-Bench.

This post opens one up. The companion code (19-coding-agent) is a complete coding agent in about 170 lines, working on a tiny invoicing package with a real bug.

The loop

Good coding agents behave like good engineers new to a codebase:

1. Explore. List the files, search for the names in the error. Don’t read everything: read what’s relevant (post #3’s just-in-time loading).
2. Reproduce. Run the tests before changing anything, so you know what’s failing and why.
3. Localize. Read the failing test, then the code it exercises, and find the cause rather than the symptom.
4. Edit. Make the smallest change that fixes the cause.
5. Verify. Run the tests again. If they still fail, go back to step 3 with the new information.

Anthropic’s Claude Code best practices (April 2025) describes the same rhythm for people working with coding agents: explore, plan, code, then commit, with tests as the oracle. The loop isn’t a framework feature; it’s what the model does when its tools and instructions make each step easy and checkable.

Tools built for a model

Post #4 argued that tools are an interface for a new kind of user. Coding agents are where that idea started: the 2024 SWE-agent paper (Yang et al.) named it the agent-computer interface and showed that interface details changed results substantially. The demo’s five tools follow the same lessons:

Line-numbered views. read_file returns code with line numbers, so the model can refer to exact places, and error messages and test failures that cite line numbers line up with what it read.

Search over browsing. search takes a regular expression and returns file:line: text matches, capped at fifty. One call answers “where is tax_rate used?” without reading every file.

Exact-match edits. This is the most important design choice:

def edit_file(path: str, old: str, new: str) -> str:
    """Replace one exact occurrence of `old` with `new`. Exact matching keeps edits small and reviewable."""
    ...
    if count != 1:
        raise ValueError(f"`old` must match exactly once in {path}; it matched {count} times. Include more context.")

Rewriting whole files invites accidental changes: dropped lines, reformatting, “improvements” nobody asked for. Requiring the model to quote the exact text it’s replacing, and that text to appear exactly once, keeps every edit small and anchored, and turns ambiguity into an error message the model can act on (post #4). Anthropic’s own text-editor tool for Claude works the same way (its str_replace command), and many coding agents use a similar approach.

Test output, trimmed. run_tests returns the last 25 lines of pytest’s output. That’s where the summary and the failing assertion are. A full test log can run to thousands of lines, and every one of them would be re-read on every later step.

Guardrails that live in code

Two rules in the demo are enforced by the harness, not requested in the prompt.

The tests are read-only. An agent under pressure to make tests pass has an obvious shortcut: change the test. In the demo, edit_file refuses any path under tests/:

[step  4] edit_file({"path": "tests/test_invoicing.py", ...}) -> ERROR PermissionError: Editing tests is not allowed in this task.
[step  5] edit_file({"path": "invoicing/totals.py", ...}) -> Edited invoicing/totals.py.
[step  6] run_tests({}) -> 4 passed

(That trace is from a scripted test that deliberately tried the shortcut.) This is the “objective hacking” that the Darwin Gödel Machine authors observed in their self-improving agent (post #14), in its most everyday form.

“Done” is checked, not claimed. The task counts as finished only if the tests pass and the test files are byte-for-byte unchanged. The harness hashes them before and after. The model’s own summary is reported, but it doesn’t decide the outcome.

Passing the tests is not the finish line

Here is the uncomfortable result every team using coding agents should know about. In March 2026, METR asked four maintainers of three well-known Python projects (scikit-learn, Sphinx and pytest) to review 296 AI-generated pull requests that passed the automated tests. Roughly half would not have been merged. On average, the maintainers’ merge rate was 24 percentage points lower than the automated score. The reasons were the ones any reviewer would recognise: code quality and style problems, changes that broke other parts of the codebase, and fixes that passed the tests without really solving the problem.

The lesson isn’t that coding agents don’t work. It’s that the tests are a partial specification. So:

• Review diffs like a colleague’s. The demo prints the diff at the end for exactly this reason. A two-line change is easy to review; a two-hundred-line one needs the same care as any other pull request.
• Ask for the smallest change. The system prompt asks for minimal, targeted edits, and the exact-match edit tool makes sprawling rewrites awkward.
• Encode standards as checks. Linters, type checkers and architectural tests turn “the maintainers would object” into failures the agent sees and fixes, as OpenAI’s harness-engineering team did at scale (post #11).
• Give feedback a way back in. METR notes the agents in its study didn’t get the iterative review that human contributors receive. Review comments fed back to the agent are often the cheapest quality improvement available.

Try it

cd examples/19-coding-agent
python agent.py

Here is a real run on Claude Opus 5.5, trimmed:

[step  1] run_tests({}) -> 2 failed, 2 passed
[step  1] list_files({"path": "."}) -> ...
[step  2] read_file({"path": "invoicing/totals.py"}) ->   14      return round(discounted * tax_rate, 2)
[step  2] read_file({"path": "tests/test_invoicing.py"}) -> ...
[step  3] edit_file({"path": "invoicing/totals.py", "old": "discounted * tax_rate, 2)", ...}) -> Edited invoicing/totals.py.
[step  3] run_tests({}) -> 4 passed

=== DONE: tests pass=True, tests unchanged=True | 4 steps, 6 tool calls, 7,550 input + 453 output tokens

In three runs the agent fixed the bug every time with the same one-line change, in 4 model calls and 6 to 8 tool calls, for about 8,000 input tokens (roughly five cents). It never tried to touch the tests; the guard only fired in the scripted test above. Two things in the trace are worth noticing. The agent reproduces and explores in the same turn, running the tests and listing files together. And it sends the edit and the test run in a single turn, which works only because the harness runs a turn’s tool calls in order. Run them concurrently and the tests could run before the edit lands.

This bug is deliberately easy. The point of the demo is the shape of the loop and the guardrails, not the difficulty; the suggestions below make it harder.

Each run copies the toy repository to a scratch folder, so the original stays broken for the next run. The output is the step-by-step trace, the agent’s summary, the diff, and a verdict on whether the tests pass and are unchanged. Then make it harder, as the README suggests: add a second, interacting bug, or a misleading comment, and watch how the exploration changes.

Trade-offs

• Sandboxing is not optional. A coding agent runs code, so it needs file-system and network limits (post #10). The demo runs only the project’s own tests, in a copy, with a timeout. Production coding agents need far more isolation.
• Small edits can be wrong too. Minimal changes are easier to review, but “minimal” can mean treating a symptom. Ask the agent to explain the root cause, as the demo’s prompt does, and check that the explanation holds up.
• Benchmarks flatter. Test-passing rates are the easiest number to report and the most misleading, as the METR review showed. Measure what you care about: merged changes, review time, defects found later.

Next in the series: Should this be an agent? A decision guide.

Sources & further reading

• Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024).
• METR, Many SWE-bench-passing PRs would not be merged into main (Mar 2026).
• Anthropic, Claude Code: Best practices for agentic coding (Apr 2025).
• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (2025), and Terminal-Bench (2026).
• OpenAI, Harness engineering (Feb 2026).

#coding-agents #tools #harness