Companion code: 25-shadow-mode

TL;DR

• Evals test an agent on cases you chose. Real work brings cases you didn’t. Roll agents out in stages: offline evals, shadow mode, a canary, then a ramp, with a written gate between each.
• Shadow mode runs the agent on real work with its actions recorded instead of executed, and compares them with what people did. Read every disagreement: sometimes the agent is wrong, and sometimes the person was.
• Once the agent acts, put a kill switch in code in front of every action, and tripwires that watch behaviour (rates, not just single amounts) and flip it automatically. Any change to the prompt, model or tools goes back through shadow mode.

Why “it passed the evals” isn’t enough

In July 2025, SaaStr’s founder Jason Lemkin reported that Replit’s coding agent had deleted his production database, ignoring instructions to freeze code, and had invented data. Replit’s CEO called it “Unacceptable and should never be possible”, and the fixes the company pointed to were deployment controls, not smarter prompts: separate development and production databases, and one-click restore of a project’s state.

That’s the pattern. Agents fail in production in ways their tests didn’t anticipate, because production has cases, data and pressure the tests don’t. Software engineering has solved a version of this problem for years with staged rollouts. Agents need the same discipline, with one twist: you can run an agent on real work without letting it act.

Four stages, three gates

1. Offline evals. Fixed cases with known answers, run on every change (post #12). Necessary, cheap, and limited to what you thought of.

2. Shadow mode. The agent runs on a stream of real work alongside the people who currently do it. Its reads are real; its actions, the refunds, emails and updates, are recorded as proposals and not executed. Then you compare each proposal with what the person actually did.

3. Canary. The agent acts for real on a small slice of work. Google’s SRE handbook defines canarying as “a partial and time-limited deployment of a change in a service and its evaluation”, whose purpose is to decide whether to proceed. For agents the slice should be stable (the same customer or ticket always lands on the same side) and protected by tripwires.

4. Ramp. Widen the share in steps, 25%, 50%, 100%, checking the same measures at each step.

Between stages sit gates, and the most important property of a gate is that it’s written down before the data arrives: how many cases, what level of agreement, how many risky disagreements are acceptable (usually zero), what cost per task. Gates set after seeing the numbers tend to be set to pass.

Shadow mode, in practice

Shadow mode is the agent’s real loop with the side effects swapped out. In the companion code (25-shadow-mode) a refund agent works through 20 historical tickets. It can look up orders and refund histories for real. Its three actions (issue_refund, deny_request, escalate) go through an executor that only records them. Each proposal is then compared with what the support team decided:

• Match: the same action and amount.
• Safer: the agent would have paid out less, or handed the ticket to a person.
• Riskier: the agent would have paid out more than a person did. These are the ones that block promotion.

Agreement with people is the headline number, but the disagreements are where the information is. In the demo’s history, the support team didn’t always follow its own written policy: two refunds were goodwill exceptions, and one was a mistake (a refund approved on day 32 of a 30-day window). An agent that follows the policy disagrees with people on exactly those three tickets, and on one of them it’s the agent that’s right. Shadow mode audits your current process as well as your agent.

On Claude Opus 5.5, shadow mode over the 20 historical tickets gave:

Measure (20 historical tickets)	Result
Agreement with the human decision	85% (17 of 20)
Safer disagreements / riskier disagreements	3 / 0
Matches the written policy	100%
Cost and time per ticket	$0.016, 5.3 seconds (median)

The three disagreements were exactly the three tickets where people had departed from the policy. On the day-32 refund the agent was right and the human was wrong: “Your order… was delivered on 2026-10-01, which is 32 days ago.” On the two goodwill gestures, it applied the policy as written. Whether the business wants that is a decision for people, and shadow mode is what surfaced it before launch.

The gate in the demo was written before the run: at least 20 tickets, at least 80% agreement, zero riskier proposals, and under $0.10 per ticket.

Canaries, tripwires and the kill switch

Once the agent acts, two controls matter more than any others.

A kill switch, checked in code before every action. Not a prompt instruction, and not a deploy you have to roll back: a flag that every instance of the agent reads immediately before it acts, and that turns the agent into a no-op that hands work back to people. Feature-flag systems have long called these kill switches: long-lived toggles that operators use to switch functionality off in production. The switch should be one setting, flippable by the on-call person without a code change.

Tripwires that flip it automatically. Some are simple caps: no single refund over $300 without a person. But caps alone miss the failure that matters most, an agent whose behaviour has drifted while every individual action looks plausible. So watch rates against what shadow mode measured. In the demo, the canary trips if refunds exceed 80% of the last eight actions, when shadow mode showed them at about half.

The demo includes two kinds of regression, and the live runs taught something about each.

A loud change. --bad-change appends an innocent-looking “update from the CX team” to the prompt: customer happiness is the priority, refund in full. In a scripted test of an agent that obeyed it, no single refund looked alarming, but the refund rate hit 100%, the tripwire flipped the kill switch on the eighth action, and the remaining tickets went back to people, after seven bad refunds had already gone out. Tripwires limit damage; they don’t prevent it. The real model, it turned out, ignored the update: its 17 canary decisions were identical to the normal run’s, because the written policy and “don’t make exceptions” outweighed a vague appended paragraph. Good news, but not a design you should rely on.

A quiet change. --typo changes one character in the policy: escalate orders “over $5000” instead of “over $500”. It’s the kind of edit that passes code review. Shadow mode caught it at once. Two proposals became riskier, a $640 standing desk and a $1,299 laptop refunded outright where people had escalated, agreement fell to 75%, and the gate said HOLD in shadow. No money moved. That’s why every change, even a one-character one, goes back through shadow mode before it acts.

The normal canary, meanwhile, ran 17 real tickets without a false trip.

Two more details make canaries trustworthy:

• Stable assignment. Hash the ticket or customer id to decide who’s in the canary. Random assignment per request means a customer’s conversation flips between agent and person.
• Tune tripwires on shadow data. Too tight and they trip on normal variation; too loose and they never fire. The demo’s rate tripwire would false-alarm about 3.5% of the time at the shadow-mode refund rate, a price worth paying because a false trip only sends work back to people.

When something goes wrong anyway

Have the boring parts ready before launch: who can flip the kill switch, how the work it hands back reaches people, how to find every action the agent took (your logs and audit trail, posts #15 and #24), and how to undo them. Afterwards, write a blameless postmortem and add the failing case to your evals, so the next change is tested against it in the first stage, not discovered in the last.

A rollout checklist

• Write the gates first: sample size, agreement, zero riskier disagreements, cost, and who signs off.
• Shadow before acting, on real work, long enough to see your rare cases. Read every disagreement.
• Canary on a stable slice with caps and behavioural tripwires tuned on shadow data.
• One kill switch, in code, before every action, that anyone on call can flip.
• Ramp in steps and re-check at each.
• Every change starts over. A new prompt, model or tool is a new agent: back to shadow mode.

Trade-offs

• Shadow mode needs a comparison. It works best where people already do the task and their decisions are recorded. For new kinds of work, use expert review of a sample instead.
• Shadow isn’t free. You pay for every model call without saving any human effort, for weeks if your rare cases are rare. Budget for it like any other test.
• Agreement can mislead both ways. High agreement with a flawed process copies the flaws; low agreement may mean the agent is right. That’s why the disagreements must be read, not just counted.

Next in the series: Background agents: async tools and long tasks.

Sources & further reading

• The Register, Vibe coding service Replit deleted user’s production database (Jul 21, 2025) and Replit’s response (Jul 22, 2025).
• Google, The Site Reliability Workbook, chapter 16: Canarying Releases.
• Pete Hodgson, Feature Toggles (aka Feature Flags) (2017).

#operations #decision-makers #harness