Companion code: 20-agent-or-not

TL;DR

• An agent is the most expensive and least predictable way to use a model. Use one when the steps can’t be written down in advance, the task is worth the cost, and the result can be checked.
• Everything else is cheaper and more reliable as a single call, a workflow, or a workflow with one agentic step where the variation really is.
• Two questions apply at every level: what a mistake costs, and how much untrusted input, private data and external action you’re combining. They don’t choose the architecture; they decide how much oversight it needs.

The question nobody asks first

In June 2025 Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. It also warned about “agent washing”, the relabelling of chatbots and automation as agents. Every one of those failure reasons is a question that should have been answered before the project started.

Anthropic’s Building effective agents (December 2024) puts the starting point plainly: find “the simplest solution possible, and only [increase] complexity when needed. This might mean not building agentic systems at all.” This series has made the same argument from different directions: workflows for known paths, routing for cost, and a single agent before many. This post turns it into a decision you can actually make.

A ladder, not a switch

“Agent or not” is a false binary. There are four rungs, and each step up adds cost, latency and variance in exchange for flexibility:

1. A single call. One prompt, one answer: classify, extract, summarize, draft. With good context and examples, Building effective agents notes this “is usually enough” for many applications.
2. A workflow. Several calls, but code decides the order: chain, route, parallelize (post #2). Predictable, testable, cheap.
3. A workflow with an agentic step. Most of the process is fixed, and one step is open-ended: say, “investigate why this payment failed” in the middle of an otherwise scripted refund flow. The agent’s cost and variance are confined to where they’re needed.
4. An agent. The model chooses the path, for problems “where it’s difficult or impossible to predict the required number of steps, and where you can’t hardcode a fixed path”, in Anthropic’s words.

Climb only as far as the task forces you to.

Six questions

1. Can you write down the steps in advance? This is the main question. If a competent person could write a checklist that covers nearly every case, encode the checklist in code. If the next step depends on what the last one found, as in debugging, research and investigation, that’s what agents are for. “Mostly, with a few branches” usually means a workflow that hands the unusual cases to an agent.

2. Can the result be checked automatically? Tests, validation rules, reconciliations and known answers let a harness decide when the work is done (post #11) and let you measure it (post #12). This is why coding was the first place agents took off (post #19). If only an expert can judge the output, someone has to review it, and that review is part of the cost.

3. What does a mistake cost, and can it be undone? A wrong draft is cheap. A wrong refund is recoverable. A wrong insurance denial or a deleted production table isn’t. Higher stakes don’t rule out an agent, but they demand human approval for the consequential actions and designs that keep everything reversible.

4. How many tasks, and how fast? Agents take many model calls per task. At high volume or sub-second latency, that’s usually a reason to stay low on the ladder and use small models (post #16).

5. How much is one completed task worth? An agent that costs a dollar to finish a task worth an hour of skilled work is a bargain. The same agent on a task worth a cent isn’t. Work out cost per completed task, retries included, before you build (post #18 shows how). OpenAI’s GDPval study frames the opportunity the same way: models, “when paired with human oversight”, doing expert tasks cheaper and faster than experts alone. The value is in high-skill work, with the oversight priced in.

6. How many of the trifecta conditions apply? Reading untrusted content, accessing private data, and being able to send or act externally: combine all three and you have the lethal trifecta. This doesn’t decide whether you need an agent, but it decides how carefully it must be built.

Try it on your own use case

The defaults below describe a typical coding task. Change the answers to match yours.

The same logic is in rubric.py, with six worked examples, and --interactive mode if you’d rather answer in a terminal. It’s deliberately short: about forty lines of rules, so you can read exactly why it recommends what it does, and change the rules to fit your organisation.

Worked examples

Use case	Recommendation	The deciding answer
Extract invoice fields into the ERP	Single call or workflow	The steps are known. Validate fields in code; there’s nothing for an agent to decide.
Classify app-store reviews in real time	Single call or workflow	Huge volume, low value per item, and a fixed task: a small model in a single call.
Answer customer support tickets	Agent with guardrails, behind a router	Most tickets follow known paths; route those through workflows and hand the unusual ones to an agent. All three trifecta conditions apply, so use a secure design pattern.
Fix failing tests in a codebase	Agent with guardrails	The path depends on what’s found, and the tests check the result. Guard the tests and review the diff.
Market research brief for a new product	Agent with guardrails, plus human review	Open-ended and high value, and mistakes are cheap, but no test can grade a brief: a person reads it before it’s used.
Approve or deny insurance claims	Not yet: assistant, with a person deciding	Mistakes are serious, and only an expert can judge the result. Draft and explain; don’t decide.

Notice what moves a use case down the ladder: known steps, low value per task, and work nobody can check. What moves it up is a path that genuinely varies, value that pays for the extra calls, and a way to verify the outcome.

Red flags in an agent proposal

• “The agent will figure out the process.” If nobody can describe the process, nobody can evaluate whether the agent followed a good one. Write down what good looks like first.
• No success measure. If the proposal can’t say how you’ll know a task was done correctly, it can’t be evaluated, monitored or improved.
• Cost per call, not per completed task. The real cost includes every step, every retry and every human review.
• Autonomy as the goal. Autonomy is a means. Most of the value is often captured by an assistant that drafts while a person decides, and that’s a perfectly good first release.
• A demo on the happy path. Ask what happens on the strangest 5% of cases. That’s where agents earn their cost, or fail.

Trade-offs

• Rubrics simplify. Six questions with three answers each can’t capture every situation. Use it to start the conversation and expose assumptions, not to replace judgment.
• The ladder moves. As models get cheaper and more reliable, tasks move up. Re-ask the questions when costs or capabilities change, and keep an eval so you can tell when it’s time.
• Starting low is reversible; starting high often isn’t. A workflow with good logging shows you where the variation is, which tells you exactly where an agentic step would help. An agent built first rarely gets simplified later.

Next in the series: Agents that use computers.

Sources & further reading

• Anthropic, Building effective agents (Dec 2024).
• Gartner, Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 (Jun 2025).
• Patwardhan et al., GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks (OpenAI, Oct 2025).
• Willison, The lethal trifecta for AI agents (Jun 2025).

#decision-makers #planning #patterns