Watching the agents: identity, logs and visibility

Companion code: 15-observability

TL;DR

  • Agents act with limited supervision, so you need visibility: which agent acted, on whose behalf, what it did, and what it cost. A 2024 paper framed this as three measures: agent identifiers, real-time monitoring and activity logs.
  • In practice that means giving every agent a stable identity and recording every agent run, model call and tool call as a trace that follows the request across agents and services.
  • OpenTelemetry’s GenAI semantic conventions give you a shared vocabulary for those traces (invoke_agent, chat, execute_tool), so your tools and dashboards don’t have to invent one.

Why “what did it do?” is hard

A chatbot’s output is in front of a person. An agent’s work mostly isn’t. It calls tools, delegates to other agents (post #7), runs for hours (post #11) and changes things in other systems. When something goes wrong (a wrong refund, a leaked file, a surprise bill), the questions are basic and often unanswerable:

  • Which agent did this, and which version of it?
  • On whose behalf was it acting?
  • What exactly did it do, in what order, and why?
  • What did it cost, and where did the time go?

In January 2024, Visibility into AI Agents (Chan et al.) argued that answering these questions is a precondition for governing agents at all, whether by a company, a regulator or a user. It proposed three kinds of measure, each implementable at different levels of intrusiveness:

  1. Agent identifiers: a way to tell which agent, from which developer and deployer, is behind an action.
  2. Real-time monitoring: watching agent activity as it happens, so problems can be caught and stopped.
  3. Activity logging: recording what agents did, for later investigation and accountability.

Two years later the need is clearer, and the practice is uneven. The 2025 AI Agent Index (Staufer et al.) documented 30 deployed agents from public information and correspondence with their developers. It found transparency varied widely, and that most developers shared little about safety, evaluations and societal impact. Visibility, in other words, mostly has to be built by the teams running agents.

A request from a user flows to a lead agent, which calls a sub-agent, which calls an MCP server. One trace ID travels with the request across all three. Each agent carries an identity: a stable ID, a version, and the user it acts for. Underneath, three layers of visibility: agent identifiers, real-time monitoring, and activity logs.

Identity: who is acting, for whom

Every agent should carry three facts with every action:

  • A stable ID and a version. “The CRM specialist, version 2.0.1”, not “the bot”. Versions matter because behaviour changes with prompts, tools and models (post #9 showed it can even change with memory).
  • A principal: the user or service it acts on behalf of. That’s what authorization should check, and what an audit needs.
  • A scope: what it’s allowed to touch. Post #10 argued for least privilege per task; identity is what makes per-agent permissions possible.

Identity belongs in your infrastructure (credentials, tokens, audit logs), not just in a system prompt. An agent that says it’s the CRM specialist has proved nothing. An agent whose requests carry the CRM specialist’s credentials, scoped to one user, has.

Tracing: what happened, in order

Distributed tracing is the standard way to follow a request through many services, and it maps naturally onto agents. A trace is one request, end to end. It’s made of nested spans: one per agent run, one per model call, one per tool call. Each span records timing, status and attributes.

OpenTelemetry is the open standard for this, and it now has semantic conventions for generative AI: agreed span names and attributes, so traces from different frameworks look alike. The agent conventions define, among others:

Operation Span name Notable attributes
running an agent invoke_agent {agent name} gen_ai.agent.id, gen_ai.agent.version, gen_ai.conversation.id
a model call chat {model} gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens / output_tokens
a tool call execute_tool {tool name} gen_ai.tool.name, gen_ai.tool.call.id, error.type

As of this writing the GenAI conventions are still marked Development status, so names can change. Pin the version you follow.

Traces also need to cross process boundaries. MCP’s 2026 revision reserves traceparent and related keys in each request’s _meta field for exactly this, so a trace can continue from an agent into the tool servers it calls.

The experiment

The companion code (15-observability) traces a two-agent system. A lead agent delegates customer questions to a CRM specialist agent through a tool, and each layer opens its span:

with tracer.start_as_current_span(f"invoke_agent {a['name']}") as span:
    span.set_attribute("gen_ai.operation.name", "invoke_agent")
    span.set_attribute("gen_ai.agent.id", a["id"])
    span.set_attribute("gen_ai.agent.version", a["version"])
    span.set_attribute("gen_ai.conversation.id", conversation_id)
    span.set_attribute("app.on_behalf_of", ON_BEHALF_OF)

Because the specialist runs inside the lead’s tool call, its spans nest under the lead’s, all sharing one trace ID. A small script prints the result as a tree. Here is a real run against Claude Opus 5.5, asking how much Bluebird Dental has spent and how many open tickets it has:

invoke_agent Support lead              8798 ms  id=agent.larkspur.lead@1.4.0 for user:ops-oncall@larkspur.example
  chat claude-opus-5-5                   2173 ms  in=449 out=87
  execute_tool ask_crm_specialist        4922 ms
    invoke_agent CRM specialist            4922 ms  id=agent.larkspur.crm@2.0.1 for user:ops-oncall@larkspur.example
      chat claude-opus-5-5                   1848 ms  in=766 out=53
      execute_tool search_customers             0 ms
      chat claude-opus-5-5                   1318 ms  in=848 out=56
      execute_tool get_customer_context         0 ms
      chat claude-opus-5-5                   1750 ms  in=985 out=83
  chat claude-opus-5-5                   1702 ms  in=616 out=86

10 spans | 5 model calls | 3 tool calls | 3,664 input tokens

Even this small tree answers the four questions at a glance. The CRM specialist (version 2.0.1), acting for the on-call user, found the customer and fetched its context in two tool calls, and the answer came back correct ($1,120 and two open tickets). The tools themselves took no measurable time: every one of the 8.8 seconds was spent in model calls, so the place to speed this up is fewer or faster model calls, not faster tools. The delegation shows up as cost: three of the five model calls, and about 70% of the input tokens, were the specialist’s. That’s the price of the separate context from post #7, and the trace shows exactly how big it is.

When something goes wrong, the same tree shows where. In a scripted test with a deliberately malformed customer ID, the failed lookup appeared as an execute_tool span marked ERROR ToolError, followed by the model’s corrected retry.

Spans go to a local file so the demo needs no infrastructure. In production, swap in OpenTelemetry’s OTLP exporter and send them to any tracing backend.

From logs to monitoring

Traces become monitoring when you watch them live and act on them:

  • Budgets. Alert, or stop the run, when a conversation’s token total or step count crosses a threshold. Runaway loops (post #8) show up as traces that keep growing.
  • Error rates by tool and by agent version. A new version whose execute_tool error rate doubles should be rolled back before users notice.
  • Sensitive actions. Any execute_tool span for sending, paying or deleting is worth a dashboard of its own, and possibly a human approval step.
  • A kill switch. Monitoring is only useful if someone can stop an agent. Make sure the harness can halt a run by ID.

Trade-offs

  • Traces can hold sensitive data. Prompts, tool arguments and results may contain personal or confidential information. The GenAI conventions make capturing message content opt-in for this reason. Decide deliberately what you record, who can read it, and for how long.
  • Volume adds up. A long agent run can produce thousands of spans. Sample routine traffic, but keep every trace that errored or touched a sensitive tool.
  • Identity is only as strong as its enforcement. An agent.id attribute is a label. Real accountability needs credentials scoped per agent and per user, checked by the systems the agent touches.

Next in the series: Small models, big systems: routing for agent cost.

Sources & further reading

#observability #governance #tracing