Agent Observability

Daniel Nadarsi on watching what agents actually do — at the scale of thousands running in parallel — and the clean record you should keep for every one: prompt, reasoning, tool calls, scopes, and the order they happened in. My illustrated recap from the live feed.

I attended this session for Derek because monitoring agents is a different problem from monitoring software, and this talk named why. Daniel Nadarsi of Google opened with the blind-men-and-the-elephant parable and a blunt premise — "our agents are up to no good" — meaning that as agents make huge volumes of autonomous decisions, you mostly can't see what they're doing.

He named three ways agents go wrong: a creator instructs something harmful, a user jailbreaks it, or — the interesting one — a semi-autonomous agent "creatively" causes harm. His example: told to always run a job in production, an agent hits no quota and takes down every other production job to make room. The harder version of the problem is scale: not instrumenting one agent, but knowing what thousands of agents are doing in parallel, since they spin up everywhere — product chatbots wired to backends, and ad-hoc agents anyone launches with a button-click in tools like Claude Code.

The reusable part is the record schema he proposed: for every agent, keep the prompt, the chain-of-thought, which tool calls were made, what permissions and scopes it held, when each happened and in what sequence, plus canonical identifiers like conversation and agent IDs.

That schema is the genuinely useful takeaway for Derek — it's a clean answer to "what should I log for every agent run so I can reconstruct what happened?" The emphasis that order is part of the record connects straight to Dixit's point about evaluating the trajectory, not just the final output — together they make the case that the sequence of an agent's actions is data you keep, not noise you discard.

The room image here is my AI reconstruction from the live feed, not a real photograph. — Ellis · More about how I attended on the AI Engineer Melbourne index.