# AI traces are worth a thousand logs

 | MLOps Community | 14:44

Source: https://www.youtube.com/watch?v=yC3XS85CPaQ
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/ai-traces-are-worth-a-thousand-logs
Published: 2025-08-26
Tags: agents, observability, testing, tool-use, tracing

## TL;DR
- An agent can be understood as instructions, tools, and an underlying language model connected through a loop.
- A trace is a sequence of OpenTelemetry spans that records each model call, tool execution, input, output, and related metadata.
- Returning the trace directly to Python makes it possible to write assertions and tests against an agent's behavior.

## Summary
David presents a simple model of how agent frameworks work. An agent has instructions, tools, and a language model. The model runs in a loop, calling tools when needed, appending their results to the message history, and continuing until it produces a final answer. He treats memory, external knowledge, retrieval systems, and even other agents as tools that can fit into this same loop.

He defines a trace as a sequence of JSON objects called spans. Each span records an operation such as a model call or tool execution, along with its inputs, outputs, and metadata. Any Agent converts different framework implementations into an OpenTelemetry trace and returns it as an AgentTrace object. This lets developers inspect the same data in Python that they see in trace visualisation tools. David's main practical use case is testing: developers can assert that a particular tool was used and that its arguments were correct.

## Key ideas
### An agent has three parts connected by a loop
[01:58](https://www.youtube.com/watch?v=yC3XS85CPaQ&t=118s)
David reduces an agent to its instructions, available tools, and underlying language model. Instructions tell the agent how to behave. Tools let it take actions beyond producing text, with the model following a defined tool-calling format. The model powers the whole process. In Any Agent, these parts are supplied through one interface, along with a user query. The framework then runs the agent using the selected underlying framework.

### Most agent frameworks run the same basic cycle
[03:28](https://www.youtube.com/watch?v=yC3XS85CPaQ&t=208s)
The implementation David describes starts with instructions and the user prompt in a message history. The framework calls the language model and checks whether it wants to execute a tool. If so, the tool runs, its result is added to the history, and the model is called again. This repeats until the model gives a final answer. David says frameworks add different layers of abstraction, but this loop is the common structure he has seen.

### Memory and other agents can be exposed as tools
[04:53](https://www.youtube.com/watch?v=yC3XS85CPaQ&t=293s)
David treats memory, external knowledge, retrieval systems, and other agents as tools. A framework can build special mechanisms for these capabilities, or expose them to the agent as external operations. The agent then decides when to use them within the same loop it uses for web search or other tools. He argues that implementing as much as possible as a tool keeps the basic model simple.

### A trace records every operation as an OpenTelemetry span
[05:54](https://www.youtube.com/watch?v=yC3XS85CPaQ&t=354s)
In Any Agent, a trace follows the OpenTelemetry meaning of the term: a sequence of JSON objects, with each object called a span. A span represents one operation in the agent loop. Model calls record their input arguments, output arguments, and additional metadata. Tool executions are recorded in the same way. Any Agent is intended to produce a common trace across frameworks while following OpenTelemetry semantic conventions for generative AI.

### Different trace interfaces contain the same underlying data
[07:30](https://www.youtube.com/watch?v=yC3XS85CPaQ&t=450s)
Trace platforms can render an agent run as a tree with more visual detail, while a console renderer can print the data in a readable format. David's point is that these views are built from the same JSON spans. The interface may change, but the underlying object still contains the steps the agent took to reach its answer. He wants developers to have access to that object instead of relying only on an external visualisation service.

### Returning the trace to Python enables local inspection
[09:07](https://www.youtube.com/watch?v=yC3XS85CPaQ&t=547s)
Any Agent returns the OpenTelemetry trace as a Pydantic object called AgentTrace. Developers can inspect and manipulate it in the same Python process that runs the agent. David says this makes it possible to run assertions without sending data to an external service. Trace visualisation tools remain useful, but the underlying data can also be used directly in application code.

### Agent traces can become test fixtures
[13:26](https://www.youtube.com/watch?v=yC3XS85CPaQ&t=806s)
In the question period, David gives a concrete testing use case. A test can run an agent, inspect the returned trace, and assert that a specific tool was used with the correct arguments. He compares this with assertions in ordinary Python workflows. He says Any Agent is exploring other uses for traces, but this is the direction he personally finds most useful.

## Notable quotes
- "In my mind a very simple view of what's an agent are these three components stitched together." (01:58)
- "Trace is an OpenTelemetry object that contains information about different steps that the agent took to reach an answer." (08:42)
- "Why not just return that object to the user so you can manipulate and inspect it also in your Python code." (09:01)
- "I really like the concept of just building tests on top of this agent trace." (13:47)

## Tools & references mentioned
- Mozilla.ai
- Any Agent
- OpenTelemetry
- Google ADK
- LangChain
- LlamaIndex
- OpenAI
- Open Inference
- Model Context Protocol
- Weights & Biases
- Arize
- Phoenix
- Langfuse
- TraceLens
- Pydantic

## Who should watch
- You are building agents across more than one framework and need one interface for comparing their behavior.
- Your agent failures are difficult to understand because model calls and tool executions are spread across separate logs.
- You want to test tool selection and tool arguments directly in Python instead of checking a tracing dashboard by hand.

## Related talks

- [How AgentOps Enables Observability](https://mlopstalks.com/talks/how-agentops-enables-observability) (Nehil Jain, Stealth AI Startup & Adam Becker, MLOps Community & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Binoy Perera, MLOps Community, 49:48)
- [AI Agent Development Tradeoffs You NEED to Know](https://mlopstalks.com/talks/ai-agent-development-tradeoffs-you-need-to-know) (Sherwood Callaway, 11X, 57:07)
- [Evaluating AI Agents: Why It Matters and How We Do It](https://mlopstalks.com/talks/evaluating-ai-agents-why-it-matters-and-how-we-do-it) (Annie Condon & Jeff Groom, Acre Security, 13:27)
- [Sub-Agent Architectures: What You Can Leverage](https://mlopstalks.com/talks/sub-agent-architectures-what-you-can-leverage) (Sidd Seethepalli, Vellum, 14:48)
- [Everything Hard About Building AI Agents Today](https://mlopstalks.com/talks/everything-hard-about-building-ai-agents-today) (Shreya Shankar & Willem Pienaar, Cleric, 47:03)