# How AgentOps Enables Observability

Nehil Jain, Stealth AI Startup & Adam Becker, MLOps Community & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Binoy Perera, MLOps Community | MLOps Reading Group | 49:48
Hosted by Adam Becker

Source: https://www.youtube.com/watch?v=seSIOfBLh6w
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-agentops-enables-observability
Published: 2024-12-27
Tags: agents, evals, observability, tracing

## TL;DR
- The paper groups agentic systems around perception, planning and reasoning, action, and adaptation.
- AgentOps is difficult because agents make hard-to-control decisions, execute multistep work, and produce complex inputs and outputs.
- Tracing, evaluation, feedback, and monitoring need to operate at several levels, from sessions and traces down to individual spans and agent steps.

## Summary
The session discusses the paper A Taxonomy of AgentOps for Enabling Observability of Foundation Model-based Agents. Nehil Jain introduces agents through four capabilities: perception, planning and reasoning, action through tools, and adaptation through examples or fine-tuning. The paper's authors describe AgentOps artifacts for creating agents, managing prompts and context, executing plans, evaluating results, and monitoring behavior. Adam Becker walks through the paper's taxonomy, including agent definitions, retrieval, prompt registries, guardrails, planning, memory, workflows, evaluation, and feedback. The group spends substantial time on traces and spans. A trace covers one request or task, while spans capture individual operations such as retrieval, model calls, tool use, or sub-agent work. The discussion also covers the limits of current evaluation methods, the difference between planning and workflows, and uncertainty around guardrails. The presenters are candid that the field has not settled on a standard vocabulary or a single observability approach.

## Key ideas
### Agents combine perception, planning, action, and adaptation
[02:04](https://www.youtube.com/watch?v=seSIOfBLh6w&t=124s)
Nehil Jain describes an agentic system through four capabilities. It can perceive inputs from different modalities, plan and reason about how to reach a goal, take actions through tools, and adapt from examples or feedback. Planning means breaking a goal into subtasks without hard-coding every decision. Action usually happens through tool calling, where the model receives instructions for interacting with an external system. Adaptation can involve few-shot prompting or fine-tuning based on inputs and feedback.

### Agent decisions are difficult to control
[03:23](https://www.youtube.com/watch?v=seSIOfBLh6w&t=203s)
Nehil Jain says agents are hard to operate because their decisions are difficult to control and language models remain largely black boxes. In his practical experience, a system with too many decisions often performs much worse than a human in the same situation. Teams may need many iterations before an agent behaves acceptably, and the cost of making a production-ready system can become high because the model may still choose the wrong action in a given scenario.

### Multistep execution makes observability harder
[04:23](https://www.youtube.com/watch?v=seSIOfBLh6w&t=263s)
Agents often plan, act, inspect feedback, and repeat across multiple subtasks. Nehil Jain calls this multihop behavior. Traditional machine learning systems usually have a defined input structure and a defined output structure. Agent systems do not fit that pattern as neatly, which makes it harder to observe, monitor, and track every part of an execution. The paper's taxonomy breaks the system into components so teams can decide what to record at each stage.

### The field is still assembling its AgentOps vocabulary
[05:56](https://www.youtube.com/watch?v=seSIOfBLh6w&t=356s)
The paper draws on GitHub projects and market maps to examine how people are building agent observability and monitoring systems. Nehil Jain's view is that there is no settled gold standard or single source of truth yet. Practitioners and researchers are still combining ideas from DevOps and MLOps, such as traces and observability, with the needs of agent systems. He expects the terminology and tooling to become more consistent over time.

### Agent creation includes prompts, tools, context, and roles
[10:32](https://www.youtube.com/watch?v=seSIOfBLh6w&t=632s)
The taxonomy describes the artifacts used to create an agent. These include multimodal inputs, toolkits for actions, retrieval or vector databases for grounding, fine-tuning, and prompt management. Prompts can be versioned and tested against different inputs. The discussion also covers agent identifiers, goals, model configuration, helper functions, and role types such as workers and coordinators. The group connects roles with possible permissions and coordination patterns, while acknowledging that these ideas are still developing.

### Evaluation should inspect both outcomes and intermediate steps
[12:08](https://www.youtube.com/watch?v=seSIOfBLh6w&t=728s)
The presenters distinguish final-result evaluation from step-level and trajectory evaluation. For a task such as booking a calendar meeting after reading emails, a team can check the final invitation, then inspect whether the agent understood the input, extracted the right information, called the right tool, and set the right parameters. Trajectory evaluation asks whether the agent selected the right next step. Because agent outputs are broad and variable, the group says current evaluation often requires inspecting execution data and choosing assertions that fit the input.

### Planning, memory, and workflows divide agent execution
[35:32](https://www.youtube.com/watch?v=seSIOfBLh6w&t=2132s)
Adam Becker describes planning as producing a set or queue of tasks. Possible approaches include task decomposition, generating and comparing several plans, using an external planner, and reflecting on previous results. Memory can hold recent interactions and intermediate outcomes in the short term, while long-term memory can include retrieved documents, knowledge bases, and past executions. The presenters are less certain about the paper's definition of workflows. They discuss workflows as human-designed relationships between components, while planning may create steps dynamically.

### Traces and spans provide the operational structure
[45:02](https://www.youtube.com/watch?v=seSIOfBLh6w&t=2702s)
A trace covers the complete path of one request or task through the system, from the user's query through retrieval, preprocessing, model calls, tool use, and the final response. A span is one operation within that trace, such as fetching data, calling a sub-agent, or processing a response. Spans can be nested, so a larger task can contain smaller operations. Sessions group related traces. This structure lets teams inspect latency, cost, errors, token use, and quality at session, trace, or span level.

## Notable quotes
- Nehil Jain: "If a system does four things, which is perception, planning and reasoning, actions, and adaptation, that is what we are calling agentic AI in today's world." (02:04)
- Nehil Jain: "It is very hard to control what it does in given scenarios." (04:03)
- Adam Becker: "The trace is all of these together." (46:44)
- Adam Becker: "The span is a single unit of work or operation within the trace." (47:04)
- Valdimar Eggertsson: "I do feel like it is starting to be important to differentiate between whatever the user put in and whatever we send to the model." (33:27)

## Tools & references mentioned
- A Taxonomy of AgentOps for Enabling Observability of Foundation Model-based Agents
- Andrew Ng
- GitHub
- DevOps
- MLOps
- OpenTelemetry
- Datadog
- Laminar
- LangSmith
- Langfuse
- LangGraph
- RAG
- Chain of Thought
- Tree of Thought
- Microsoft Azure

## Who should watch
- You are building an agent with several model calls, tools, or sub-agents and need a practical way to think about what to record.
- Your current evaluations only check the final answer, while failures can happen during planning, retrieval, tool selection, or execution.
- You are choosing an AgentOps or observability approach and want to understand the emerging vocabulary around sessions, traces, spans, prompts, memory, and workflows.

## Related talks

- [Sub-Agent Architectures: What You Can Leverage](https://mlopstalks.com/talks/sub-agent-architectures-what-you-can-leverage) (Sidd Seethepalli, Vellum, 14:48)
- [AI Agent Development Tradeoffs You NEED to Know](https://mlopstalks.com/talks/ai-agent-development-tradeoffs-you-need-to-know) (Sherwood Callaway, 11X, 57:07)
- [Architecting Modern AI Systems](https://mlopstalks.com/talks/architecting-modern-ai-systems) (, 56:58)
- [Omnigent: Composition, Control, and Collaboration for AI Agents](https://mlopstalks.com/talks/omnigent-composition-control-and-collaboration-for-ai-agents) (Denny Lee, Databricks, 58:17)
- [Decomposing the Agent Orchestration System: Lessons Learned](https://mlopstalks.com/talks/decomposing-the-agent-orchestration-system-lessons-learned) (Niels Bantilan, Union.ai, 30:13)