# Logs Are All You Need: Rethinking Observability with AI Agents

Sherwood Callaway, Sazabi | MLOps Podcast | Episode 381 | 46:40
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=RSs0PDsULJM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/logs-are-all-you-need-rethinking-observability-with-ai-agents
Published: 2026-06-22
Tags: agents, evals, observability, security

## TL;DR
- Sazabi argues that an AI agent should answer questions about production telemetry through chat, instead of making engineers inspect dashboards and log-search interfaces.
- Sherwood Callaway believes logs can replace metrics and traces because they are simpler to instrument, while Sazabi can reconstruct the information those signals provide.
- Sazabi generates alerts from production logs, code, and other context, then recommends remediation without opening pull requests or merging code itself.

## Summary
Sherwood Callaway describes Sazabi as an AI-native observability platform for teams that ship quickly. Its interface is chat, including a Slackbot, because engineers can ask an agent what went wrong instead of inspecting telemetry directly. Sazabi focuses on logs and reconstructs metrics and traces from them, which removes the need to configure Prometheus, understand metric types, or propagate trace context. It also replaces static monitors with alerts generated by an agent that reads logs, code, and connected systems. The agent runs mainly through a Daytona sandbox with Bash access, while log queries go through a read-only SQL-like interface protected by a proxy and database row-level security. Sazabi stores memory in Git repositories and branches, allowing threads and parallel agents to share findings. Callaway is candid about evals: good context, models, harnesses, and sandboxes get an agent far, but evaluating Bash-driven side effects becomes difficult as the system grows. The company currently builds its eval tooling in-house.

## Key ideas
### Sazabi wants engineers to ask an agent what production is doing
[02:24](https://www.youtube.com/watch?v=RSs0PDsULJM&t=144s)
Sherwood Callaway says Sazabi does AI for observability rather than observability for AI. The product assumes engineers increasingly work with coding agents, background agents, and parallel changes, while traditional production debugging remains slow. Sazabi therefore hides direct telemetry access behind an agent. Engineers ask questions such as whether the system is up, what an error means, which customers were affected, or which commit caused the problem. The proposed user experience is chat, with a Slackbot as another way to ask for an explanation of production behavior.

### Logs can replace metrics and traces when an agent interprets them
[04:44](https://www.youtube.com/watch?v=RSs0PDsULJM&t=284s)
Callaway rejects the traditional three-pillar model of metrics, logs, and traces. He argues that logs are enough, especially when an agent can interpret them and reconstruct metrics and traces on the backend. This reduces instrumentation to console logging or print statements. Teams do not need to configure a Prometheus server, choose between counters, gauges, histograms, and rates, or propagate span context and trace IDs across a call stack. Demetrios Brinkmann describes this as being able to 'vibe observe'. Callaway says natural-language questions and simpler instrumentation should provide the benefits teams expected from all three telemetry types.

### Sazabi generates alerts from context instead of evaluating configured monitors
[07:17](https://www.youtube.com/watch?v=RSs0PDsULJM&t=437s)
Callaway says static monitors and thresholds are no longer useful to Sazabi. Instead of asking an AI system to rank or enrich an existing alert stream, Sazabi lets the agent decide when an alert matters. The agent can combine production logs, the codebase, and other connected context, then send a Slack notification describing a suspected problem, the relevant commit, and possible remediation steps. It may offer to start a Cursor cloud agent, but Sazabi stops short of generating code, opening pull requests, or merging changes. At the user's direction, it can create Linear tickets and remember recurring preferences.

### MCP and skills both create context-management problems
[10:51](https://www.youtube.com/watch?v=RSs0PDsULJM&t=651s)
The conversation compares MCP servers with local skill files. Callaway says MCP servers have drawn criticism because loading them can fill the agent's context, although tool search and progressive disclosure reduce that cost. A sandbox exposes a much larger set of programs, including Unix utilities and installed CLIs, yet models often already know that commands such as cat or ls may exist and can recover when one is unavailable. Demetrios Brinkmann worries that skills can also grow into skills bloat. Callaway agrees, while noting that an agent can inspect skill headers before reading full instructions, which gives skills a form of progressive disclosure.

### The agent uses a sandbox for investigation and read-only access for logs
[16:05](https://www.youtube.com/watch?v=RSs0PDsULJM&t=965s)
Sazabi's agent is built with Vercel AI SDK workflows and runs Bash commands inside a Daytona sandbox. The main exception is its log database query tool, which provides a SQL-like read-only interface. Queries pass through a proxy, and row-level security prevents the database user from updating or dropping tables. The sandbox can contain CLIs such as the AWS CLI, a cloned copy of the customer's source code, and files created during investigation. That lets the agent connect a log error to a specific file, line, commit, and author. Callaway says almost everything else happens through Bash in the sandbox.

### Git branches give agent threads shared, persistent memory
[19:11](https://www.youtube.com/watch?v=RSs0PDsULJM&t=1151s)
Sazabi stores memory as Markdown files in a prescribed folder structure, backed by a self-hosted Git repository. Each Sazabi project gets its own repository, while each conversation thread gets a branch. A sandbox pulls the branch when it starts and pushes changes back after each Bash command finishes. The state survives the sandbox and can include anything from a small note to a parsed collection of log findings. Other threads can merge memories from different branches. Parallel agents can also encounter merge conflicts when they update the same issue, and a background workflow resolves those conflicts.

### Parallel agents can share findings without sharing one context window
[29:20](https://www.youtube.com/watch?v=RSs0PDsULJM&t=1760s)
Each thread has one sandbox and one relevant branch, so Callaway does not see much use for worktrees in Sazabi's current design. The system already lets a main agent spawn sub-agents and background agents, with each sub-agent using its own sandbox and shared Git-backed memory. Sub-agents report findings to the main agent and commit them to memory. This means later work can benefit from collective discoveries without putting every finding into the main agent's context. Callaway also considers running many agents on the same request and merging their outputs, although Sazabi does not yet do that automatically.

### Evals become harder when the agent acts through Bash
[30:36](https://www.youtube.com/watch?v=RSs0PDsULJM&t=1836s)
Callaway says teams can get a capable agent without writing evals by improving context, the harness, the sandbox, model choice, and sub-agent behavior. Evals matter when the team wants to push beyond that baseline. Sazabi can test factual answers, response quality, root-cause explanations, alert decisions, and whether the agent exposes implementation details. Tool calls are easier to evaluate than Bash because a dedicated alert tool produces a binary event. With Bash, the agent might trigger the same result through curl, a script, or different parameters. Testing the side effect directly requires increasingly complete mock services and external dependencies, so Sazabi keeps its eval tooling home rolled.

### Security boundaries limit what the observability agent can do
[40:47](https://www.youtube.com/watch?v=RSs0PDsULJM&t=2447s)
Sazabi is designed as a read-only system with no unrestricted public internet access. Callaway says domains and IPs must be allowlisted, which reduces the risk of a prompt injection causing the agent to exfiltrate a customer's codebase. The company also evaluates whether the agent reveals too much about its own implementation. Customer security is a separate concern from protecting Sazabi's internal details. Callaway mentions certifications and compliance work, drawing on the team's experience building banking infrastructure at Brex. The sandbox model adds complexity, but its isolation and access controls are part of the product's security boundary.

## Notable quotes
- Sherwood Callaway: "The official answer is that we do AI for observability, not observability for AI." (02:24)
- Sherwood Callaway: "The best UX for observability is chat." (04:24)
- Sherwood Callaway: "We're just using AI to generate the alerts." (08:25)
- Sherwood Callaway: "Our memory is very similar to something like OpenClaw where it's all based in Markdown files and in a prescribed folder structure." (19:31)
- Sherwood Callaway: "Evals are hard." (30:36)

## Tools & references mentioned
- Sazabi
- Datadog
- Brex
- Arize
- LangSmith
- Phoenix
- Lens
- Cursor
- Linear
- Claude Code
- MCP
- Model Context Protocol
- Vercel AI SDK workflows
- Daytona
- Prometheus
- AWS CLI
- Postgres
- TurboRepo
- TypeScript
- Codex
- OpenClaw
- Jeppa
- LocalStack
- RDS
- EKS
- SOC 2 Type 2
- ISO 27001
- GDPR
- HIPAA

## Who should watch
- You are deciding whether an AI agent should sit between engineers and production telemetry instead of dashboards and alert rules.
- Your team runs coding agents in parallel and needs a way to investigate incidents with isolated environments and shared findings.
- You are building agent evals around CLI actions, external services, or side effects and want an honest account of why that gets difficult.

## Related talks

- [Everything Hard About Building AI Agents Today](https://mlopstalks.com/talks/everything-hard-about-building-ai-agents-today) (Shreya Shankar & Willem Pienaar, Cleric, 47:03)
- [From Spikes to Stories: AI-Augmented Troubleshooting in the Network Wild](https://mlopstalks.com/talks/from-spikes-to-stories-ai-augmented-troubleshooting-in-the-network-wild) (Shraddha Yeole, Cisco ThousandEyes, 11:49)
- [Sandboxing, Agent Harnesses, and Agent Teamwork](https://mlopstalks.com/talks/sandboxing-agent-harnesses-and-agent-teamwork) (Shahram Anver, Cleric, 1:19:54)
- [How AgentOps Enables Observability](https://mlopstalks.com/talks/how-agentops-enables-observability) (Nehil Jain, Stealth AI Startup & Adam Becker, MLOps Community & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Binoy Perera, MLOps Community, 49:48)
- [Evaluating AI Agents: Why It Matters and How We Do It](https://mlopstalks.com/talks/evaluating-ai-agents-why-it-matters-and-how-we-do-it) (Annie Condon & Jeff Groom, Acre Security, 13:27)