# AI Agent Development Tradeoffs You NEED to Know

Sherwood Callaway, 11X | MLOps Community | 57:07
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=sni_-awZTgw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/ai-agent-development-tradeoffs-you-need-to-know
Published: 2025-07-22
Tags: agents, evals, observability, voice

## TL;DR
- AI agents automate complex, ambiguous work by looping through model reasoning, tool calls, observations, and further actions.
- LangGraph gives 11X more control and reliability than a simple agent loop, but its structure can become obsolete when better models arrive.
- Hosted agent platforms reduce infrastructure work while creating limits around scaling, deployment pipelines, observability, and vendor lock-in.

## Summary
Sherwood Callaway explains how 11X builds Alice, an AI sales development representative, and Julian, a voice agent for inbound sales. He compares text and voice agents, arguing that voice systems are harder because they must interpret tone, parallel cues, and latency-sensitive interaction. Alice uses agents for audience sourcing, lead research, outreach sequencing, and message writing. Sherwood describes the tradeoff between giving users control and letting the agent operate on autopilot. He then explains why 11X chose LangGraph and LangGraph Cloud, including the benefits of hosted queues and scaling infrastructure, along with problems involving deployment limits, traffic spikes, observability, and platform coupling. He expects agents to move toward simpler structures as models improve, although current model limits still make graphs useful. The discussion ends with monitoring and evaluation. 11X uses metrics, logs, traces, production evaluations, and Arize to check whether generated messages contain unsupported claims.

## Key ideas
### Agents automate work that was previously too ambiguous for software
[05:09](https://www.youtube.com/watch?v=sni_-awZTgw&t=309s)
Sherwood defines an agent as software that can automate complex and ambiguous tasks that could not previously be automated. Agents use tools, sometimes memory, and usually an agent framework. At the center is a loop: a language model plans what to do, calls a tool, observes the result, and decides whether to continue or finish. In Alice, that loop can source prospects, research individual leads, choose an outreach sequence, and write each message. The system decides when to contact someone and what product claims or personal details to include.

### Voice agents have extra difficulty because meaning depends on tone and timing
[03:01](https://www.youtube.com/watch?v=sni_-awZTgw&t=181s)
Sherwood contrasts his work on Alice with his earlier company, Opkit, which built phone-calling software for healthcare providers. Healthcare operations often required calls to insurance companies for eligibility checks, prior authorizations, and claims follow-up. Voice agents have to interpret inflections and parallel cues that are absent from typed text. They also have to do this in real time, with latency constraints. Sherwood says the engineering problems at 11X resemble those Opkit faced, although 11X can build on more mature technology and more sophisticated systems.

### Alice balances user control against letting the agent make better decisions
[08:41](https://www.youtube.com/watch?v=sni_-awZTgw&t=521s)
Alice creates campaigns by sourcing an audience, researching leads, choosing a message sequence, and generating the content of each message. Sales users may want to impose a particular structure, case study, or target segment, even when 11X believes Alice would perform better with more freedom. Sherwood describes the product problem as a balance between control and results. The company believes Alice does its best work when users put it on autopilot, while still allowing customers to guide brand voice and other parts of the output.

### Graphs provide reliability while models still have limits
[10:21](https://www.youtube.com/watch?v=sni_-awZTgw&t=621s)
Sherwood says LangGraph's graph structure is flexible enough to get an agent into production. A simpler agent can use one node and repeatedly follow a reason, act, and observe loop. That structure becomes more attractive as models improve. Current models still make mistakes, cost money, and produce inconsistent results, so 11X adds graph structure around steps that need predictable performance. A graph can make a campaign stage work consistently, which matters when the product promises customers that the system will perform reliably.

### Agent architecture can become obsolete before its extra structure pays off
[15:11](https://www.youtube.com/watch?v=sni_-awZTgw&t=911s)
Sherwood worries about premature optimization in agent graphs. Teams can add nodes and loops to improve one part of a workflow, then repeat that process as they add capabilities. A later model may achieve the same result with a much simpler design. The accumulated graph becomes difficult to change and can prevent the agent from doing other things. 11X therefore has to decide whether an investment in scaffolding will pay off before a new model makes it unnecessary. Sherwood is candid that this decision is still based partly on practical experimentation and intuition.

### Hosted agent infrastructure trades speed for control
[20:28](https://www.youtube.com/watch?v=sni_-awZTgw&t=1228s)
11X uses LangGraph Cloud to run its agents as Node processes in containers behind an API and queue. A request creates an agent run, the queue holds it, and a worker starts the process with the initial state. This prevents requests from being lost during peak traffic, although they may take longer to process. The hosted service saved 11X from building and operating substantial infrastructure while the company was rebuilding Alice under time pressure. The costs included deployment complexity, limits on individual deployments, connection timeouts during spikes, and less control over supporting processes.

### Observability is split across overlapping tools and walled gardens
[30:30](https://www.youtube.com/watch?v=sni_-awZTgw&t=1830s)
11X sends general logs, traces, and metrics to Datadog, while LangSmith provides visibility into agents running on LangGraph Cloud. Sherwood prefers telemetry in one place because developers should not have to search several systems during an incident. Moving agent data from LangSmith into Datadog has been difficult. A hosted cloud also limits options such as running a sidecar for metrics and traces, which would be easier on AWS, GCP, or Azure. Sherwood sees a growing overlap among agent hosting, observability, and cloud infrastructure products.

### Production evals check whether generated messages stay grounded in research
[45:28](https://www.youtube.com/watch?v=sni_-awZTgw&t=2728s)
11X evaluates a sample of message-writer traces with an LLM judge. The evaluation compares the research report and demographic information used as inputs with the generated email. It checks whether the message contains claims that do not match the research, including hallucinated facts about a prospect or company. Sherwood says the team samples about 1% because running an LLM evaluation across all traces would be expensive. The resulting score acts as a signal for whether a prompt change has made messages more or less reliable. Arize is the current tool used for this evaluation experiment.

## Notable quotes
- Sherwood Callaway: "I think that agents the characteristics that they have are they can automate complex and ambiguous tasks, tasks that really were not you could not automate previously." (05:09)
- Sherwood Callaway: "As models get better all agents start to collapse on this simpler form factor." (11:21)
- Sherwood Callaway: "We don't have it down to a science and it's a lot of vibe decision-making there." (19:36)
- Sherwood Callaway: "There's no free lunch." (27:19)
- Sherwood Callaway: "Agents are just like software. They're just regular software programs." (43:10)

## Tools & references mentioned
- 11X
- Alice
- Julian
- Opkit
- YC Summer 2021
- LangGraph
- LangGraph Cloud
- LangGraph Platform
- LangChain
- LangSmith
- OpenAI Agent SDK
- OpenAI
- Anthropic
- Amazon Bedrock
- AWS
- Datadog
- Arize
- Phoenix
- OpenTelemetry
- MCP
- Replit Agent
- Cursor
- GitHub Copilot
- Graphite
- Kubernetes
- PostgreSQL
- LogRocket
- PostHog

## Who should watch
- You are building a text or voice agent and need to decide how much workflow structure to add before models improve.
- Your agent runs in production, but its logs, traces, evaluations, and application telemetry are spread across different systems.
- You are choosing between hosting agent infrastructure yourself and using a managed platform with queues, scaling, and deployment support.

## Related talks

- [The Hidden Bottlenecks Slowing Down AI Agents](https://mlopstalks.com/talks/the-hidden-bottlenecks-slowing-down-ai-agents) (Paul van der Boor & Bruce Martens, Prosus Group, 48:00)
- [Architecting Modern AI Systems: Platforms, Agents, and Integration](https://mlopstalks.com/talks/architecting-modern-ai-systems-platforms-agents-and-integration) (Allen Roush, BuzzHPC & Frédéric Bénard, Mila & Shuo Wang, Bell Canada, 57:00)
- [Everything Hard About Building AI Agents Today](https://mlopstalks.com/talks/everything-hard-about-building-ai-agents-today) (Shreya Shankar & Willem Pienaar, Cleric, 47:03)
- [Sub-Agent Architectures: What You Can Leverage](https://mlopstalks.com/talks/sub-agent-architectures-what-you-can-leverage) (Sidd Seethepalli, Vellum, 14:48)
- [The Truth About AI Agents](https://mlopstalks.com/talks/the-truth-about-ai-agents) (Silen Naihin, AutoGPT, 31:40)