# The Hidden Infrastructure Behind Every AI Agent

Erica Hughberg, Tetrate | MLOps Community | 16:16
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=QcVLVakQBUc
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-hidden-infrastructure-behind-every-ai-agent
Published: 2025-08-27
Tags: agents, model-serving, observability, open-source

## TL;DR
- AI agents create chains of network requests to models, APIs, tools, databases, and embedding services, so their infrastructure must route, authorize, track, and optimize that traffic.
- AI gateways need dynamic routing, access control, cost-aware load balancing, model-aware failover, and observability for multi-model systems.
- Envoy AI Gateway builds on Envoy Proxy to handle the operational complexity of agent traffic so developers can focus on building agentic systems.

## Summary
Erica Hughberg explains why AI agents need infrastructure beyond a conventional API gateway. A simple instruction can trigger calls to OpenAI, Bedrock, Anthropic, Gemini, internal tools, databases, and embedding services. Each request needs routing, authorization, tracking, and cost control. The traffic is dynamic because providers can slow down or go offline, while models have different prices, capabilities, and access requirements. Hughberg presents Envoy AI Gateway as an open-source effort built on the Envoy Proxy foundation. It addresses dynamic routing, provider access, cost-aware load balancing, failover that preserves the user experience, observability, and connectivity with MCP and A2A. Her main practical point is that a gateway takes care of the messy infrastructure work while developers build and ship agentic systems. She also mentions hosted and self-managed options from Tetrate, along with reference architecture resources and an MLOps Community podcast for deeper discussion.

## Key ideas
### An agent's apparent autonomy depends on infrastructure handling every action
[00:35](https://www.youtube.com/watch?v=QcVLVakQBUc&t=35s)
Hughberg compares an AI agent with James Bond, who has a team assigning missions, supplying equipment, and keeping operations running. An agent may look autonomous, but every action creates work behind the scenes. It queries a model, calls an API, or fetches a tool, and something has to handle that traffic. The gateway takes the role of the infrastructure team behind the agent. The agent's success therefore depends on the systems that route and control its requests, not only on the model's intelligence.

### A simple agent task can create a large chain of requests and failure points
[03:10](https://www.youtube.com/watch?v=QcVLVakQBUc&t=190s)
AI agents are talkative because each decision can start more activity. A request such as "summarize it, book it, look it up" may lead to calls to OpenAI, Bedrock, Anthropic, Gemini, internal tools, databases, and embedding services. Those calls move across different parts of an organization's systems. Each one is network traffic that needs routing, authorization, tracking, and optimization. The resulting chain creates more places where a request can fail and more opportunities for security problems.

### AI traffic has different operating requirements from traditional API traffic
[04:34](https://www.youtube.com/watch?v=QcVLVakQBUc&t=274s)
Hughberg says traditional API traffic was generally light, fast, and expected to behave deterministically. Agent requests are more expensive and less predictable. The system must account for changing model and provider performance, the cost of different targets, the sensitivity of the information being sent, and the importance of the missions agents are handling. This makes resiliency more important. A gateway has to help the agent complete work safely, quickly, reliably, and affordably.

### Dynamic routing and access control must account for models and providers
[07:37](https://www.youtube.com/watch?v=QcVLVakQBUc&t=457s)
Providers can slow down, go offline, or experience latency spikes, so the gateway needs to reroute traffic. Access control must answer what an agent can access, on whose behalf it acts, and how it authenticates with each upstream provider. Different providers may require tokens, OAuth, or other access methods. The gateway therefore needs rules that understand both the agent's permissions and the provider's connection requirements.

### Load balancing and failover need to consider cost and the request's purpose
[08:27](https://www.youtube.com/watch?v=QcVLVakQBUc&t=507s)
Different models and providers have different prices, contracts, and operating arrangements. An organization may also host models itself. Hughberg says the gateway needs to decide which target makes sense for a particular request. Failover also needs care. Switching from one model or provider to an arbitrary alternative may damage the experience, so the replacement needs to make sense for the work being done.

### Observability gives teams a way to diagnose provider and model behavior
[08:51](https://www.youtube.com/watch?v=QcVLVakQBUc&t=531s)
The gateway should show how providers and models are performing. Hughberg reduces the point to a short rule: "If you can't see it, you can't fix it." Observability covers the behavior of the upstream systems as well as the traffic moving through the gateway. Without that visibility, teams cannot tell where latency, availability, or other operational problems originate.

### Envoy AI Gateway extends an established proxy foundation for agent traffic
[09:45](https://www.youtube.com/watch?v=QcVLVakQBUc&t=585s)
Hughberg describes Envoy AI Gateway as an open-source project built on the Envoy Proxy foundation. The project addresses usage control, usage observation, authorization, failover, and connections to different providers. She also mentions connectivity involving MCP and A2A. The approach keeps using an existing proxy foundation while adding capabilities needed for AI traffic rather than building an entirely separate network layer.

### A gateway removes infrastructure work so developers can focus on agent systems
[11:00](https://www.youtube.com/watch?v=QcVLVakQBUc&t=660s)
When integrations pile up, debugging becomes painful and development slows down. Hughberg says an AI gateway handles traffic routing, authentication, observability, and cost efficiency so developers can focus on building agents and agentic systems. She frames this as a platform engineering concern that supports product engineering. She also points to self-managed reference architecture and Tetrate's hosted Agent Router Service for teams that want to get started without installing and operating everything themselves.

## Notable quotes
- Erica Hughberg: "They may look autonomous on the surface, but every time they take an action, query a model, hit an API, fetch a tool, someone, well, something has to handle that traffic." (01:21)
- Erica Hughberg: "If you can't see it, you can't fix it." (08:51)
- Erica Hughberg: "The gateway helps handle the messy stuff, the traffic routing, the auth, the observability and help make it cost efficient." (11:32)
- Erica Hughberg: "Questions is one of the best ways to start contributing in open source." (15:35)

## Tools & references mentioned
- Envoy Proxy
- Envoy AI Gateway
- Envoy Gateway
- Istio
- Kubernetes Gateway API
- OpenAI
- Bedrock
- Anthropic
- Gemini
- Model Context Protocol
- MCP
- A2A
- Tetrate
- Agent Operations Director
- Agent Router Service
- MLOps Community
- Demetrios Brinkmann

## Who should watch
- You are building agents that call several models, tools, databases, or embedding services and need to understand where that traffic should be controlled.
- Your team is deciding whether a conventional API gateway is enough for AI workloads with changing latency, provider availability, and model costs.
- You work on platform or network infrastructure and want a concrete overview of what Envoy AI Gateway is intended to handle.

## Related talks

- [GenAI Traffic: Why API Infrastructure Must Evolve... Again](https://mlopstalks.com/talks/genai-traffic-why-api-infrastructure-must-evolve-again) (Erica Hughberg, Tetrate, 1:06:25)
- [AI Agent Development Tradeoffs You NEED to Know](https://mlopstalks.com/talks/ai-agent-development-tradeoffs-you-need-to-know) (Sherwood Callaway, 11X, 57:07)
- [The Future of Compute: How AI Agents Are Reshaping Infrastructure](https://mlopstalks.com/talks/the-future-of-compute-how-ai-agents-are-reshaping-infrastructure) (Diego Oppenheimer, Guardrails AI, 38:34)
- [Architecting Modern AI Systems: Platforms, Agents, and Integration](https://mlopstalks.com/talks/architecting-modern-ai-systems-platforms-agents-and-integration) (Allen Roush, BuzzHPC & Frédéric Bénard, Mila & Shuo Wang, Bell Canada, 57:00)
- [Designing for Forward Compatibility in Gen AI](https://mlopstalks.com/talks/designing-for-forward-compatibility-in-gen-ai) (Rohit Agarwal, Portkey.ai, 1:00:18)