# GenAI Traffic: Why API Infrastructure Must Evolve... Again

Erica Hughberg, Tetrate | MLOps Podcast | Episode 296 | 1:06:25
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=PblnxZXCcIk
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/genai-traffic-why-api-infrastructure-must-evolve-again
Published: 2025-03-14
Tags: latency, model-serving, security

## TL;DR
- GenAI traffic is slower, larger, less predictable, and more open-ended than the microservice traffic that shaped current API infrastructure.
- AI gateways need to inspect requests and responses, support token-based usage limits, and handle model routing without giving up the connection efficiency of event-driven proxies.
- Many gateway problems appearing in GenAI were already visible in financial analysis APIs, so the infrastructure changes can help workloads beyond LLMs.

## Summary
Erica Hughberg traces the evolution of API infrastructure from the C10K problem through event-driven proxies, monolith-to-microservices migrations, Kubernetes scheduling, and Envoy. Each shift addressed a limitation of the previous model. GenAI creates another mismatch because requests and responses are larger, connections last longer, responses stream, and processing time is unpredictable. Gateways may need to inspect bodies for security, model selection, and response controls, even though payload limits and streaming make that difficult. Erica explains how Envoy AI Gateway extends Envoy Gateway with a control plane and external processing so teams can combine efficient connection handling with smarter logic such as model selection and request transformation. She also discusses token-based quotas and metrics such as response tokens per second. Her experience with large, variable financial-analysis responses led her to see GenAI gateway problems as older infrastructure problems arriving at a larger scale.

## Key ideas
### The C10K problem changed proxies from one thread per connection to event-driven handling
[02:17](https://www.youtube.com/watch?v=PblnxZXCcIk&t=137s)
Erica begins with the growth of the internet in the early 2000s, when more devices and interactive websites created many concurrent connections. She compares a thread-based proxy to a waiter who stays at one table while food cooks. The C10K problem described the difficulty of handling 10,000 concurrent connections, and the problem kept growing with internet use. Event-driven proxies changed the model: a waiter could enter an order into a system, serve other tables, and let another worker deliver the result. Connections could share threads while backend work continued.

### Microservices made applications more efficient while creating a routing problem
[07:02](https://www.youtube.com/watch?v=PblnxZXCcIk&t=422s)
Around the early 2010s, and especially from 2015, teams began breaking large monoliths into smaller services. Erica uses a box filled with different software features, then compares extracting one teddy bear into its own smaller box. Instead of cloning an entire application to scale one feature, teams could scale a smaller service with less wasted memory and compute. The change made services easier to scale, but their locations changed as they moved between machines. Proxies had to discover destinations dynamically rather than rely on static configuration.

### GenAI breaks assumptions built around fast and lightweight microservice calls
[12:21](https://www.youtube.com/watch?v=PblnxZXCcIk&t=741s)
Microservice networking was optimized for small requests, fast responses, and processes that responded in single-digit milliseconds. Erica says an LLM service can be 10 to 100 times slower even at its fastest when measuring the process itself rather than network latency. GenAI responses also tend to stream, which makes time to first byte important. The workloads can be much larger, and future systems may involve image and video generation. That creates pressure for infrastructure that can handle long-lived connections and heavier data instead of assuming every API call is short and predictable.

### Gateway inspection becomes harder when requests and responses are large and streamed
[18:20](https://www.youtube.com/watch?v=PblnxZXCcIk&t=1100s)
Many gateways impose request and response size limits, especially when they inspect bodies for security. GenAI systems may need to examine incoming content to block malicious inputs or prevent sensitive information from leaving. An application developer might also send a request to a general API while a lightweight model selects the most appropriate model or provider. That requires access to the request body, which may be unpredictable in size. Responses create another problem because streamed output may need inspection before it reaches the user, unlike a controlled Netflix stream that can pass through without content checks.

### Envoy AI Gateway combines event-driven traffic handling with external smart logic
[25:09](https://www.youtube.com/watch?v=PblnxZXCcIk&t=1509s)
Erica says Python gateways are attractive because machine learning developers know Python and can write logic such as automatic model selection. At large connection volumes, however, Python's interpreted, threaded execution can run into the single-waiter problem. Envoy Proxy already handles many connections through an event-driven architecture. Envoy AI Gateway adds extensions and external processing so smart logic can run when needed without making the proxy itself handle every connection through that constrained model. A separate Python service can scale independently, which avoids blocking unrelated requests at the proxy.

### AI gateway infrastructure needs both a control plane and provider-facing request handling
[37:17](https://www.youtube.com/watch?v=PblnxZXCcIk&t=2237s)
Erica explains that Envoy Proxy can be difficult to configure, especially across a fleet that scales up and down. Envoy AI Gateway extends the Gateway control plane so configuration can be propagated efficiently across Envoy proxies. It also adds an external process for GenAI-specific work, including request transformation. One example is a unified API that lets application developers use different model providers without learning every provider's interface. Installing Envoy AI Gateway brings in Envoy Gateway, Envoy Proxy, the control-plane extensions, and the external process on Kubernetes.

### GenAI makes usage limits and performance metrics depend on tokens and streaming progress
[50:51](https://www.youtube.com/watch?v=PblnxZXCcIk&t=3051s)
Traditional rate limits count requests, but a few large LLM requests can consume far more capacity than many small ones. Erica describes token quotas as a better measure for some GenAI workloads. Envoy Proxy had to support usage measurements based on another data point, such as tokens. Performance measurement also changes when responses stream. Response tokens per second can show whether an LLM is making progress, while byte rate can provide a similar signal for large financial-analysis responses. These measures help distinguish a slow response from one that is actively producing output.

### The gateway problems exposed by GenAI also apply to older variable-size APIs
[48:07](https://www.youtube.com/watch?v=PblnxZXCcIk&t=2887s)
Erica connects current LLM gateway challenges to her earlier work with financial-analysis APIs. The same endpoint could receive a small portfolio or a multinational portfolio with 10,000 holdings, producing very different processing times and response sizes. Some responses reached the 10 megabyte limits found in many gateways. She had been told that the API design was the problem. GenAI made the same pattern more widely recognized: unpredictable payloads and response times are infrastructure concerns as well as application concerns. Dynamic usage limits and streaming-aware measurements can therefore help financial APIs and other workloads too.

## Notable quotes
- Erica Hughberg: "We went from monolith big box with lots of stuff shoved into it to lots of smaller boxes with just one or a few items in each box that could scale together." (09:42)
- Erica Hughberg: "We now optimized how we think about networking in this microservices era to be fast, the process itself should respond within single digit milliseconds." (13:21)
- Erica Hughberg: "The problem comes back into the foundations of the Python language." (27:15)
- Erica Hughberg: "The good news is the concurrent connection problem has been solved." (32:54)
- Erica Hughberg: "I was like, and now everyone seems to be on board with that this is a real challenge." (50:30)

## Tools & references mentioned
- Tetrate
- MLOps Community
- Envoy Proxy
- Envoy Gateway
- Envoy AI Gateway
- Kubernetes
- Apple Intelligence
- Netflix
- Instagram
- Python
- Rust
- Streamlit
- React
- Vercel
- Next.js
- OpenAI
- Envoy Gateway 1.3

## Who should watch
- You are building an LLM or GenAI API and are running into gateway timeouts, payload limits, streaming complications, or unpredictable response times.
- Your team is considering a Python gateway and needs to understand where connection volume can become a scaling constraint.
- You maintain platform infrastructure for variable-size APIs and want to compare request-count limits with token or byte-based usage measures.

## Related talks

- [The Hidden Infrastructure Behind Every AI Agent](https://mlopstalks.com/talks/the-hidden-infrastructure-behind-every-ai-agent) (Erica Hughberg, Tetrate, 16:16)
- [Designing for Forward Compatibility in Gen AI](https://mlopstalks.com/talks/designing-for-forward-compatibility-in-gen-ai) (Rohit Agarwal, Portkey.ai, 1:00:18)
- [Kubernetes, AI Gateways, and the Future of MLOps](https://mlopstalks.com/talks/kubernetes-ai-gateways-and-the-future-of-mlops) (Alexa Griffith, Bloomberg, 50:12)
- [The Future of User Interfaces: AI is Changing Everything](https://mlopstalks.com/talks/the-future-of-user-interfaces-ai-is-changing-everything) (, 27:33)
- [Speed and Scale: How Today's AI Datacenters Are Operating Through Hypergrowth](https://mlopstalks.com/talks/speed-and-scale-how-todays-ai-datacenters-are-operating-through-hypergrowth) (Kris Beevers, NetBox Labs, 1:07:17)