GenAI traffic is slower, larger, less predictable, and more open-ended than the microservice traffic that shaped current API infrastructure.
2
AI gateways need to inspect requests and responses, support token-based usage limits, and handle model routing without giving up the connection efficiency of event-driven proxies.
3
Many gateway problems appearing in GenAI were already visible in financial analysis APIs, so the infrastructure changes can help workloads beyond LLMs.
Summary
Erica Hughberg traces the evolution of API infrastructure from the C10K problem through event-driven proxies, monolith-to-microservices migrations, Kubernetes scheduling, and Envoy. Each shift addressed a limitation of the previous model. GenAI creates another mismatch because requests and responses are larger, connections last longer, responses stream, and processing time is unpredictable. Gateways may need to inspect bodies for security, model selection, and response controls, even though payload limits and streaming make that difficult. Erica explains how Envoy AI Gateway extends Envoy Gateway with a control plane and external processing so teams can combine efficient connection handling with smarter logic such as model selection and request transformation. She also discusses token-based quotas and metrics such as response tokens per second. Her experience with large, variable financial-analysis responses led her to see GenAI gateway problems as older infrastructure problems arriving at a larger scale.
The C10K problem changed proxies from one thread per connection to event-driven handling
Erica begins with the growth of the internet in the early 2000s, when more devices and interactive websites created many concurrent connections. She compares a thread-based proxy to a waiter who stays at one table while food cooks. The C10K problem described the difficulty of handling 10,000 concurrent connections, and the problem kept growing with internet use. Event-driven proxies changed the model: a waiter could enter an order into a system, serve other tables, and let another worker deliver the result. Connections could share threads while backend work continued.
Microservices made applications more efficient while creating a routing problem
Around the early 2010s, and especially from 2015, teams began breaking large monoliths into smaller services. Erica uses a box filled with different software features, then compares extracting one teddy bear into its own smaller box. Instead of cloning an entire application to scale one feature, teams could scale a smaller service with less wasted memory and compute. The change made services easier to scale, but their locations changed as they moved between machines. Proxies had to discover destinations dynamically rather than rely on static configuration.
GenAI breaks assumptions built around fast and lightweight microservice calls
Microservice networking was optimized for small requests, fast responses, and processes that responded in single-digit milliseconds. Erica says an LLM service can be 10 to 100 times slower even at its fastest when measuring the process itself rather than network latency. GenAI responses also tend to stream, which makes time to first byte important. The workloads can be much larger, and future systems may involve image and video generation. That creates pressure for infrastructure that can handle long-lived connections and heavier data instead of assuming every API call is short and predictable.
Gateway inspection becomes harder when requests and responses are large and streamed
Many gateways impose request and response size limits, especially when they inspect bodies for security. GenAI systems may need to examine incoming content to block malicious inputs or prevent sensitive information from leaving. An application developer might also send a request to a general API while a lightweight model selects the most appropriate model or provider. That requires access to the request body, which may be unpredictable in size. Responses create another problem because streamed output may need inspection before it reaches the user, unlike a controlled Netflix stream that can pass through without content checks.
Envoy AI Gateway combines event-driven traffic handling with external smart logic
Erica says Python gateways are attractive because machine learning developers know Python and can write logic such as automatic model selection. At large connection volumes, however, Python's interpreted, threaded execution can run into the single-waiter problem. Envoy Proxy already handles many connections through an event-driven architecture. Envoy AI Gateway adds extensions and external processing so smart logic can run when needed without making the proxy itself handle every connection through that constrained model. A separate Python service can scale independently, which avoids blocking unrelated requests at the proxy.
AI gateway infrastructure needs both a control plane and provider-facing request handling
Erica explains that Envoy Proxy can be difficult to configure, especially across a fleet that scales up and down. Envoy AI Gateway extends the Gateway control plane so configuration can be propagated efficiently across Envoy proxies. It also adds an external process for GenAI-specific work, including request transformation. One example is a unified API that lets application developers use different model providers without learning every provider's interface. Installing Envoy AI Gateway brings in Envoy Gateway, Envoy Proxy, the control-plane extensions, and the external process on Kubernetes.
GenAI makes usage limits and performance metrics depend on tokens and streaming progress
Traditional rate limits count requests, but a few large LLM requests can consume far more capacity than many small ones. Erica describes token quotas as a better measure for some GenAI workloads. Envoy Proxy had to support usage measurements based on another data point, such as tokens. Performance measurement also changes when responses stream. Response tokens per second can show whether an LLM is making progress, while byte rate can provide a similar signal for large financial-analysis responses. These measures help distinguish a slow response from one that is actively producing output.
The gateway problems exposed by GenAI also apply to older variable-size APIs
Erica connects current LLM gateway challenges to her earlier work with financial-analysis APIs. The same endpoint could receive a small portfolio or a multinational portfolio with 10,000 holdings, producing very different processing times and response sizes. Some responses reached the 10 megabyte limits found in many gateways. She had been told that the API design was the problem. GenAI made the same pattern more widely recognized: unpredictable payloads and response times are infrastructure concerns as well as application concerns. Dynamic usage limits and streaming-aware measurements can therefore help financial APIs and other workloads too.
"We went from monolith big box with lots of stuff shoved into it to lots of smaller boxes with just one or a few items in each box that could scale together."Erica Hughberg09:42
Who should watch
You are building an LLM or GenAI API and are running into gateway timeouts, payload limits, streaming complications, or unpredictable response times.
Your team is considering a Python gateway and needs to understand where connection volume can become a scaling constraint.
You maintain platform infrastructure for variable-size APIs and want to compare request-count limits with token or byte-based usage measures.
