# Kubernetes, AI Gateways, and the Future of MLOps

Alexa Griffith, Bloomberg | MLOps Podcast | Episode 294 | 50:12
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=18aHn9E5Sb8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/kubernetes-ai-gateways-and-the-future-of-mlops
Published: 2025-03-07
Tags: build-vs-buy, developer-experience, model-serving, platform-teams

## TL;DR
- Alexa Griffith says abstraction helps ML engineers use inference platforms without needing to understand every Kubernetes concept underneath them.
- Envoy AI Gateway provides a unified interface for routing AI requests across on-premise and cloud model services, with token-aware controls for large language models.
- Alexa Griffith argues that technical teams should validate business value early, release a small usable version, and learn from real user behavior.

## Summary
Alexa Griffith describes her path from computational chemistry into software engineering, beginning with operational problems in Airflow and Kubernetes. She explains how those experiences led to open-source work, including KServe and Envoy AI Gateway. The conversation then focuses on abstractions for serving models. KNative, KServe, and Envoy AI Gateway hide much of the infrastructure detail while providing autoscaling, scale-to-zero, unified APIs, and routing across different model providers and environments. Alexa also discusses the practical differences between infrastructure engineers, ML engineers, and data scientists, especially their preferred tools and interfaces. Later, she turns to how teams choose work. Clear business value, user feedback, small initial releases, and direct observation of users help prevent teams from building platforms that nobody adopts. She is candid that technical quality alone is insufficient. Teams also need people who explain the work, find users, and connect it to the company's needs.

## Key ideas
### Airflow exposed the operational cost of running large workloads
[03:00](https://www.youtube.com/watch?v=18aHn9E5Sb8&t=180s)
Alexa Griffith's first infrastructure work involved running Airflow themselves, supporting data scientists, building DAGs, and creating a way to view job logs while tasks ran. Their Airflow scheduler crashed when it had to handle many workloads and fan-out tasks at once. Resource tuning may have contributed, but Alexa also wondered whether Airflow was the right tool for that workload. At the time, the available workflow tooling was less mature, so the team considered alternatives such as Kubeflow and Argo Workflows after already investing in Airflow.

### Different engineering backgrounds lead people toward different workflow tools
[23:50](https://www.youtube.com/watch?v=18aHn9E5Sb8&t=1430s)
Demetrios Brinkmann and Alexa Griffith discuss how people approach workflow tools based on their technical background. Infrastructure and DevOps engineers may prefer Argo Workflows, Go, or YAML configuration. Data engineers may feel more comfortable with Airflow, Dagster, or Prefect. ML engineers may gravitate toward Kubeflow, Metaflow, or ZenML. Alexa adds that similar differences appear in how teams interact with APIs. ML engineers often prefer Python, while infrastructure engineers may favor Go or YAML.

### Platform abstractions should hide infrastructure details while explaining actionable errors
[10:58](https://www.youtube.com/watch?v=18aHn9E5Sb8&t=658s)
Alexa Griffith says her team is working to abstract Kubernetes concepts away from platform users. An ML engineer may not need to understand pods, containers, or virtual services to deploy a model. The platform should explain what different YAML errors mean and tell the person who deployed the service what action to take. This does not remove the underlying infrastructure. It changes how users interact with it, so they can focus on deploying and using models rather than managing every Kubernetes resource.

### Envoy AI Gateway applies AI-specific traffic controls to model requests
[12:22](https://www.youtube.com/watch?v=18aHn9E5Sb8&t=742s)
Alexa Griffith describes Envoy AI Gateway as an open-source project from Bloomberg and Tetrate that builds on Envoy Gateway. Envoy handles traffic control, request routing, and observability. Large language models add different operational problems because they are larger and use tokens as a unit of work. A service may therefore need rate limiting based on token counts rather than only request counts. Envoy AI Gateway adds features for these model-serving patterns and provides a unified API for reaching models across different environments.

### KNative and KServe reduce the amount of configuration needed to run inference
[15:42](https://www.youtube.com/watch?v=18aHn9E5Sb8&t=942s)
Alexa explains that KNative provides a serverless-style abstraction on top of Kubernetes, including autoscaling and scale-to-zero. Scale-to-zero can free a GPU when a service has received no requests for a period of time, although some services cannot use it. KServe builds on these ideas for ML inference. A short YAML configuration can run supported models, while custom predictors allow teams to provide their own serving logic. KServe also gives different models a unified API and can infer details such as metrics ports from the model configuration.

### AI infrastructure must address model download time and scarce GPU capacity
[23:35](https://www.youtube.com/watch?v=18aHn9E5Sb8&t=1415s)
Alexa Griffith points to model caching as an active KServe problem. Large models can take a long time to download, and downloading the same model every time a pod starts creates a painful cold start. Caching models can reduce that repeated work. GPU utilization is another concern because GPU resources are limited. Large models can also require more than one node, which creates distributed serving problems. These issues make inference infrastructure different from the systems used for smaller or older models.

### Open-source contributions are easier to justify when they solve a real company problem
[26:45](https://www.youtube.com/watch?v=18aHn9E5Sb8&t=1605s)
Alexa's first open-source contribution came from fixing a small Celery issue connected to Airflow. Her broader advice is to work on open-source projects that the company actually needs and uses in production. That connection creates a clear business case for the time spent contributing. People who want to start outside work can look for good first issues and join community meetings, where maintainers can help with pull requests. Alexa credits supportive teams and mentors with giving her work that helped her grow.

### Teams should test business value before building a large platform
[30:46](https://www.youtube.com/watch?v=18aHn9E5Sb8&t=1846s)
Alexa Griffith says engineers should understand a project's requirements, expected users, cost, and purpose before committing substantial time. A request from one person may not justify a large feature. User and customer feedback can reveal whether anyone will use the result. She warns that building something nobody cares about wastes time and money, and it can leave a team trying to justify months of work. Directly sitting with a frequent user can reveal pain points that surveys and support chats miss.

### A small release and active communication matter as much as technical design
[33:17](https://www.youtube.com/watch?v=18aHn9E5Sb8&t=1997s)
Alexa repeats the engineering motto, "as simple as possible, as powerful as necessary." She recommends releasing an MVP, getting people to use it, and iterating from feedback rather than trying to solve the entire problem at the start. Demetrios Brinkmann adds an example of a technically impressive ML platform that received no adoption because its creators did not explain its value to data scientists. Alexa also describes keeping a quarterly personal record of her work and sharing it with her manager, which helps make contributions visible.

## Notable quotes
- Alexa Griffith: "I think there's a lot of work now and it's not exactly always easy but to abstract away Kubernetes and those main things." (10:58)
- Alexa Griffith: "The goal is that you don't have to have this whole team that knows everything about how to run AI models." (18:28)
- Alexa Griffith: "As simple as possible as powerful as necessary." (33:37)
- Alexa Griffith: "The worst thing you can do is make something that no one cares about." (31:22)
- Alexa Griffith: "You need to know what's not a good business value." (30:30)

## Tools & references mentioned
- Bloomberg
- Tetrate
- Airflow
- Kubernetes
- Kubeflow
- Argo Workflows
- Argo CD
- Dagster
- Prefect
- Metaflow
- ZenML
- KServe
- KNative
- Envoy
- Envoy Gateway
- Envoy AI Gateway
- Llama 3.1
- Celery
- Google Cloud
- SageMaker
- Vertex AI
- Bedrock
- OpenAI

## Who should watch
- You are building an internal platform for ML engineers and need to decide which Kubernetes details should remain visible to users.
- Your inference system has to route requests across cloud and on-premise models, or you need controls that account for token usage and GPU limits.
- Your team keeps building technically polished tools with weak adoption and needs a more direct way to test business value and learn from users.

## Editor's note

Alexa Griffith says platform teams should release a small usable version, watch real users, and avoid building a large platform that nobody adopts. ZenML lets ML engineers write workflows as Python steps while choosing the infrastructure underneath them through a configured stack, so a platform team can offer a simpler interface without rewriting the workflow for each environment.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Building an ML Platform: Insights, Community, and Advocacy](https://mlopstalks.com/talks/building-an-ml-platform-insights-community-and-advocacy) (Stephen Batifol, Wolt, 45:49)
- [DevOps, Security, and Observability in ML](https://mlopstalks.com/talks/devops-security-and-observability-in-ml) (Luke Marsden, MLOps Consulting, 32:46)
- [How to Leverage ML Tooling Ecosystem](https://mlopstalks.com/talks/how-to-leverage-ml-tooling-ecosystem) (Mariya Davydova, Neu.ro, 55:57)
- [Packaging MLOps Tech Neatly for Engineers and Non-engineers](https://mlopstalks.com/talks/packaging-mlops-tech-neatly-for-engineers-and-non-engineers) (Jukka Remes, Haaga-Helia University of Applied Sciences, 8wave AI, 55:31)
- [Building an ML Platform from Scratch: Live Coding Session - Part 2](https://mlopstalks.com/talks/building-an-ml-platform-from-scratch-live-coding-session-part-2) (Alon Gubkin, Aporia, 1:12:53)