Alexa Griffith says abstraction helps ML engineers use inference platforms without needing to understand every Kubernetes concept underneath them.
2
Envoy AI Gateway provides a unified interface for routing AI requests across on-premise and cloud model services, with token-aware controls for large language models.
3
Alexa Griffith argues that technical teams should validate business value early, release a small usable version, and learn from real user behavior.
Summary
Alexa Griffith describes her path from computational chemistry into software engineering, beginning with operational problems in Airflow and Kubernetes. She explains how those experiences led to open-source work, including KServe and Envoy AI Gateway. The conversation then focuses on abstractions for serving models. KNative, KServe, and Envoy AI Gateway hide much of the infrastructure detail while providing autoscaling, scale-to-zero, unified APIs, and routing across different model providers and environments. Alexa also discusses the practical differences between infrastructure engineers, ML engineers, and data scientists, especially their preferred tools and interfaces. Later, she turns to how teams choose work. Clear business value, user feedback, small initial releases, and direct observation of users help prevent teams from building platforms that nobody adopts. She is candid that technical quality alone is insufficient. Teams also need people who explain the work, find users, and connect it to the company's needs.
Airflow exposed the operational cost of running large workloads
Alexa Griffith's first infrastructure work involved running Airflow themselves, supporting data scientists, building DAGs, and creating a way to view job logs while tasks ran. Their Airflow scheduler crashed when it had to handle many workloads and fan-out tasks at once. Resource tuning may have contributed, but Alexa also wondered whether Airflow was the right tool for that workload. At the time, the available workflow tooling was less mature, so the team considered alternatives such as Kubeflow and Argo Workflows after already investing in Airflow.
Different engineering backgrounds lead people toward different workflow tools
Demetrios Brinkmann and Alexa Griffith discuss how people approach workflow tools based on their technical background. Infrastructure and DevOps engineers may prefer Argo Workflows, Go, or YAML configuration. Data engineers may feel more comfortable with Airflow, Dagster, or Prefect. ML engineers may gravitate toward Kubeflow, Metaflow, or ZenML. Alexa adds that similar differences appear in how teams interact with APIs. ML engineers often prefer Python, while infrastructure engineers may favor Go or YAML.
Platform abstractions should hide infrastructure details while explaining actionable errors
Alexa Griffith says her team is working to abstract Kubernetes concepts away from platform users. An ML engineer may not need to understand pods, containers, or virtual services to deploy a model. The platform should explain what different YAML errors mean and tell the person who deployed the service what action to take. This does not remove the underlying infrastructure. It changes how users interact with it, so they can focus on deploying and using models rather than managing every Kubernetes resource.
Envoy AI Gateway applies AI-specific traffic controls to model requests
Alexa Griffith describes Envoy AI Gateway as an open-source project from Bloomberg and Tetrate that builds on Envoy Gateway. Envoy handles traffic control, request routing, and observability. Large language models add different operational problems because they are larger and use tokens as a unit of work. A service may therefore need rate limiting based on token counts rather than only request counts. Envoy AI Gateway adds features for these model-serving patterns and provides a unified API for reaching models across different environments.
KNative and KServe reduce the amount of configuration needed to run inference
Alexa explains that KNative provides a serverless-style abstraction on top of Kubernetes, including autoscaling and scale-to-zero. Scale-to-zero can free a GPU when a service has received no requests for a period of time, although some services cannot use it. KServe builds on these ideas for ML inference. A short YAML configuration can run supported models, while custom predictors allow teams to provide their own serving logic. KServe also gives different models a unified API and can infer details such as metrics ports from the model configuration.
AI infrastructure must address model download time and scarce GPU capacity
Alexa Griffith points to model caching as an active KServe problem. Large models can take a long time to download, and downloading the same model every time a pod starts creates a painful cold start. Caching models can reduce that repeated work. GPU utilization is another concern because GPU resources are limited. Large models can also require more than one node, which creates distributed serving problems. These issues make inference infrastructure different from the systems used for smaller or older models.
Open-source contributions are easier to justify when they solve a real company problem
Alexa's first open-source contribution came from fixing a small Celery issue connected to Airflow. Her broader advice is to work on open-source projects that the company actually needs and uses in production. That connection creates a clear business case for the time spent contributing. People who want to start outside work can look for good first issues and join community meetings, where maintainers can help with pull requests. Alexa credits supportive teams and mentors with giving her work that helped her grow.
Teams should test business value before building a large platform
Alexa Griffith says engineers should understand a project's requirements, expected users, cost, and purpose before committing substantial time. A request from one person may not justify a large feature. User and customer feedback can reveal whether anyone will use the result. She warns that building something nobody cares about wastes time and money, and it can leave a team trying to justify months of work. Directly sitting with a frequent user can reveal pain points that surveys and support chats miss.
A small release and active communication matter as much as technical design
Alexa repeats the engineering motto, "as simple as possible, as powerful as necessary." She recommends releasing an MVP, getting people to use it, and iterating from feedback rather than trying to solve the entire problem at the start. Demetrios Brinkmann adds an example of a technically impressive ML platform that received no adoption because its creators did not explain its value to data scientists. Alexa also describes keeping a quarterly personal record of her work and sharing it with her manager, which helps make contributions visible.
