# 2 tools = 90% operational ML

Michael Del Balso, Tecton & Willem Pienaar, Feast & David Aronchick, Kubeflow | MLOps Meetup | Episode 50 | 56:39
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=iWMQxCGFdU0
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/2-tools-90-operational-ml
Published: 2021-02-05
Tags: data-engineering, feature-engineering, feature-stores, orchestration, platform-teams

## TL;DR
- Kubeflow is a Kubernetes-native way to deploy and connect interchangeable ML components, rather than an end-to-end replacement for every tool.
- Feature stores address the difficult data work between development and production, including feature definitions, validation, schemas, and consistent access to training and serving data.
- The panel argues for interoperable tools with shared contracts, because a single monolithic ML platform becomes difficult to adapt when use cases change.

## Summary
The panel discusses why operational ML needs infrastructure for both models and data. David Aronchick describes Kubeflow as a Kubernetes-native deployment and pipeline system that lets teams combine components such as notebooks, training systems, feature stores, and serving tools. It is not intended to replace every part of an ML stack. Michael Del Balso and Willem Pienaar explain why feature stores address a separate set of problems, including repeated feature transformations, inconsistent data between development and production, and the movement of features into training and serving workflows. The speakers favor best-of-breed components with clear interfaces over one platform that does everything. They also argue that practical, widely adopted contracts matter more than formal standards without working implementations. Their failure stories point to two recurring mistakes: building ML code outside the software development lifecycle and choosing complex automation before trying a simpler solution.

## Key ideas
### Kubeflow connects ML components through Kubernetes-native pipelines
[05:28](https://www.youtube.com/watch?v=iWMQxCGFdU0&t=328s)
David Aronchick says Kubeflow began with a notebook, a TensorFlow custom resource, and a serving component. Users soon needed pipelines and an SDK to connect those pieces. Kubeflow provides a declarative pipeline that can be written in Python and compiled into Argo for execution. He says it does not replace Jupyter, Feast, TensorFlow, PyTorch, Seldon, Spark, or Hadoop. Instead, it provides a deployment mechanism and a way to wire components together while leaving data processing and serving systems available separately.

### Reusable components are easier to adapt than a monolithic platform
[08:41](https://www.youtube.com/watch?v=iWMQxCGFdU0&t=521s)
Michael Del Balso describes how Uber's Michelangelo platform began as a large monolithic system. It worked for the use cases it was designed around, but later teams needed variations such as serving models on a vehicle, training a graph neural network, or replacing the training system. The platform eventually moved toward reusable components that teams could swap in and out. David says Kubeflow followed a similar idea, influenced by Kubernetes: users should choose services, describe how they connect, and run them as a pipeline.

### Feature stores handle the data work that often blocks production ML
[17:06](https://www.youtube.com/watch?v=iWMQxCGFdU0&t=1026s)
David calls the feature store one of the most overlooked parts of a production-ready ML pipeline. He points to schema definition, feature engineering in a well-defined step, validation, and interfaces for later consumers. He says the hardest work in ML is often understanding and engineering data rather than training. Willem adds that feature stores sit between offline and production environments, where teams need consistent transformations and access to features for both model development and operational use.

### Shared contracts make ML tools replaceable across environments
[18:17](https://www.youtube.com/watch?v=iWMQxCGFdU0&t=1097s)
The speakers argue that feature stores and other ML components need published contracts for data, APIs, schemas, inputs, and outputs. Willem says these boundaries let teams swap components without creating a separate deployment for every cloud provider or environment. David distinguishes practical, lowercase standards from formal standards processes. A standard needs code, users, ongoing support, and more than one company or implementation behind it. He also says open governance and support for multiple platforms are useful signals when evaluating a project.

### A single end-to-end ML tool is convenient, but it limits change
[27:27](https://www.youtube.com/watch?v=iWMQxCGFdU0&t=1647s)
Michael rejects the idea that one ML tool will dominate the whole stack. He expects the space to divide into different tool categories and says teams should optimize for compatibility between them. Willem says a cloud provider's integrated platform can be attractive when a team lacks the capacity to run its own infrastructure, but open tools should remain easy to adopt and leave room to change later. David identifies Kubernetes as the closest thing to a common underlying platform because it can host independent components with standard interfaces.

### Feature stores bridge the mismatch between development data and production data
[39:41](https://www.youtube.com/watch?v=iWMQxCGFdU0&t=2381s)
Willem explains that production and offline environments often contain different data, run different systems, and belong to different teams. A model built against warehouse data may have no equivalent data source when deployed, while production data may not be available to developers. Feature stores can automate parts of the pipelines that move or reproduce features across these environments. Michael describes a bank whose developers could not easily test against the data used in production, making validation and deployment difficult.

### ML code needs to enter the software development lifecycle
[47:56](https://www.youtube.com/watch?v=iWMQxCGFdU0&t=2876s)
David's failure story is a capable developer who builds a model on a laptop, gets it running in production, and then leaves. The company is left with an undocumented binary or library that nobody knows how to maintain. He says data scientists are software developers and need tools that connect their work to the software development lifecycle. Declarative pipelines, feature stores, structured repositories, and automated processes can reduce the risk of manual steps and undocumented work.

### Simple solutions should come before automated retraining
[50:01](https://www.youtube.com/watch?v=iWMQxCGFdU0&t=3001s)
Michael recalls an Uber project where a team spent substantial effort exploring automated retraining and online learning. The eventual finding was that most of the benefit could have come from giving the model more timely signals through real-time features. The model did not need to be updated constantly. His broader advice is to try the simplest operational solution first and add complexity only when the simpler approach is insufficient.

## Notable quotes
- David Aronchick: "What we are is a deployment mechanism for those other tools and a pipeline system to let you wire them together in whatever way it makes sense." (08:02)
- David Aronchick: "The feature store in my opinion is probably one of the most overlooked components when it comes to building a production ready pipeline." (17:06)
- David Aronchick: "Publishing a standard is to some degree meaningless. I can write a standard for anything, but unless there's code and human beings and work behind it to support it and use it and share it, it's not a standard." (24:42)
- David Aronchick: "The difference between that Jupyter notebook converging on her local laptop and that thing being packaged, security checked, rolled out in production, monitored, studied using live data is the Grand Canyon." (35:20)
- Michael Del Balso: "The broader theme was just about be very careful about the operational overhead of stuff you're taking on and try to figure out the simplest possible way to solve this problem and most likely try that first." (51:16)

## Tools & references mentioned
- MLOps Community
- Demetrios Brinkmann
- Kubeflow
- Kubernetes
- GKE
- Feast
- Tecton
- TensorFlow
- PyTorch
- Seldon
- Jupyter
- Argo
- Spark
- Hadoop
- Michelangelo
- Uber
- Helm
- CNAB
- Heroku
- Ruby on Rails
- Pandas DataFrame
- IETF
- Cookiecutter Data Science
- Jenkins
- GitHub Actions

## Who should watch
- You are choosing between an integrated ML platform and a stack of focused tools, and want the panel's reasoning for using interchangeable components.
- Your models work in development but production data, feature transformations, or serving conditions differ from what the team tested.
- You are building ML infrastructure and need practical guidance on contracts, deployment pipelines, and avoiding unnecessary operational complexity.

## Editor's note

David Aronchick says a model built by one developer can reach production as an undocumented binary that nobody knows how to maintain. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace a model back to the data and code that produced it. Its pipeline code can run on different configured infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Building ML Blocks with Kubeflow Orchestration with Feature Store](https://mlopstalks.com/talks/building-ml-blocks-with-kubeflow-orchestration-with-feature-store) (Aniruddha Choudhury, Publicis Sapient, 1:26:03)
- [Kubeflow vs MLflow](https://mlopstalks.com/talks/kubeflow-vs-mlflow) (Byron Allen, Servian, 54:57)
- [MLflow vs Kubeflow 2022](https://mlopstalks.com/talks/mlflow-vs-kubeflow-2022) (Byron Allen, Contino, 1:05:40)
- [Optimizing Your ML Workflow with Kubeflow 1.0](https://mlopstalks.com/talks/optimizing-your-ml-workflow-with-kubeflow-1-0) (Josh Bottum, Arrikto, 1:03:41)
- [Why and When to Use Kubeflow for MLOps](https://mlopstalks.com/talks/why-and-when-to-use-kubeflow-for-mlops) (Ryan Russon, Maven Wave Partners, 58:57)