An ML platform usually needs feature management, workflow orchestration, a model registry, model serving, and model quality monitoring.
2
Smaller teams should start with a repeatable way to package and deploy models instead of copying the full platform architecture of a large technology company.
3
Platform priorities should come from the company's own requirements, such as privacy, reliability, scale, latency, and the needs of its data scientists.
Summary
Ernest Chan explains what he learned by studying ML systems at companies including Netflix, DoorDash, Spotify, Uber, PayPal, and Intuit. His interest started after helping build Duo Trust Monitor and seeing how much of the first system could not be reused for later ML projects. He describes five common platform components: feature stores, workflow orchestration, model registries, serving systems, and model monitoring. He compares them with familiar software engineering systems while explaining where ML adds different requirements, especially feature parity and model quality metrics. Ernest advises smaller teams to begin with model packaging and deployment, after establishing a dependable data platform. He also describes Duo's early work, including an MLflow-backed model registry, access to production artifacts, and Delta Lake for consistent table updates. The discussion covers batch and real-time serving, shadow deployments, flexible versus opinionated interfaces, and the value of choosing established technologies over fashionable ones.
Duo's first ML product exposed the need for reusable infrastructure
After Duo Security launched Duo Trust Monitor, Ernest Chan saw that many future ML projects would have to start over. The first system had a reusable workflow orchestration component, but much of the rest was custom. He advocated for a small ML infrastructure team and studied companies that published details about their platforms. The goal was to understand common design choices and use that research to make later ML work easier at Duo. Ernest also wrote about the systems so the lessons could help other teams. His case-study approach focused on companies that were ahead of Duo in their infrastructure work, while avoiding the especially comprehensive systems at Google and Facebook because they felt less applicable to smaller organizations.
The common platform has five components with different responsibilities
Ernest groups the systems he studied into a feature store, workflow orchestration, a model registry, model serving, and model monitoring. The feature store supplies data for both training and serving. Orchestration runs training pipelines. The model registry stores the resulting model artifacts. Serving makes predictions in batch or online. Monitoring tracks model quality alongside operational metrics. He compares orchestration with continuous integration, a model registry with an artifact repository, and online model serving with a stateless service. Feature stores are less like ordinary databases because they must support offline and online use, maintain parity between those paths, and provide a place for teams to collaborate.
A small team does not need every platform component at the start
Ernest says workflow orchestration can be more than many data scientists need in the early stages. A complex DAG with branching and fan-out can sometimes become one scheduled job. He recommends starting with how a model is packaged and served because the difficult part is often integrating the model with an application, rather than developing the model itself. For a nonprofit with limited money and limited engineering skills, he used serverless components and a repository structure that let data scientists test, build, and deploy their models. He deliberately avoided a feature store and Kubernetes because the team would have had difficulty maintaining them.
A dependable data platform should come before a broad ML platform
Ernest distinguishes ML infrastructure from an ML platform. Infrastructure can unlock a new capability without making the process repeatable. A platform should make it easier to deliver that capability repeatedly. Before building either one, he recommends checking whether the company can reliably obtain the data needed for analytics and modeling. Duo initially believed its data warehouse and ETL platform were solid, then had to rebuild them. That work delayed later ML infrastructure. Ernest also says that a custom first implementation can reveal the real requirements, which makes it easier to generalize the system into a platform later.
Duo started with visibility into production models
Duo's production systems were difficult for data scientists to inspect because most employees did not have production access for security reasons. Ernest's first practical improvement was a model registry. Duo already had workflow orchestration, model training, and batch serving, but it lacked a registry, a repeatable model quality monitoring system, and a feature store. The registry gave data scientists a user interface for model metrics, and Duo used MLflow to send those metrics to its data warehouse for trend analysis. Since Duo trained models every day for every customer, the warehouse made it possible to study performance over time. The team also copied production artifacts to an access-controlled S3 location that data scientists could inspect.
Platform priorities must follow the company's own constraints
Ernest does not recommend copying the order or architecture of a large technology company. The companies he studied optimized for different needs. Intuit put emphasis on privacy compliance and reliability, while other systems focused on high-scale, low-latency serving. Duo prioritized visibility and access to model outputs because of its own security constraints and daily per-customer training. Ernest's prioritization combined pain points from across the data science lifecycle with expected impact, implementation effort, and dependencies between improvements. The easiest task was not always first. He also notes that published case studies show what a company has built, but rarely explain how it decided what to build first.
Shadow mode tests models without putting production traffic at risk
Ernest describes shadow mode as a way to compare a candidate model with a production model without making the candidate responsible for live results. PayPal's system made shadow deployments easier by letting teams reserve a time slot instead of using unlimited compute. A useful design also pairs the shadowed model with the model it is being compared against. Shadow deployments must have lower resource priority than production and must not disrupt production traffic. They can still add load when both models access the same data or when traffic is routed to several shadow models. Shadow mode may not repair a model that is already failing because of a major data shift, but it can speed up testing of changes.
Simple interfaces can support different levels of engineering maturity
Ernest says platform design has to balance simplicity with flexibility. Some companies first optimize for scale, reliability, and availability, then discover that data scientists find the interfaces difficult or cannot work comfortably in the system's language. Adding support for Python models may reduce efficiency while making the business faster. He favors interfaces that offer different levels of control to users with different engineering backgrounds. He also warns against standardizing too early on a new framework. Established tools such as XGBoost and PyTorch have broad use, while a framework with a recent announcement and few users creates more risk. His advice is close to Dan McKinley's idea of choosing boring technologies.
"The main components are there's a feature store which kind of feeds into both model training and serving, then there is workflow orchestration, which helps you orchestrate your model training pipelines, the output of your model training pipelines goes into a model registry, and then there's a serving system that can serve your model in online or batch."Ernest Chan06:24
Who should watch
You are building your first few production ML systems and need to decide which platform work can wait.
Your data science team has models in production but cannot inspect production metrics or artifacts.
You are comparing platform designs from large companies and want a way to adapt their lessons to a smaller team.
