# Luigi in Production

Luigi Patruno, ML in Production | MLOps Coffee Sessions | Episode 18 | 47:23
Hosted by David Aponte

Source: https://www.youtube.com/watch?v=ShBod1yXUeg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/luigi-in-production
Published: 2020-11-09
Tags: data-engineering, deployment, monitoring

## TL;DR
- Luigi Patruno started ML in Production because online machine learning material focused on toy examples and algorithms, while production work centered on data, pipelines, deployment, and monitoring.
- He learns from companies such as Google, Microsoft, Amazon, Netflix, and Stitch Fix by studying their papers, engineering systems, and cloud products, then adapting the lessons to his own constraints.
- Teams should log inputs and outputs, monitor models, version models and datasets, and automate work only after running it manually long enough to understand its failure modes.

## Summary
Luigi Patruno explains why he started ML in Production, first as a newsletter and later as a blog and course project. His focus came from working across data engineering, machine learning engineering, and data science, where algorithms took little of the total effort. Data structures, pipelines, deployment, and monitoring took far more work, yet online material often concentrated on beginner examples and model selection. He describes how he studies production systems from companies such as Google, Microsoft, Amazon, Netflix, and Stitch Fix, along with tools such as SageMaker, to understand the decisions behind them. He advises teams to log predictions and their inputs, monitor model behavior, version models, and store training, validation, and test data. For choosing use cases, he weighs potential business impact against the quality and availability of the data. He also argues that a simple model shipped to users can create more value than a complex model kept in experimentation.

## Key ideas
### Production work is mostly outside the algorithm
[02:10](https://www.youtube.com/watch?v=ShBod1yXUeg&t=130s)
Luigi's experience spans data engineering, ML engineering, and data science. Across those roles, he found that choosing an algorithm took little of the total effort. More time went into data structures, querying data, building transformation pipelines, moving data between systems, deploying models, and monitoring them after release. A production failure can affect the business directly, so the operational parts matter. He started ML in Production after seeing that much online material focused on classifiers, regressors, and beginner examples rather than the work required to make machine learning useful to end users.

### Best practices come from simple systems and hard-won experience
[07:48](https://www.youtube.com/watch?v=ShBod1yXUeg&t=468s)
Luigi recommends starting with the simplest workable approach, finding out why it is insufficient, and then improving it. He does not claim to have every answer in advance. He studies companies that have operated machine learning systems for years, especially Google and Microsoft, along with ad technology companies that had to collect data, build models, and serve them at scale. Their papers and products contain decisions shaped by real operating problems. Cloud services also encode some of those lessons, so he studies how systems such as SageMaker were built and considers what those choices might teach his own teams.

### Engineering choices depend on goals and constraints
[11:21](https://www.youtube.com/watch?v=ShBod1yXUeg&t=681s)
Luigi says data science and engineering involve substantial subjectivity. The right choice depends on the team's goals, the company's existing infrastructure, and the need to ship responsibly. A company built around AWS may be unable to adopt a Google Cloud tool because that conflicts with its technology organization. He recommends considering the desired outcome and the constraints before choosing a tool or design. A product that works within the existing environment may be a better decision than a theoretically attractive system that introduces unnecessary infrastructure or organizational friction.

### Learning improves when people reconstruct systems in their heads
[17:16](https://www.youtube.com/watch?v=ShBod1yXUeg&t=1036s)
Luigi describes himself as largely self-taught through papers, talks, blog posts, and the systems described in them. He tries to mentally reconstruct the architecture rather than passively consume the material. When reading about Uber's Michelangelo system and estimated delivery times for Uber Eats, he considered which features would come from historical data, which would arrive in a request, how features would be made available at runtime, and how the deployed system could be monitored. He also thought about feedback signals, such as the actual delivery time, that could be compared with predictions.

### Companies reveal MLOps lessons when real products force the issue
[20:53](https://www.youtube.com/watch?v=ShBod1yXUeg&t=1253s)
Luigi points to Google, Amazon, Microsoft, Netflix, and Stitch Fix because their machine learning work grew from concrete products. Google and Microsoft had to serve ads or search results. Stitch Fix and Netflix had to personalize recommendations, while Amazon had to recommend products. These companies developed tools and methods because their products demanded them at scale. Luigi also mentions Dropbox, whose engineering posts surprised him because they showed a company he did not primarily associate with machine learning. He liked that the posts began with a product problem, described simple heuristics or statistics, and then showed how the systems became more complex.

### Logging and monitoring prevent silent failures
[25:33](https://www.youtube.com/watch?v=ShBod1yXUeg&t=1533s)
Luigi describes SageMaker's ability to capture model inputs and outputs and store them in S3. Those records can later be used to inspect behavior, look for drift, and compare predictions with ground truth when that signal becomes available. He connects this to a failure on one of his own teams. A production system generated real-time predictions that informed later decisions, but the team had not logged part of the stack. They discovered an anomaly during unrelated data analysis. Monitoring a simple measure such as the average classifier output could have exposed the regression much earlier.

### Automation should follow manual understanding
[30:52](https://www.youtube.com/watch?v=ShBod1yXUeg&t=1852s)
Luigi's operating advice is to log and monitor models, automate deployments where possible, tag the exact model versions used for predictions, and store training, validation, and test sets. He agrees that manual work can be useful at the start. Running a deployment or analysis manually helps a team discover pitfalls before encoding the process in automation. The same applies to model development and error analysis. People should inspect samples and misclassified examples themselves before relying on automated methods. Once the process and its failure modes are understood, automation can reduce future development time.

### Use cases should be ranked by impact and data readiness
[35:34](https://www.youtube.com/watch?v=ShBod1yXUeg&t=2134s)
For a small team, Luigi begins with the business goal and ranks problems by possible impact across the company. He considers how many users or business areas a solution could affect, then weighs that against the likely performance of a predictive model. A less accurate model can still create useful value when the potential reach is large. He also examines how developed the required datasets are. Missing data, inconsistent data spread across databases, and the need to build normalized datasets can consume substantial time. Impact and data readiness provide the first filters for deciding whether a machine learning project is worth pursuing.

### Simple useful models often beat complex models that never ship
[38:48](https://www.youtube.com/watch?v=ShBod1yXUeg&t=2328s)
Luigi says people underestimate how much value an industry team can create without the most complex or highest-performing model. Model performance matters, but its effect on business value depends on the use case. A team may learn more by putting a workable model in users' hands, measuring a product or business metric, and running a small A/B test than by returning to the lab to gain another increment in model performance. He identifies a common failure mode from his own experience: starting to build a complex solution before understanding the business problem and the value it might create.

## Notable quotes
- Luigi Patruno: "The algorithms took up almost no time of my work." (05:48)
- Luigi Patruno: "I think a lot of people might disagree with me there, but for all the talk about how sort of technical the field is it's very subjective." (11:21)
- Luigi Patruno: "Do log everything, do monitor your models." (30:52)
- Luigi Patruno: "You want to do the thing manually first until to discover pitfalls and to discover issues until you feel confident enough that you can automate it." (32:48)
- Luigi Patruno: "The real bottleneck there is the domain knowledge." (42:12)

## Tools & references mentioned
- ML in Production
- 2U
- SageMaker
- TWIML
- Google
- Microsoft
- Amazon
- Netflix
- Stitch Fix
- Dropbox
- Uber Michelangelo
- S3
- Google's Introduction to Machine Learning Problem Framing

## Who should watch
- You are building a machine learning product and spend more time on data pipelines, deployment, and monitoring than on model selection.
- Your team is deciding which machine learning use cases to pursue with limited people and uneven data quality.
- You want practical ways to learn from large companies without copying their systems wholesale.

## Editor's note

Luigi Patruno says teams should log model inputs and outputs because an unlogged production stack hid a regression until an unrelated analysis found it. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a model or artifact can be traced back to the data and code that produced it. Unchanged steps are cached instead of recomputed.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Luigi in Production Part 2](https://mlopstalks.com/talks/luigi-in-production-part-2) (Luigi Patruno, 2U, 58:54)
- [Scaling AI in Production](https://mlopstalks.com/talks/scaling-ai-in-production) (Srivatsan Srinivasan, AIEngineering, 51:56)
- [Tecton Round-table // Get your ML Application Into Production](https://mlopstalks.com/talks/tecton-round-table-get-your-ml-application-into-production) (Kevin Stumpf, Derek Salama, Eddie Esquivel & Isaac Cameron, Tecton, 55:42)
- [MLOps Insights](https://mlopstalks.com/talks/mlops-insights) (David Aponte-Demetrios Brinkmann-Vishnu Rachakonda, 37:47)
- [War Stories Productionising ML](https://mlopstalks.com/talks/war-stories-productionising-ml) (Nick Masca, Marks and Spencer, 50:48)