# Luigi in Production Part 2

Luigi Patruno, 2U | MLOps Coffee Sessions | Episode 36 | 58:54

Source: https://www.youtube.com/watch?v=z7lsOv2AKeQ
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/luigi-in-production-part-2
Published: 2021-04-23
Tags: data-quality, monitoring, platform-teams, privacy

## TL;DR
- Luigi Patruno moved from writing code to managing five people and the machine learning projects his company relies on, while staying involved in technical decisions.
- Teams should solve problems manually before automating them, then turn what they learn into checklists, templates, and eventually platform work when the process is understood.
- Machine learning teams should communicate through business value, shared metrics, and experiments because senior leadership determines which problems matter most.

## Summary
Luigi Patruno discusses his move from individual contributor work into management at 2U. He now manages five people and oversees machine learning projects while remaining involved in architecture, code reviews, and technical decisions. He explains how his team moved models from Jupyter notebooks into production processes with data validation, drift monitoring, and end-to-end monitoring. For automation, he recommends starting manually, documenting the process, and creating templates after the team understands the work. He is candid about earlier failures at 2U, where the team worked on technically successful projects that did not have enough business impact. Luigi argues that strong executive leadership is needed to identify the right problems. He also discusses learning through books, his gaps in business and product knowledge, and the tension between machine learning and privacy. His advice on data collection is to use the least data necessary and explain its purpose in plain language.

## Key ideas
### Management lets Luigi work at a higher level without leaving technical decisions
[02:15](https://www.youtube.com/watch?v=z7lsOv2AKeQ&t=135s)
Luigi paused his newsletter and MLInProduction work after his son was born and because those activities took about 30 to 40 hours each week alongside his full-time job. At work, his team grew from two people to five, and he became responsible for the machine learning projects the company relies on each day. He no longer writes code every day, but he still joins code reviews, gives technical advice, and thinks through models and architecture. He describes the change as moving up several levels of abstraction. He also enjoys matching people to the work where they are strongest, so modelers spend time modeling and engineers spend time engineering.

### Production work needs a repeatable process from validation through monitoring
[07:22](https://www.youtube.com/watch?v=z7lsOv2AKeQ&t=442s)
Luigi says his team took several models from prototype code in Jupyter notebooks to production systems that produced tens of thousands of predictions each day with essentially no downtime. The process includes data validation throughout the pipelines, checks for drift, validation of field types and expected values, and monitoring at the end of the full process. He feels confident about the productionizing process because the team has applied it repeatedly. Project intake is less settled. Luigi relies on strong senior leadership and a chief data scientist with access to executive discussions to help identify the company's most important problems.

### Automation should follow manual learning instead of preceding it
[13:30](https://www.youtube.com/watch?v=z7lsOv2AKeQ&t=810s)
Luigi prefers automation, but he warns that teams can invest heavily in the wrong abstraction before they understand the work. His approach is to perform a task manually, then create a repeatable human-level process, and only later consider code-level automation. For model building, his team uses markdown artifacts based on Microsoft's TDSP, including write-ups after stages such as model building and error analysis. The team applied the same idea to A/B testing. After running experiments manually and learning what the process required, they created templates that capture metrics, sample-size calculations, and other required steps. A future experimentation platform may automate this work, but Luigi wants more evidence first.

### Documented team practice can spread beyond data science
[17:24](https://www.youtube.com/watch?v=z7lsOv2AKeQ&t=1044s)
The templates created from manual experimentation give newer data scientists a checklist and preserve the knowledge gained by the people who first ran the work. Luigi says these artifacts do not need to stay within the data science team. Other parts of a company can use experimentation to improve sales contact strategies, call cadence, or conversation topics, although those cases are more complicated than measuring a click on a web page. The broader practice is to institutionalize knowledge from day-to-day work. He sees this as a way for data science methods to become useful to other teams without immediately requiring a large platform investment.

### A history of failed priorities helps shape a better company vision
[21:12](https://www.youtube.com/watch?v=z7lsOv2AKeQ&t=1272s)
Luigi has worked at 2U for almost four years, longer than at earlier startups where he stayed about a year. Remaining at one company gave him time to learn which approaches and stakeholder relationships produced poor outcomes. He says the team often failed during the first half to two-thirds of his time there, not because the technology failed, but because the team worked on problems with limited business impact. He treats that as a leadership failure. A useful vision starts with the business problem, considers the ideal long-term solution, and breaks it into smaller steps that the team can deliver and revise.

### Machine learning teams need executive context to choose high-impact work
[26:55](https://www.youtube.com/watch?v=z7lsOv2AKeQ&t=1615s)
Luigi says individual contributors and mid-level managers usually do not see many conversations about the company's real problems. They may recognize that their team is having little impact, but still lack enough context to know which work matters or which stakeholders to engage. He has had more success when senior leaders know the right problems and have influence at the executive level. Teams should communicate with non-technical stakeholders through shared metrics or a shared business goal. Revenue and cost measures are easy for everyone to understand. When individual models often fail validation, Luigi suggests viewing the work as a funnel and tracking the number of experiments or models that enter the process alongside the eventual conversion rate.

### Luigi treats business and product knowledge as a personal blind spot
[36:18](https://www.youtube.com/watch?v=z7lsOv2AKeQ&t=2178s)
Luigi's background is strongly technical, including mathematics, computer science, physics, data engineering, machine learning engineering, data science, and technical management. He is actively learning about cost drivers, revenue drivers, product value, customer needs, user interface, and user experience. He says managers can miss opportunities when they avoid problems that require skills they do not have. The answer is not to become an expert in every area. It is to bring in people with those skills and work with them. This is a change from his earlier workhorse habit of writing the code himself and handling every part of a problem.

### Reading books gives Luigi a fast way to build working knowledge
[41:32](https://www.youtube.com/watch?v=z7lsOv2AKeQ&t=2492s)
Luigi follows a personal rule to default to reading whenever he has downtime. He estimates that many books can be read in five to ten hours, spread across days or weeks, while the author may have spent thousands of hours developing the ideas. He recommends reading established books and says that studying the top books in a new field can provide enough structure to discuss most ordinary questions without making someone an expert. For management, he names High Output Management by Andy Grove. For machine learning, he recommends Hands-On Machine Learning by Aurélien Géron. He also requires his team to read Trustworthy Online Controlled Experiments because it covers organizational and operational issues as well as experiment statistics.

### Privacy requires collecting and explaining less data
[51:18](https://www.youtube.com/watch?v=z7lsOv2AKeQ&t=3078s)
Luigi connects machine learning ethics to privacy, liberty, regulation, and personal integrity. He sees a tension between machine learning's use of data to uncover insights and the desire to avoid collecting information that is not needed for operations. His preference is to extract more value from data already available rather than obtain additional data from users. He recommends collecting the least data necessary and explaining to users what the company is trying to learn in plain language. He criticizes long legal documents that people accept without reading. A company can describe the problem it is solving, such as identifying which prospective buyers are most likely to become customers, without revealing how its models work.

## Notable quotes
- "I was always focused on solving a problem rather than on using a particular tool." (04:56)
- "You don't go from doing it manually to perfectly automated." (14:14)
- "Focusing on the wrong problems leads to the biggest problems." (22:18)
- "If you find yourself having downtime, read something." (43:02)
- "I prefer to try to really squeeze out all the juice from the data that we have then getting additional data from users that we don't really need for operational purposes." (53:13)

## Tools & references mentioned
- 2U
- MLInProduction.com
- Microsoft TDSP
- Jupyter notebooks
- Creativity, Inc.
- Pixar
- Disney
- Nassim Taleb
- the Lindy effect
- High Output Management
- Andy Grove
- Intel
- Hands-On Machine Learning
- Aurélien Géron
- Trustworthy Online Controlled Experiments

## Who should watch
- You manage a data science or machine learning team and need a practical way to move from individual technical work into prioritization, delegation, and communication.
- Your team is considering an ML platform or experimentation platform before it has documented how the work actually happens.
- You need to explain machine learning work to business leaders, or you are thinking through privacy and data collection choices.

## Editor's note

Luigi Patruno says his team moved models from Jupyter notebooks into production systems by repeating a process that includes data validation, drift checks, and end-to-end monitoring. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace a model back to the data and code that produced it. It also lets the same pipeline code run on different infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Luigi in Production](https://mlopstalks.com/talks/luigi-in-production) (Luigi Patruno, ML in Production, 47:23)
- [Lessons on Data Teams Leadership](https://mlopstalks.com/talks/lessons-on-data-teams-leadership) (Luigi Patruno, 2U, Inc, 1:13:32)
- [Reinvent Yourself and Be Curious](https://mlopstalks.com/talks/reinvent-yourself-and-be-curious) (Stefano Bosisio, Synthesia, 57:16)
- [Managing Machine Learning Projects](https://mlopstalks.com/talks/managing-machine-learning-projects) (Simon Thompson, GFT, 45:02)
- [What Business Stakeholders Want to See from the ML Teams](https://mlopstalks.com/talks/what-business-stakeholders-want-to-see-from-the-ml-teams) (Peter Guagenti, Tabnine, 1:21:28)