# Lessons Learned From Hosting the ML Engineered Podcast

Charlie You, Workday | MLOps Coffee Sessions | Episode 28 | 1:04:21
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=J1WpAJRt3rg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/lessons-learned-from-hosting-the-ml-engineered-podcast
Published: 2021-01-29
Tags: careers, deployment, monitoring, privacy

## TL;DR
- Machine learning projects have a second lifecycle around data, models, drift, monitoring, and retraining, so they cannot follow software development practices unchanged.
- Machine learning engineering is broad enough that it will likely split into more specialised roles, with tools connecting those roles and encoding proven practices.
- An engineer's advantage is more likely to come from data, deployment, and model maintenance than from trying to compete with research specialists on algorithms.

## Summary
Charlie You explains how sports betting led him from scraping public statistics to building a neural network in high school, then into machine learning engineering at Workday. He started the ML Engineered podcast after realising that putting models into production was much harder than the simple train-and-ship workflow he had expected. The conversation focuses on why machine learning projects have a different risk profile from ordinary software. Teams must manage data, model lifecycles, drift, monitoring, research uncertainty, and changing accuracy requirements. Charlie expects the broad machine learning engineer role to split into more specialised jobs, while tools connect those roles and preserve practical knowledge. He also discusses security and privacy requirements at Workday, including the need to remove customer data and its derived artifacts within 14 days. For careers, he recommends finding a personal area of interest and building an advantage in data, deployment, and maintenance rather than chasing every new algorithm or paper.

## Key ideas
### Charlie You entered machine learning through sports betting and early neural networks
[01:52](https://www.youtube.com/watch?v=J1WpAJRt3rg&t=112s)
Charlie You first encountered coding through his high school's robotics team, then began scraping basketball and baseball statistics to find an edge in sports betting. He moved from Excel to MATLAB and Python and built his first neural network in high school to create real-time embeddings for NBA players. In college, Andrew Karpathy's article "The Unreasonable Effectiveness of Recurrent Neural Networks" helped convince him to pursue the field. He later worked at startups, a research lab, and Amazon AI before joining Workday as a deep learning engineer.

### Production machine learning has a second lifecycle beyond ordinary software delivery
[06:44](https://www.youtube.com/watch?v=J1WpAJRt3rg&t=404s)
Charlie You says he started the podcast after realising that machine learning engineering was much harder than he had expected. The simple picture is to take data, train a model, ship it, and repeat the normal software feedback cycle. In practice, teams also manage streaming data, model lifecycles, data drift, monitoring, and retraining. He adds that deep learning monitoring and deep learning itself were still far from solved when the episode was recorded. The podcast helped him find practitioners who had workable answers he could apply on the job.

### A book about fine-tuning produced a direct solution to a work problem
[08:58](https://www.youtube.com/watch?v=J1WpAJRt3rg&t=538s)
Charlie You gives an example from reading ML Design Patterns on a flight. His team had been trying to solve a problem before he left for vacation, without finding a satisfactory approach. On the flight home, he read about fine-tuning models from an earlier epoch and running them on new data, alongside embedding techniques. The combination produced a possible solution to the problem. After returning to work, he added Jira stories and began implementing it. The example shows how practical ideas can come from outside a team's immediate work and become useful quickly.

### Machine learning projects have more technical uncertainty than software projects
[10:35](https://www.youtube.com/watch?v=J1WpAJRt3rg&t=635s)
Charlie You contrasts ordinary software's main risk, delivering a feature customers do not value, with machine learning's technical risk. A customer may clearly want an automated workflow, while the team still does not know whether it can build a model that performs well enough. He describes a project where the original requirement was 80 percent top-five accuracy, then changed halfway through the quarter to 80 percent top-one accuracy. The product change sounded small to the product manager, but Charlie says it was far harder technically. Sprints can help with progress reviews, though they do not create the same tight feedback loop as software.

### Project methods should follow machine learning's risk profile
[16:24](https://www.youtube.com/watch?v=J1WpAJRt3rg&t=984s)
Charlie You argues that machine learning project management may need to borrow from areas with similar uncertainty, including hardware research and semiconductor work, rather than copying software agile practices. He traces agile back to lean methods influenced by Toyota's production system and suggests that those older methods may also contain useful lessons. The discussion points out that project management should fit the available expertise and resources. Demetrios Brinkmann says hardware teams often depend on highly specialised subject matter experts, which affects how projects are managed. Charlie agrees that the method should be shaped by the work and the people doing it.

### The machine learning engineer role is likely to split into specialised roles
[19:40](https://www.youtube.com/watch?v=J1WpAJRt3rg&t=1180s)
Charlie You sees machine learning engineering following the earlier split of data science and the emergence of DevOps engineering. His own role spans meetings with legal about data contracts, infrastructure about GPU architecture, and research scientists about model mathematics. He expects further specialisation, although he does not predict the exact shape of the new roles. The hosts discuss possible areas such as monitoring, infrastructure, annotation, ethics, privacy, and legal work. Charlie also points out that the job varies widely by company, since a machine learning engineer at Workday, Facebook, or a small startup may need a very different mix of broad skills and deep expertise.

### Tools can connect specialised roles and turn tacit practice into guardrails
[25:16](https://www.youtube.com/watch?v=J1WpAJRt3rg&t=1516s)
Charlie You connects role specialisation to tooling. Jira lets project managers, product managers, and engineers coordinate without relying on spreadsheets or scattered Slack messages. Kubernetes gives DevOps engineers a focused way to manage and deploy containers. In machine learning, the lack of mature, widely accepted tools means one person often has to handle work across the full lifecycle. He says tools can also encode best practices and provide guardrails, using Docker as an example of a system that prevents some actions people should not take. Many machine learning practices still exist mainly as tacit knowledge in practitioners' heads.

### Enterprise privacy requirements can force centralised data infrastructure
[34:28](https://www.youtube.com/watch?v=J1WpAJRt3rg&t=2068s)
At Workday, Charlie You says security and privacy sit at the top of infrastructure decisions because the company handles human capital management and financial data for enterprise customers. Contracts can require the company to remove a customer's data and every artifact created from it within 14 days. Meeting that obligation requires full data provenance, yet Charlie says infrastructure tools did not provide it directly. His description suggests a pattern that differs from the usual software preference for small, separate services. Data and its derived artifacts may need to remain inside one secure cloud environment, with only outputs that contain no sensitive data allowed to leave.

### Engineers should build an advantage around data, deployment, and maintenance
[48:53](https://www.youtube.com/watch?v=J1WpAJRt3rg&t=2933s)
For career decisions, Charlie You repeats advice from Luigi Patruno: machine learning engineers who want to build useful products are unlikely to gain an advantage from competing on algorithms. Charlie says there were only a few deep learning papers per week in 2014 or 2015, but roughly 25,000 machine learning papers were published on arXiv in the previous year. He recommends following personal interest and finding a specific area where sustained attention can create an advantage. That area may be data, deployment, or retraining pipelines. He does not try to match researchers at Google or OpenAI on modelling, and instead waits for techniques to become proven before applying them.

## Notable quotes
- Charlie You: "I started the podcast because as I was working I had like an inkling that machine learning engineering was harder than most people thought and like way harder than I personally thought." (07:04)
- Charlie You: "The question is more of can we actually do this." (11:29)
- Charlie You: "The purpose of tools is in one sense to connect different types of roles." (25:16)
- Charlie You: "Your edge is never going to be in the algorithm." (55:29)
- Charlie You: "We can make avocado chairs with machine learning, but most companies can't even put a random forest into production." (59:54)

## Tools & references mentioned
- ML Engineered
- MLOps Community
- Workday
- AWS AI
- Andrew Karpathy
- The Unreasonable Effectiveness of Recurrent Neural Networks
- ML Design Patterns
- Jira
- Confluence
- Kubernetes
- Docker
- SageMaker
- Snowflake
- Snorkel
- H2O
- Luigi Patruno
- Josh Tobin
- Alex Bonos
- Todd Underwood
- Francois Chollet
- Pavle Jeremic
- OpenAI
- DALL-E
- The Man Who Solved the Market
- James Simons
- Bob Iger
- Dan Burke
- Toyota Production System

## Who should watch
- You are adapting agile or DevOps processes for machine learning and need to account for research risk, changing metrics, and slower feedback.
- Your machine learning role covers data, infrastructure, legal questions, and deployment, and you want to understand why further specialisation may be coming.
- You are overwhelmed by new papers and tools and want a practical way to decide where to build depth.

## Editor's note

Charlie You says enterprise data deletion can require removing customer data and every artifact derived from it within 14 days, which demands full data provenance. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace an artifact or model back to the data and code that produced it. That record supports this kind of deletion work.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Machine Learning Care](https://mlopstalks.com/talks/machine-learning-care) (Matthew Dombrowski, 16:15)
- [Reliable Machine Learning](https://mlopstalks.com/talks/reliable-machine-learning) (Niall Murphy, Stanza Systems & Todd Underwood, Google, 1:02:25)
- [Practical MLOps](https://mlopstalks.com/talks/practical-mlops) (Noah Gift, Pragmatic AI Labs, 58:36)
- [Maturing Machine Learning in Enterprise](https://mlopstalks.com/talks/maturing-machine-learning-in-enterprise) (Kyle Gallatin, Etsy, 47:09)
- [Machine Learning Engineering in Action](https://mlopstalks.com/talks/machine-learning-engineering-in-action) (Ben Wilson, 53:05)