# How A Manager Became a Believer in DevOps for Machine Learning

Keith Trnka, 98.6 | MLOps Podcast | Episode 152 | 55:49
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=up66re4897g
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-a-manager-became-a-believer-in-devops-for-machine-learning
Published: 2023-04-04
Tags: deployment, engineering-culture, reliability, team-adoption

## TL;DR
- Keith Trnka says machine learning projects usually fail because teams misunderstand user or business needs, or because the surrounding deployment and operations work is weak.
- Teams can move from exploration to production by simplifying notebooks, separating stable code, using limited beta releases, and letting users and developers work together closely.
- Keith learned to lead engineering work by finding the team's current bottleneck, building missing expertise, and creating a culture where people improve each other's code without becoming defensive.

## Summary
Keith Trnka describes how he moved from language modeling into broader machine learning and then into software engineering, operations, product context, and management. He says project failures often come from building something users do not need or from weak deployment, rollback, monitoring, and incident practices. He recommends cleaning up exploratory notebooks gradually, moving stable pieces into Python files, and testing promising work with a limited internal beta before a full release. Keith also explains how he learned enough AWS and engineering practice to support his team when hiring did not solve its bottlenecks. He prefers retrospectives, direct user contact, and code reviews that help people improve rather than punish them. The conversation ends with his account of moving healthcare machine learning services from a monolith on Appable to AWS Lambda with zero downtime, while acknowledging that the team's load testing did not fully predict production cold-start behavior.

## Key ideas
### Machine learning projects often fail around the model rather than inside it
[05:31](https://www.youtube.com/watch?v=up66re4897g&t=331s)
Keith says the machine learning part is usually not what makes a project fail. Teams may build something users or the business do not need, become too excited about technology, or miss the needs of the people they are serving. Operational gaps also matter: teams must know how to deploy, revert, detect failures, and avoid waking people repeatedly with noisy alarms. These experiences pushed Keith to look beyond model development and spend more time understanding users, deployment, reliability, and safety.

### Readable code is a form of communication between developers
[04:31](https://www.youtube.com/watch?v=up66re4897g&t=271s)
Keith explains that code communicates through variable names, function names, and structure. Developers need to make their intent understandable to other people, since many engineering mistakes are misunderstandings. He connects this challenge to data science, where mathematical terminology and specialized backgrounds can make communication harder. In exploratory work, he recommends taking short breaks to explain notebook cells, split up giant notebooks, and turn stable sections into Python files that can be imported elsewhere.

### Context helps engineers make the small decisions that requirements leave open
[13:57](https://www.youtube.com/watch?v=up66re4897g&t=837s)
A mission statement, Jira ticket, or requirements document may describe a few large tasks, but it cannot specify all the small decisions that affect users. Keith says engineers make better choices when they understand the product, the business, and the people using it. In healthcare, learning that doctors care about liability and safety changed the questions he asked about proposed work. He learned through conversations with doctors, artificial scenarios, chat logs, feedback on features, and repeated questions about why people worked in particular ways.

### Teams should test promising notebook work with a limited group before a full launch
[29:48](https://www.youtube.com/watch?v=up66re4897g&t=1788s)
Keith describes several stages between a notebook experiment and a production release. His team created an internal beta group where software still had to meet security and privacy requirements, but a short outage was acceptable. Developers and beta users then worked together in a Slack channel and fixed low-hanging problems week by week. Some experiments were shelved when the team could not find a valuable feature. In successful cases, beta users became advocates who pulled the feature into the product by asking for it.

### Productionizing an experiment starts with removing unnecessary complexity
[27:34](https://www.youtube.com/watch?v=up66re4897g&t=1654s)
Keith's first step after a successful experiment is to simplify the notebook. Exploratory code often contains subtle features that add a lot of size and complexity for little benefit. Removing that material makes every later production step easier. Early in his team's development, they converted notebooks into training and serving code inside a monolith, with existing processes for model versioning, rebuilding, deployment, hosting, and scaling. Later they moved toward separate services, often pairing a research scientist with a software engineer.

### Managers should work with a team's motivation instead of forcing every practice immediately
[42:01](https://www.youtube.com/watch?v=up66re4897g&t=2521s)
Keith says mandates are sometimes necessary when a team has outages or privacy breaches, but people learn best when they are motivated and ready. He uses code reviews and retrospectives to connect engineering practices to real problems. A bug caused by unclear code gives the team a reason to discuss clarity. He also warns that aggressive reviews make people defensive and can lead them to delay submitting code until it seems perfect. The healthier pattern is for people with different strengths to help each other improve.

### Leaders should find the current bottleneck before choosing how to improve the team
[38:25](https://www.youtube.com/watch?v=up66re4897g&t=2305s)
Keith recommends regular retrospectives where people discuss what went badly, where work slowed down, and what they could change. He also asks managers to view software development as a system and identify its limiting factor. At different points, his team's bottleneck was hiring, AWS knowledge, or another engineering concern. A leader can address that by learning personally, hiring, contracting, or bringing in expertise from elsewhere, but the first step is recognizing what is holding the team back.

### Zero-downtime migrations require tested switching and fast rollback
[48:05](https://www.youtube.com/watch?v=up66re4897g&t=2885s)
Keith's team moved machine learning services from Appable to AWS after deployment and registry problems became frustrating. They compared ECS and Lambda with load tests and chose Lambda for their needs. Because the services supported healthcare users, Keith required a migration with zero downtime, allowing at most one or two failed requests, plus a rollback within about a minute. They used a faster DNS indirection layer instead of relying on slow name-server changes. The team tested deployment and reversal in a lower environment, then completed the production migration while Keith was on vacation.

## Notable quotes
- Keith Trnka: "Usually it wasn't the machine learning part that led to a project failing. Usually it was trying to do something that didn't make sense that the users didn't need or the business didn't need." (05:31)
- Keith Trnka: "Writing code effectively, there's a lot to it of course, but a lot of it is communication when it comes down to it." (04:53)
- Keith Trnka: "Having that context gives you a way to fill in those blanks in a way that's aligned with your users, that's aligned with the business, that's aligned with the other business units." (14:20)
- Keith Trnka: "The deployment has to happen with zero downtime, maybe one or two failed requests okay, but not 10 minutes or something." (48:40)
- Keith Trnka: "You want a culture where people are looking out for each other, and our code review has a mixture of protecting your users and also kind of leveling each other up." (44:23)

## Tools & references mentioned
- MLOps Community
- Aeropress
- The College of New Jersey
- University of Delaware
- Swype
- Nuance
- 98.6
- AWS
- Jupyter notebooks
- Python
- Flask
- Docker
- Jenkins
- SageMaker
- ECS
- Lambda
- Appable
- Tony Fadell
- Build: An Unorthodox Guide to Making Things Worth Making
- The Mom Test
- Slack
- GitHub
- Locust
- HIPAA

## Who should watch
- You manage data science or machine learning teams and need practical ways to improve engineering habits without imposing every practice from above.
- Your team has notebooks that work in exploration but struggles with deployment, rollback, monitoring, or user feedback before launch.
- You are planning a migration or service redesign in a healthcare setting and want to hear how Keith approached downtime, testing, and rollback risk.

## Editor's note

Keith Trnka says teams often struggle when they move exploratory notebook work into production code and services. ZenML lets teams write workflows as Python steps on a configured stack, so the same pipeline code can run on a laptop, Kubernetes, Airflow, or a cloud provider's services. This gives teams a way to keep workflow code separate from the infrastructure that runs it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Challenges Operationalizing ML (And Some Solutions)](https://mlopstalks.com/talks/challenges-operationalizing-ml-and-some-solutions) (Nathan Ryan Frank, WW Grainger, 52:28)
- [Practical MLOps](https://mlopstalks.com/talks/practical-mlops) (Noah Gift, Pragmatic AI Labs, 58:36)
- [MLOps and DevOps, Parallels and Deviations](https://mlopstalks.com/talks/mlops-and-devops-parallels-and-deviations) (Damian Brady, Microsoft, 55:32)
- [Reinvent Yourself and Be Curious](https://mlopstalks.com/talks/reinvent-yourself-and-be-curious) (Stefano Bosisio, Synthesia, 57:16)
- [The Godfather Of MLOps](https://mlopstalks.com/talks/the-godfather-of-mlops) (D. Sculley, Google, 51:25)