Keith Trnka says machine learning projects usually fail because teams misunderstand user or business needs, or because the surrounding deployment and operations work is weak.
2
Teams can move from exploration to production by simplifying notebooks, separating stable code, using limited beta releases, and letting users and developers work together closely.
3
Keith learned to lead engineering work by finding the team's current bottleneck, building missing expertise, and creating a culture where people improve each other's code without becoming defensive.
Summary
Keith Trnka describes how he moved from language modeling into broader machine learning and then into software engineering, operations, product context, and management. He says project failures often come from building something users do not need or from weak deployment, rollback, monitoring, and incident practices. He recommends cleaning up exploratory notebooks gradually, moving stable pieces into Python files, and testing promising work with a limited internal beta before a full release. Keith also explains how he learned enough AWS and engineering practice to support his team when hiring did not solve its bottlenecks. He prefers retrospectives, direct user contact, and code reviews that help people improve rather than punish them. The conversation ends with his account of moving healthcare machine learning services from a monolith on Appable to AWS Lambda with zero downtime, while acknowledging that the team's load testing did not fully predict production cold-start behavior.
Machine learning projects often fail around the model rather than inside it
Keith says the machine learning part is usually not what makes a project fail. Teams may build something users or the business do not need, become too excited about technology, or miss the needs of the people they are serving. Operational gaps also matter: teams must know how to deploy, revert, detect failures, and avoid waking people repeatedly with noisy alarms. These experiences pushed Keith to look beyond model development and spend more time understanding users, deployment, reliability, and safety.
Readable code is a form of communication between developers
Keith explains that code communicates through variable names, function names, and structure. Developers need to make their intent understandable to other people, since many engineering mistakes are misunderstandings. He connects this challenge to data science, where mathematical terminology and specialized backgrounds can make communication harder. In exploratory work, he recommends taking short breaks to explain notebook cells, split up giant notebooks, and turn stable sections into Python files that can be imported elsewhere.
Context helps engineers make the small decisions that requirements leave open
A mission statement, Jira ticket, or requirements document may describe a few large tasks, but it cannot specify all the small decisions that affect users. Keith says engineers make better choices when they understand the product, the business, and the people using it. In healthcare, learning that doctors care about liability and safety changed the questions he asked about proposed work. He learned through conversations with doctors, artificial scenarios, chat logs, feedback on features, and repeated questions about why people worked in particular ways.
Teams should test promising notebook work with a limited group before a full launch
Keith describes several stages between a notebook experiment and a production release. His team created an internal beta group where software still had to meet security and privacy requirements, but a short outage was acceptable. Developers and beta users then worked together in a Slack channel and fixed low-hanging problems week by week. Some experiments were shelved when the team could not find a valuable feature. In successful cases, beta users became advocates who pulled the feature into the product by asking for it.
Productionizing an experiment starts with removing unnecessary complexity
Keith's first step after a successful experiment is to simplify the notebook. Exploratory code often contains subtle features that add a lot of size and complexity for little benefit. Removing that material makes every later production step easier. Early in his team's development, they converted notebooks into training and serving code inside a monolith, with existing processes for model versioning, rebuilding, deployment, hosting, and scaling. Later they moved toward separate services, often pairing a research scientist with a software engineer.
Managers should work with a team's motivation instead of forcing every practice immediately
Keith says mandates are sometimes necessary when a team has outages or privacy breaches, but people learn best when they are motivated and ready. He uses code reviews and retrospectives to connect engineering practices to real problems. A bug caused by unclear code gives the team a reason to discuss clarity. He also warns that aggressive reviews make people defensive and can lead them to delay submitting code until it seems perfect. The healthier pattern is for people with different strengths to help each other improve.
Leaders should find the current bottleneck before choosing how to improve the team
Keith recommends regular retrospectives where people discuss what went badly, where work slowed down, and what they could change. He also asks managers to view software development as a system and identify its limiting factor. At different points, his team's bottleneck was hiring, AWS knowledge, or another engineering concern. A leader can address that by learning personally, hiring, contracting, or bringing in expertise from elsewhere, but the first step is recognizing what is holding the team back.
Zero-downtime migrations require tested switching and fast rollback
Keith's team moved machine learning services from Appable to AWS after deployment and registry problems became frustrating. They compared ECS and Lambda with load tests and chose Lambda for their needs. Because the services supported healthcare users, Keith required a migration with zero downtime, allowing at most one or two failed requests, plus a rollback within about a minute. They used a faster DNS indirection layer instead of relying on slow name-server changes. The team tested deployment and reversal in a lower environment, then completed the production migration while Keith was on vacation.
"Usually it wasn't the machine learning part that led to a project failing. Usually it was trying to do something that didn't make sense that the users didn't need or the business didn't need."Keith Trnka05:31
Who should watch
You manage data science or machine learning teams and need practical ways to improve engineering habits without imposing every practice from above.
Your team has notebooks that work in exploration but struggles with deployment, rollback, monitoring, or user feedback before launch.
You are planning a migration or service redesign in a healthcare setting and want to hear how Keith approached downtime, testing, and rollback risk.
