# Don't Listen Unless You Are Going to Do ML in Production

Kyle Morris, banana.dev | MLOps Coffee Sessions | Episode 87 | 51:30

Source: https://www.youtube.com/watch?v=eDFCZNZnN-Q
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/dont-listen-unless-you-are-going-to-do-ml-in-production
Published: 2022-03-17
Tags: gpus, inference, latency, model-serving

## TL;DR
- banana.dev takes a trained model and returns an API that scales, manages latency, and limits hosting costs so teams can focus on model quality.
- Production requirements should be written down as latency, cost, reliability, and scaling expectations before engineering work begins.
- Teams should secure their infrastructure, understand how customers use their tools, and avoid paying for compute that sits idle.

## Summary
Kyle Morris describes banana.dev as a way to move a trained model into production without making every data scientist learn infrastructure operations. The service accepts model artifacts and returns an API that handles scaling, latency, and hosting costs. He explains why notebook-based deployments fail when traffic grows, updates need to be rolled out, or machines need to restart. Production work should begin with numbers for latency, price, traffic changes, model load time, and reliability. Kyle calls these constraints a production cube and is clear that banana.dev focuses on production rather than data pipelines or model quality. He also discusses practical lessons from building the service, including learning Docker, Kubernetes, Pulumi, GPU programming, and cloud infrastructure. His most serious incident involved a public Redis queue that attackers used for remote code execution and cryptocurrency mining. He advises teams to examine real customer failure modes instead of testing situations their product does not support.

## Key ideas
### banana.dev packages production infrastructure behind a model API
[03:56](https://www.youtube.com/watch?v=eDFCZNZnN-Q&t=236s)
Kyle says banana.dev accepts a model whose quality is already good enough and returns an API that can be called in two lines of code. The model can then be used like a third-party service, while banana.dev handles scaling, cost, and inference speed. He frames this as separating differentiated work, model quality, from the common production work every company faces. The service is aimed at teams that want to reach production without waiting months to hire infrastructure engineers, as well as teams whose infrastructure engineers want better tooling without being locked into a vendor.

### GPU workloads need fast startup and careful handling of idle time
[06:01](https://www.youtube.com/watch?v=eDFCZNZnN-Q&t=361s)
Kyle says serverless GPUs are banana.dev's most valuable early offering because machines can start quickly and customers pay for usage rather than leaving GPUs running. The company works with language models, computer vision, and text-to-speech workloads. He says some model setups that took more than 15 minutes to start were reduced to seconds. GPUs can make inference much faster, although CPUs remain suitable for cases where latency is less important. He connects startup time and inference speed directly to user experience, especially for interactive applications.

### Notebook deployments break when real users and updates arrive
[09:07](https://www.youtube.com/watch?v=eDFCZNZnN-Q&t=547s)
Kyle describes teams moving directly from Google Colab or another notebook into production by exposing the notebook through Flask or Sanic. In one version, closing the laptop brought production down. These setups also fail during traffic spikes because inference can take seconds, and they make safe updates difficult. A team may already have hundreds of users when it discovers that changing the deployment takes production down. Kyle compares this with software in a car, where changes affect an existing ecosystem of users and interactions. Production requires a way to scale, roll out updates, and recover from failures.

### Machine learning engineering is software engineering applied to a particular tool
[12:29](https://www.youtube.com/watch?v=eDFCZNZnN-Q&t=749s)
Kyle treats engineering as problem solving with the tools required for the problem. He sees machine learning as another way to make a computer perform a task, while acknowledging that neural networks have their own architecture and that machine learning also includes methods such as support vector machines, k-nearest neighbors, and principal component analysis. His company looks for people who can understand customer requirements and build a solution. Early employees need to be generalists who can adapt as the product changes. Kyle values coding ability, an understanding of servers, and Python more than a complete list of infrastructure skills at hiring time.

### Production requirements should be agreed in numbers before building
[16:57](https://www.youtube.com/watch?v=eDFCZNZnN-Q&t=1017s)
Kyle says production work needs explicit requirements for inference latency, traffic spikes, price, model load time, and reliability. Model quality matters during training, while production is judged by the user's experience and the business cost. He describes a production cube with latency, cost, and scaling as its three axes. If a customer needs seven seconds or 500 milliseconds and the team delivers 30 seconds, weeks of work can still end in failure. Kyle's division of responsibility is deliberate: customers provide model artifacts and own their weights, architecture, notebooks, and data pipelines, while banana.dev focuses on hosting and production performance.

### Customer expectations differ between interactive and batch workloads
[22:07](https://www.youtube.com/watch?v=eDFCZNZnN-Q&t=1327s)
For consumer applications, Kyle says response time may matter most because a chatbot that takes 20 seconds per message will lose users compared with one that responds in half a second. Batch processing can tolerate higher latency, so price or reliability may matter more. He says customers generally assume high uptime because services such as Amazon, Google Cloud, and Azure have set that expectation. His advice is to find customers willing to pay a high price for a valuable experience, then reduce the price as the product reaches more users. Latency needs separate investigation because it can create a hard threshold where users stop using the product.

### The first production lessons involve orchestration, clouds, and GPU performance
[33:27](https://www.youtube.com/watch?v=eDFCZNZnN-Q&t=2007s)
Kyle identifies Docker, Kubernetes, and Pulumi as useful tools to learn early. Docker provides a consistent environment, Kubernetes helps with larger orchestration and autoscaling, and infrastructure-as-code tools can create cloud resources without relying entirely on a console. Supporting AWS and Google Cloud adds another layer of operational knowledge because their machine and accelerator offerings differ. Improving inference speed can require CUDA, GPU programming, operating-system knowledge, drivers, and specialized tools. Kyle is honest that these subjects took weeks to learn and are more painful when a customer is waiting for a production system.

### Testing should follow real customer failure modes and secure every queue
[39:30](https://www.youtube.com/watch?v=eDFCZNZnN-Q&t=2370s)
Kyle recommends acting like the customer and testing the common paths that matter for the product. A company does not need to stress-test Korean servers if it has no servers there, and a batch job may not need low regional latency if users do not wait for it. He recalls an early incident where a public Redis queue allowed attackers to submit a script that wiped machines and started mining bitcoin across GPUs. He says automated scanners find exposed services quickly. Teams also need to understand how downstream users employ their tools, because people may use a tool in harmful or unsupported ways.

## Notable quotes
- "We make productionizing ML really easy." (03:56)
- "Paying for idle GPUs is really dumb." (31:03)
- "You really have to understand in numbers what the customer is looking for and what they expect, and then set those expectations before you build." (18:47)
- "If you don't understand how downstream people are using your tool, really bad things can happen." (45:23)

## Tools & references mentioned
- banana.dev
- Google Colab
- Flask
- Sanic
- Docker
- Kubernetes
- Pulumi
- AWS
- Google Cloud
- CUDA
- Redis
- GPT-J
- Uber
- Cruise
- Carnegie Mellon University
- Harvard

## Who should watch
- You have a trained model and need to expose it to real users without building the entire hosting layer yourself.
- Your production bill contains idle GPU time, or traffic spikes and slow inference are hurting the product.
- You are moving from notebooks to production and need practical guidance on orchestration, security, customer requirements, and ownership boundaries.

## Editor's note

Kyle Morris says notebook deployments fail when traffic grows, machines restart, or teams need safe updates. ZenML lets teams write ML workflows as Python steps and run the same pipeline code on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud provider's services by changing the configured stack. That keeps workflow code separate from the infrastructure it runs on.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Tecton Round-table // Get your ML Application Into Production](https://mlopstalks.com/talks/tecton-round-table-get-your-ml-application-into-production) (Kevin Stumpf, Derek Salama, Eddie Esquivel & Isaac Cameron, Tecton, 55:42)
- [Luigi in Production](https://mlopstalks.com/talks/luigi-in-production) (Luigi Patruno, ML in Production, 47:23)
- [MLOps + Machine Learning](https://mlopstalks.com/talks/mlops-machine-learning) (James Sutton, Algorithmia, 1:01:50)
- [War Stories Productionising ML](https://mlopstalks.com/talks/war-stories-productionising-ml) (Nick Masca, Marks and Spencer, 50:48)
- [Scaling AI in Production](https://mlopstalks.com/talks/scaling-ai-in-production) (Srivatsan Srinivasan, AIEngineering, 51:56)