# The Long Tail of ML Deployment

Tuhin Srivastava, Baseten | MLOps Podcast | Episode 161 | 50:37
Hosted by Abi Aryan

Source: https://www.youtube.com/watch?v=SUfRbLVepco
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-long-tail-of-ml-deployment
Published: 2023-06-13
Tags: careers, cost, deployment, model-serving

## TL;DR
- Tuhin Srivastava thinks machine learning will become a standard part of every software engineer's toolkit, while the data scientist role shrinks from its earlier form.
- General-purpose models can cover many low-value use cases, but specialized products will need smaller or adapted models for better speed, cost, and performance.
- Deploying large models still creates difficult infrastructure problems around latency, cost, scale to zero, cold starts, and model versioning.

## Summary
Tuhin Srivastava describes his move from investment banking to machine learning research and then to building production systems. He says that training a model in a notebook and showing a confusion matrix was never enough, so he learned software engineering to put models into products. His view of the field has changed: data scientists once carried most of the machine learning work, but software engineers are now adopting models directly and treating machine learning as part of product development. Tuhin expects general-purpose models to handle many low-value use cases. More valuable, specialized applications will need fine-tuning, smaller models, or other adaptations to meet their cost, speed, and performance needs. He also explains why large models create difficult serving problems, including cold starts, autoscaling, latency, and scale to zero. Baseten's approach is to expose models, logs, and serverless functions through APIs while keeping production versions separate from other versions.

## Key ideas
### Tuhin learned machine learning by moving from finance into applied research
[08:42](https://www.youtube.com/watch?v=SUfRbLVepco&t=522s)
Tuhin says investment banking taught him to work hard under unpleasant conditions, but he eventually left after two years and moved to Boston for biomedical research. Around 2011 and 2012, he used machine learning to predict the prognosis of neuromuscular disease so that intervention could happen earlier. The work involved support vector machines, with gradient-boosted trees becoming more popular at the time. His signal-processing background gave him probability and information theory foundations, and he spent his final months in banking relearning statistics on weekends.

### Production work pushed Tuhin from machine learning into engineering
[12:58](https://www.youtube.com/watch?v=SUfRbLVepco&t=778s)
Tuhin worked on fraud detection, content moderation, and lead scoring, which he describes as common applied machine learning use cases in the early 2010s. Companies often hired a few machine learning people and hoped something useful would emerge, but he did not want to stop at model training. He learned how to build the surrounding product because a notebook and a confusion matrix were not the goal. At Gumroad, limited engineering support meant that he learned server-side and front-end engineering, while a colleague with a mathematics PhD also became an engineer.

### Software engineers are taking on machine learning as a normal product skill
[15:59](https://www.youtube.com/watch?v=SUfRbLVepco&t=959s)
Tuhin says the field has shifted from data scientists learning engineering to software engineers learning machine learning. He thinks data scientists emerged because companies did not yet know how to turn machine learning into value, so analysts were asked to take on a research-oriented role. In his view, engineers have more ability to productize models without waiting for another team. Tuhin expects machine learning to become a large part of every engineer's toolkit, even when the engineer does not understand every internal layer of a model.

### Engineers can create useful products without becoming model researchers
[21:19](https://www.youtube.com/watch?v=SUfRbLVepco&t=1279s)
Tuhin uses Refusion as an example of engineers applying machine learning through product and domain knowledge. The project builds on Stable Diffusion and generates music from prompts, while its creators brought substantial experience in music. They fine-tuned models and used Weights & Biases, but Tuhin says they did not need a deep conceptual understanding of every internal detail to create value. He describes the field as moving from exploration toward exploitation, where higher-level abstractions let engineers treat models as black boxes and focus on what the product should do.

### General models will cover broad use cases while specialized products go deeper
[25:52](https://www.youtube.com/watch?v=SUfRbLVepco&t=1552s)
Tuhin agrees that off-the-shelf GPT-4-style APIs will cover much of the low-value, commoditized long tail. Applications with concentrated value, such as medical imaging or legal work, will need models adapted to their context. That adaptation might involve using a different model, fine-tuning, or choosing a smaller and more targeted system. He frames the decision around the task, then asks what matters for that task, including response time, cost, and performance. His restaurant analogy is that a specialist can usually make one type of food better than a restaurant covering every cuisine.

### Large models create unsolved serving and scaling problems
[31:57](https://www.youtube.com/watch?v=SUfRbLVepco&t=1917s)
Abi asks about the compute and storage costs of deploying increasingly large models. Tuhin says Baseten is working on serving a model with around 60 billion parameters, and that larger models create problems with scaling, scale to zero, cost, latency, and cold starts. A model that is needed for only a few hours each day may not be worth running continuously, but bringing it back online quickly is also difficult. He says OpenAI made its models usable through serving infrastructure, fine-tuning support, and documentation, while the broader industry still has work to do before engineers get the same simple experience.

### Model documentation and version tracking remain open problems
[35:38](https://www.youtube.com/watch?v=SUfRbLVepco&t=2138s)
Abi asks about Baseten's approach to documentation as code. Tuhin describes a workflow where engineers contribute living documentation as code, with the documentation connected to the development process. He says machine learning makes this especially difficult because models and other artifacts can disappear from a team's shared understanding. Teams may not know which model was trained when, which version matters, or which artifact is running in production. Tools such as Weights & Biases help with training records, but Tuhin says many teams still cannot identify the exact production artifact.

### Baseten uses APIs and separate production versions to manage model services
[38:37](https://www.youtube.com/watch?v=SUfRbLVepco&t=2317s)
Tuhin says Baseten treats models, model versions, logs, and serverless functions as API-based components. A team can add a serverless caching layer around a large model and scale that layer separately from the model itself. He acknowledges that microservices add operational complexity, so Baseten is working toward a more Git-based workflow. The intended model is similar to a web application: one primary production version receives live traffic, while other versions remain available for testing or explicit calls. Tuhin says this makes the problem closer to familiar software engineering workflows involving CI/CD, staging, and production.

## Notable quotes
- Tuhin Srivastava: "I was like all right if I'm going to ML, I need to understand how to actually get this to value for someone as opposed to training a model in the Jupyter notebook and showing a confusion matrix." (14:01)
- Tuhin Srivastava: "My take is actually what we're going to see now is that the data people who became machine learning people, I think that's actually going to shrink back, they're just going to go back to being data people and machine learning is going to become just like a massive massive part of every engineer's toolkit." (18:51)
- Tuhin Srivastava: "You can just go really really far without having to go that deep." (23:55)
- Tuhin Srivastava: "When you need concentrated value and you care about speed, cost and performance, which is the holy trifecta, you're going to have to go smaller, more targeted." (28:47)
- Tuhin Srivastava: "I don't think most customers of ours would be able to tell you which artifact is running in production." (38:17)

## Tools & references mentioned
- Baseten
- Gumroad
- Stanford
- Stable Diffusion
- Refusion
- Weights & Biases
- GPT-4
- OpenAI
- Databricks
- Greylock
- Conviction
- Hugging Face
- Replicate
- Vercel
- Eric Schmidt
- Matei Zaharia
- Taylor Swift

## Who should watch
- You build machine learning products and still treat deployment as the final step after notebook work.
- Your team is deciding whether a general-purpose model is enough or whether a specialized, smaller model is worth the extra engineering.
- You are dealing with model serving costs, cold starts, version tracking, or documentation that does not fit your existing software workflow.

## Editor's note

Tuhin Srivastava says many teams still cannot identify the exact production artifact running in production. ZenML records each run's steps, inputs, outputs, and code version, so a model can be traced back to the data and code that produced it. Its model registry also keeps lineage across model versions.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Tecton Round-table // Get your ML Application Into Production](https://mlopstalks.com/talks/tecton-round-table-get-your-ml-application-into-production) (Kevin Stumpf, Derek Salama, Eddie Esquivel & Isaac Cameron, Tecton, 55:42)
- [Scaling ML Capabilities in Large Organizations](https://mlopstalks.com/talks/scaling-ml-capabilities-in-large-organizations) (Bertjan Broeksema & Axel Goblet, BigData Republic, 1:02:47)
- [ML Scalability Challenges](https://mlopstalks.com/talks/ml-scalability-challenges) (Waleed Kadous, Anyscale, 1:00:03)
- [10 Years Deploying ML in the Enterprise: The Inside Scoop!](https://mlopstalks.com/talks/10-years-deploying-ml-in-the-enterprise-the-inside-scoop) (Charles Martin, MLOps Community, 1:02:48)
- [Practical MLOps](https://mlopstalks.com/talks/practical-mlops) (Noah Gift, Pragmatic AI Labs, 58:36)