# 10 Years Deploying ML in the Enterprise: The Inside Scoop!

Charles Martin, MLOps Community | MLOps Meetup | Episode 9 | 1:02:48
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=WmZL1AQPbOw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/10-years-deploying-ml-in-the-enterprise-the-inside-scoop
Published: 2020-05-12
Tags: data-quality, deployment, engineering-culture, platform-teams

## TL;DR
- Machine learning operations failures usually come from organizational constraints, especially deployment systems that were built for web software.
- Teams should run experiments quickly before automating a workflow, because automating an unproven process can lock a company into brittle tools and dependencies.
- Successful enterprise ML work needs a small, weakly coupled area where people can access data and deploy models without passing through every existing enterprise control.

## Summary
Charles Martin argues that enterprise ML projects often fail because organizations force new machine learning workflows into deployment systems built for older web applications. A typical ML system pulls data, trains a model, and deploys a microservice, but it also depends on data that may be incomplete, unvalidated, or difficult to access. Existing systems may lack the memory, swap, permissions, dependency management, or documentation that the work requires. Martin is especially critical of automating experiments before anyone knows which process works. He compares this with trading environments, where teams put algorithms into production quickly, keep what makes money, and remove what does not. He recommends a carve-out with weak connections to the rest of the organization, so researchers can test ideas without inheriting every brittle dependency. He also says leaders must want to learn and must give capable people authority to push through internal resistance.

## Key ideas
### Successful ML clients want to learn with the team
[11:40](https://www.youtube.com/watch?v=WmZL1AQPbOw&t=700s)
Martin contrasts a successful Slovenian client with projects that fail inside rigid companies. The client taught itself neural networks through a fast.ai class and asked Martin to guide the work rather than do everything for them. Together they built a tool to measure text coherency, since perplexity alone did not answer whether generated text was realistic. Martin says the client had to want to learn, and everyone involved had to engage. Projects fail when a client says its existing deployment process is mandatory and expects the ML team to spend months making a new system fit it.

### ML operations is mainly an organizational problem
[13:37](https://www.youtube.com/watch?v=WmZL1AQPbOw&t=817s)
Martin says MLOps is an organizational problem because enterprises already have deployment systems designed around the software they know. A web application may be a simple Flask API or a group of microservices, while ML has different hardware, memory, swap, and model artifacts. The existing environment may have no way to handle the workflow. Teams then spend their time fitting an ML project into old processes instead of testing the model. Martin describes this as a failure of organizational judgment, especially when people insist that the established process must remain unchanged.

### Enterprise infrastructure can make basic ML work take months
[17:00](https://www.youtube.com/watch?v=WmZL1AQPbOw&t=1020s)
Martin gives examples of infrastructure blocking ordinary work. A machine provisioned for an ML application may have no memory or swap because the organization designed it for Hadoop. Python and NumPy may need permissions to configure CPU throttling for the expected performance. One client had built an internal copy of Heroku with no documentation or support, and its dependency rules made using Conda painful. Martin says a simple deployment could have taken a day on an Ubuntu machine with a shell script, while the company's prescribed route took six weeks.

### Data access and data quality are part of deployment
[20:54](https://www.youtube.com/watch?v=WmZL1AQPbOw&t=1254s)
Martin describes ML systems as having three basic parts: pulling data from a database, training a model, and deploying a microservice. Enterprise teams often overlook the first part. A training query with many joins cannot safely run against a production database every day, yet replicas may be incomplete or poorly maintained. He describes finding a person backing up a database to a laptop because the formal backup process was impractical. ML data also lacks the validation found at normal application input points, so malformed values can break regular expressions or other processing long after they entered the system.

### Experiments should be automated before products are scaled
[35:33](https://www.youtube.com/watch?v=WmZL1AQPbOw&t=2133s)
Martin says a data scientist's immediate goal is to run experiments, since the team does not yet know what will work. He uses BlackRock as an example of a group that put algorithms into production quickly, measured whether they worked, and automated them after they proved useful. In trading, an algorithm that stops working must be removed rather than preserved as a maintenance burden. Martin therefore wants automation to target repeated experimental work and known blockers. Automating every step before the product is understood can preserve obsolete choices and prevent new experiments.

### Automation becomes dangerous when it creates tight dependencies
[39:55](https://www.youtube.com/watch?v=WmZL1AQPbOw&t=2395s)
Martin warns that automation can lock an organization to old versions and brittle assumptions. He describes environments where development used Ubuntu 16 while deployment still required Ubuntu 12, and older systems that maintained incompatible Perl versions because the automated setup could not be changed easily. ML changes quickly, so a team may need to move between frameworks or methods. Martin recommends a carve-out with weak coupling to the wider organization. The ML group should be able to get data and deploy an application without inheriting dense dependencies from every existing system.

### Leaders must choose people and projects that can change
[49:43](https://www.youtube.com/watch?v=WmZL1AQPbOw&t=2983s)
Martin says companies often announce AI plans because executives want funding, growth, or better earnings guidance, even when they have not committed to doing real ML work. He has seen companies place people without AI experience in charge of AI teams. The clients who succeed are more often those where founders or senior executives actively support the work and overrule internal blockers. When Martin is asked how to educate a company about the ML lifecycle, his answer is to find people who want to learn. The project also needs to make those people look successful, because a failed initiative may not receive another budget.

## Notable quotes
- Charles Martin: "MLOps is not a technology problem, it's an organization problem." (13:37)
- Charles Martin: "One should not solve a problem by posing an intermediate problem which is harder to solve." (20:49)
- Charles Martin: "The last thing in the world you want to do is automate something before you know what it is." (42:35)
- Charles Martin: "What you need to do is create a carve-out from the organization with a weak coupling to the rest of the organization." (42:58)
- Charles Martin: "You can't teach a student who doesn't want to learn." (55:21)

## Tools & references mentioned
- MLOps Community
- Demetrios Brinkmann
- Google
- Wired
- Panda
- Penguin
- fast.ai
- UC Berkeley
- BlackRock
- eBay
- Hadoop
- Python
- NumPy
- AWS
- Heroku
- Conda
- Flask
- SQL Server
- Vapnik
- Ruby on Rails
- Ubuntu
- Docker
- Perl
- Amazon
- IBM Watson

## Who should watch
- You are trying to put ML into an enterprise platform built for ordinary web services and need concrete reasons the existing process is blocking the work.
- Your organization wants to automate every ML step before it has proved that the model, data, and experiment are useful.
- You lead an ML initiative and need to decide whether the team has executive backing, usable data access, and enough freedom to test ideas.

## Editor's note

Charles Martin says enterprise teams waste months fitting ML work into deployment systems built for older web applications. ZenML lets teams define workflows as Python pipelines and choose the orchestrator and infrastructure through configuration, so the same workflow code can run on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud provider's services.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Maturing Machine Learning in Enterprise](https://mlopstalks.com/talks/maturing-machine-learning-in-enterprise) (Kyle Gallatin, Etsy, 47:09)
- [Just Build It! Tips for Making ML Engineering and MLOps Real](https://mlopstalks.com/talks/just-build-it-tips-for-making-ml-engineering-and-mlops-real) (Andy McMahon, NatWest Group, 48:17)
- [From Idea to Production ML](https://mlopstalks.com/talks/from-idea-to-production-ml) (Lex Beattie, Spotify, 53:18)
- [Tecton Round-table // Get your ML Application Into Production](https://mlopstalks.com/talks/tecton-round-table-get-your-ml-application-into-production) (Kevin Stumpf, Derek Salama, Eddie Esquivel & Isaac Cameron, Tecton, 55:42)
- [The Long Tail of ML Deployment](https://mlopstalks.com/talks/the-long-tail-of-ml-deployment) (Tuhin Srivastava, Baseten, 50:37)