Machine learning operations failures usually come from organizational constraints, especially deployment systems that were built for web software.
2
Teams should run experiments quickly before automating a workflow, because automating an unproven process can lock a company into brittle tools and dependencies.
3
Successful enterprise ML work needs a small, weakly coupled area where people can access data and deploy models without passing through every existing enterprise control.
Summary
Charles Martin argues that enterprise ML projects often fail because organizations force new machine learning workflows into deployment systems built for older web applications. A typical ML system pulls data, trains a model, and deploys a microservice, but it also depends on data that may be incomplete, unvalidated, or difficult to access. Existing systems may lack the memory, swap, permissions, dependency management, or documentation that the work requires. Martin is especially critical of automating experiments before anyone knows which process works. He compares this with trading environments, where teams put algorithms into production quickly, keep what makes money, and remove what does not. He recommends a carve-out with weak connections to the rest of the organization, so researchers can test ideas without inheriting every brittle dependency. He also says leaders must want to learn and must give capable people authority to push through internal resistance.
Martin contrasts a successful Slovenian client with projects that fail inside rigid companies. The client taught itself neural networks through a fast.ai class and asked Martin to guide the work rather than do everything for them. Together they built a tool to measure text coherency, since perplexity alone did not answer whether generated text was realistic. Martin says the client had to want to learn, and everyone involved had to engage. Projects fail when a client says its existing deployment process is mandatory and expects the ML team to spend months making a new system fit it.
Martin says MLOps is an organizational problem because enterprises already have deployment systems designed around the software they know. A web application may be a simple Flask API or a group of microservices, while ML has different hardware, memory, swap, and model artifacts. The existing environment may have no way to handle the workflow. Teams then spend their time fitting an ML project into old processes instead of testing the model. Martin describes this as a failure of organizational judgment, especially when people insist that the established process must remain unchanged.
Enterprise infrastructure can make basic ML work take months
Martin gives examples of infrastructure blocking ordinary work. A machine provisioned for an ML application may have no memory or swap because the organization designed it for Hadoop. Python and NumPy may need permissions to configure CPU throttling for the expected performance. One client had built an internal copy of Heroku with no documentation or support, and its dependency rules made using Conda painful. Martin says a simple deployment could have taken a day on an Ubuntu machine with a shell script, while the company's prescribed route took six weeks.
Data access and data quality are part of deployment
Martin describes ML systems as having three basic parts: pulling data from a database, training a model, and deploying a microservice. Enterprise teams often overlook the first part. A training query with many joins cannot safely run against a production database every day, yet replicas may be incomplete or poorly maintained. He describes finding a person backing up a database to a laptop because the formal backup process was impractical. ML data also lacks the validation found at normal application input points, so malformed values can break regular expressions or other processing long after they entered the system.
Experiments should be automated before products are scaled
Martin says a data scientist's immediate goal is to run experiments, since the team does not yet know what will work. He uses BlackRock as an example of a group that put algorithms into production quickly, measured whether they worked, and automated them after they proved useful. In trading, an algorithm that stops working must be removed rather than preserved as a maintenance burden. Martin therefore wants automation to target repeated experimental work and known blockers. Automating every step before the product is understood can preserve obsolete choices and prevent new experiments.
Automation becomes dangerous when it creates tight dependencies
Martin warns that automation can lock an organization to old versions and brittle assumptions. He describes environments where development used Ubuntu 16 while deployment still required Ubuntu 12, and older systems that maintained incompatible Perl versions because the automated setup could not be changed easily. ML changes quickly, so a team may need to move between frameworks or methods. Martin recommends a carve-out with weak coupling to the wider organization. The ML group should be able to get data and deploy an application without inheriting dense dependencies from every existing system.
Leaders must choose people and projects that can change
Martin says companies often announce AI plans because executives want funding, growth, or better earnings guidance, even when they have not committed to doing real ML work. He has seen companies place people without AI experience in charge of AI teams. The clients who succeed are more often those where founders or senior executives actively support the work and overrule internal blockers. When Martin is asked how to educate a company about the ML lifecycle, his answer is to find people who want to learn. The project also needs to make those people look successful, because a failed initiative may not receive another budget.
"What you need to do is create a carve-out from the organization with a weak coupling to the rest of the organization."Charles Martin42:58
Who should watch
You are trying to put ML into an enterprise platform built for ordinary web services and need concrete reasons the existing process is blocking the work.
Your organization wants to automate every ML step before it has proved that the model, data, and experiment are useful.
You lead an ML initiative and need to decide whether the team has executive backing, usable data access, and enough freedom to test ideas.
