# Product Management in Machine Learning

Laszlo Sragner, Hypergolic | MLOps Meetup | Episode 54 | 57:52
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=Sl7WrlbXf9E
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/product-management-in-machine-learning
Published: 2021-03-05
Tags: continual-learning, human-in-the-loop, monitoring, product-strategy

## TL;DR
- Laszlo Sragner argues that model evaluation, monitoring, and labeling should form one continuous feedback process rather than separate stages.
- Incremental machine learning reduces the cost of failure by getting a production-quality first version into use and updating it as labelers, users, and business conditions reveal problems.
- Machine learning product management must connect business problems with data science, engineering, and ongoing model maintenance, while keeping machine learning products separate from MLOps products.

## Summary
Laszlo Sragner describes how quant finance and software engineering shaped the way his team built machine learning at a fintech startup. He contrasts the usual waterfall process, where teams collect and clean data, train a model, evaluate it, and deploy it, with an incremental process that gathers feedback throughout development. His team used human labeling, model-generated examples, active learning, version control, and model candidates to update systems as they learned more about the problem. Sragner also separates machine learning from MLOps. Machine learning teams solve business problems with models, while MLOps teams build products that help others use those models. He argues that both areas need product ownership. Industrial machine learning also needs more than a state-of-the-art model. It needs an affordable, maintainable solution that fits the business, with engineering and domain experts working together.

## Key ideas
### Finance made model monitoring part of the product process
[09:29](https://www.youtube.com/watch?v=Sl7WrlbXf9E&t=569s)
Sragner says his quant background taught him to treat evaluation, monitoring, and labeling as one tightly connected process. In a quant fund, a model is never placed on a shelf as finished because its financial result gives immediate feedback. When performance changes, the team asks whether to replace the model, creates candidates, and labels new data. At Arkera, this meant labeling data on demand while monitoring model performance and creating a feedback loop. He describes the resulting process as an ad hoc active learning loop. The approach is less necessary for a stationary problem such as recognizing cats, but it matters when news content, business conditions, or other inputs change.

### Machine learning and MLOps need separate product ownership
[15:02](https://www.youtube.com/watch?v=Sl7WrlbXf9E&t=902s)
Sragner draws a line between machine learning and MLOps. Machine learning involves creating models that solve business problems. MLOps involves products that allow a business to use machine learning. He says these roles should be separated conceptually or organizationally so each person focuses on the right work. A product manager for MLOps is a customer for an MLOps product, while a machine learning product manager works with the business and the modeling team. In a small startup, one person may wear the MLOps product management hat only part of the week, but the responsibilities still need to be handled coherently.

### The normal machine learning pipeline spends too long waiting for feedback
[17:27](https://www.youtube.com/watch?v=Sl7WrlbXf9E&t=1047s)
Sragner describes traditional machine learning as waterfall-like: collect data, clean it, train a model, evaluate it, and deploy it. Teams spend much of their effort cleaning data, yet they may receive their first business feedback only after deployment. That means the company pays the full cost of the machine learning problem before learning whether the approach works. His team tried iterative machine learning for natural language processing. The first version could reach an acceptable state, with people manually cleaning the remaining difficult cases. Later iterations could use what the team learned from those cases instead of requiring the whole dataset and process to be completed upfront.

### A useful MVP must be production quality and designed for revision
[19:25](https://www.youtube.com/watch?v=Sl7WrlbXf9E&t=1165s)
Sragner uses Snorkel's way of thinking about MVPs: create an initial version, gather feedback quickly, and update the model from that feedback. He is clear that an MVP does not mean cutting corners. In a topic-modeling example, labelers discovered that the original instructions did not cover important edge cases. The team could revise the instructions and relabel data, inspect model-labeled articles, and compare a new candidate with the previous model. Ensemble techniques let them add a component that responded to new feedback while regression testing against the original data. This preserved earlier behavior where needed and made each change visible.

### Small teams can run iterative machine learning with strong engineering practices
[23:46](https://www.youtube.com/watch?v=Sl7WrlbXf9E&t=1426s)
The iterative process did not require a large organization. Sragner's team had three or four data scientists, four labelers, five domain experts, and about three interns. They relied on a strong technical culture, version-controlled code, version-controlled data, and minimal infrastructure. Labeling, data cleaning, training, and evaluation happened at the same time, with different groups working on different components. He says this parallel work is how a team breaks the waterfall structure instead of merely wrapping a waterfall process into a circle. The cost is coordination and engineering discipline, rather than a large platform or a large headcount.

### Industrial machine learning needs a business solution rather than the best model alone
[41:02](https://www.youtube.com/watch?v=Sl7WrlbXf9E&t=2462s)
Sragner contrasts academic and industrial machine learning. Research groups often optimize for publication or state-of-the-art models under their own constraints. An industrial team has to close the gap between what is technically possible and what can make money or operate within the business. A model may need a human check for an unclear fraud case, or the company may need to accept a less advanced technique because it is feasible and maintainable. Product management helps assess the value, cost, risks, false positives, false negatives, and ongoing work. A business problem can remain after a particular model changes, so the machine learning product needs continuing maintenance.

### Machine learning APIs often fail when the business problem is highly specific
[46:38](https://www.youtube.com/watch?v=Sl7WrlbXf9E&t=2798s)
Sragner rejects the idea that a machine learning model's short effective lifetime means the product should only sell a subscription to model updates. The underlying business problem can remain for years, so the models need ongoing maintenance. He also says unique business circumstances make one-size-fits-all API products a poor fit outside narrow cases such as translation. His team tried commercial named entity recognition offerings, but none reached the performance they wanted. In his view, product managers must decide whether machine learning is the right solution, estimate the costs and benefits, and understand how model errors affect the business.

### Notebooks are useful for analysis, but large-scale code belongs in scripts and jobs
[52:58](https://www.youtube.com/watch?v=Sl7WrlbXf9E&t=3178s)
Sragner does not reject Jupyter notebooks completely. He compares them to a whiteboard or an advanced spreadsheet. His team used notebooks to inspect data produced by pipelines, load data into frames, plot it, and analyze labeling and statistics. They stopped putting the main body of code in notebooks because scripts were easier to maintain and use for development. Their exploratory analysis also involved hundreds of gigabytes of text, which required large machines and parallel processing jobs. For that work, a notebook was not a practical execution environment. The team used the results to find changes for the next model iteration.

## Notable quotes
- Laszlo Sragner: "The evaluation, the monitoring and the labeling exercise is actually just one tightly coupled process that's happening at the same time." (10:39)
- Laszlo Sragner: "Machine learning and machine learning operations are not the same thing." (15:24)
- Laszlo Sragner: "MVP doesn't mean terrible." (21:54)
- Laszlo Sragner: "State-of-the-art doesn't mean good quality from a business perspective or feasible money-making solution." (42:17)
- Laszlo Sragner: "Notebooks are whiteboard." (53:40)

## Tools & references mentioned
- Hypergolic
- Humanloop
- Snorkel
- Alex Ratner
- TensorFlow
- Hugging Face
- Neil Luthier
- Monzo
- Todd Underwood
- Jupyter notebooks
- Machine Learning Product Manual

## Who should watch
- You are building machine learning features in a startup and need feedback before spending the full cost of data preparation and model development.
- You manage data scientists, labelers, domain experts, or engineers and need a clearer division of responsibilities between machine learning and MLOps.
- You are a product manager moving into machine learning and want a practical way to assess model costs, errors, maintenance, and business fit.

## Editor's note

Laszlo Sragner argues that teams can spend too long cleaning data before they learn whether a model solves the business problem. ZenML lets teams write the workflow as Python steps and run the same pipeline on different infrastructure. Each run records its inputs, outputs, steps, and code version, so model iterations can be traced and repeated.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [ML Stepping Stones: Challenges & Opportunities for Companies](https://mlopstalks.com/talks/ml-stepping-stones-challenges-opportunities-for-companies) (John Crousse, 47:47)
- [Project/Product Management for MLOps](https://mlopstalks.com/talks/project-product-management-for-mlops) (Korri Jones-Simarpal Khaira-Veselina Staneva, 57:22)
- [ProductizeML: Assisting Your Team to Better Build ML Products](https://mlopstalks.com/talks/productizeml-assisting-your-team-to-better-build-ml-products) (Adrià Romero, ProductizeML, 50:53)
- [Reliable LLM Products, Fueled by Feedback](https://mlopstalks.com/talks/reliable-llm-products-fueled-by-feedback) (Chinar Movsisyan, Feedback Intelligence, 49:17)
- [Managing Machine Learning Projects](https://mlopstalks.com/talks/managing-machine-learning-projects) (Simon Thompson, GFT, 45:02)