# MLOps Investments

Sarah Catanzaro, Amplify Partners | MLOps Coffee Sessions | Episode 33 | 46:18

Source: https://www.youtube.com/watch?v=twvHm8Fa5jk
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/mlops-investments
Published: 2021-04-06
Tags: data-engineering, monitoring, open-source, platform-teams

## TL;DR
- MLOps teams often struggle with data management, especially the relationship between structured metadata and unstructured images, text, and video.
- Open source is a product and a strategic choice. Startups need to maintain the project, respond to users, and decide whether they want contributions, distribution, or simply free software.
- MLOps tooling has too many point products today, and consolidation will happen where tightly coupled products provide enough value to justify switching costs.

## Summary
Sarah Catanzaro describes how her experience leading data teams shaped her work as a partner at Amplify Partners. She discusses the operational pain she saw at Mattermark, where model and pipeline failures forced the team to manually review thousands of articles. That experience informed Amplify's investment in Flywheel. She explains why companies adopt open source, including security concerns, distribution, and the need for extensibility, while stressing that open source software requires ongoing product and community work. The conversation then turns to MLOps consolidation. Sarah expects point products to blend where the connection between them is strong enough to justify switching costs, while rejecting the idea that every company should adopt an end-to-end platform. Her broadest argument concerns structured and unstructured data. Research increasingly works with text, images, and video, while many industry systems still depend on structured data. Better tools for managing their relationship could make machine learning teams more productive and help researchers work with more realistic benchmarks.

## Key ideas
### Sarah stays close to technical work through research and volunteer data work
[06:30](https://www.youtube.com/watch?v=twvHm8Fa5jk&t=390s)
Sarah says Amplify Partners invests in technical tools and platforms across machine intelligence, distributed systems, and data management. To stay close to the problems faced by technical founders and buyers, she curates Projects to Know with three research papers, three open source projects, and three practitioner-oriented pieces of content. She also volunteers as the data lead for All Raise. That work keeps her involved in database schema changes, data quality, cross-team coordination, and the difficulty of getting resources from other groups. She says these activities help her retain empathy for the people building and using the tools in which she invests.

### Model failures at Mattermark showed Sarah the cost of weak monitoring
[10:00](https://www.youtube.com/watch?v=twvHm8Fa5jk&t=600s)
At Mattermark, Sarah's team used NLP fact-extraction models to collect information about startup funding events from news articles. When the system failed, the team had to arrive early and read thousands of articles before customers started work. She says the failures happened at least once a quarter. Some came from data drift, while others came from the ML pipelines, with Mesos outages being a common cause. This experience informed Amplify's investment in Flywheel. Sarah describes its evaluation store as useful for monitoring and for stabilizing pipelines, so teams can address issues before an outage rather than respond afterward.

### ML platform teams usually grow out of repeated work by ML engineers
[12:38](https://www.youtube.com/watch?v=twvHm8Fa5jk&t=758s)
Sarah says most companies in 2021 did not yet have an ML platform team, and many did not have a data platform team either. She expects data platform work to develop first. When an ML platform group does appear, it is often made up of former ML engineers who have encountered the same problem repeatedly and want to solve it systematically. The two roles therefore have similar technical concerns. In startups, Sarah sees one difference: platform engineers have more reason to build internal solutions instead of buying them, because creating those systems is part of the team's purpose.

### Open source decisions should match the company's actual goals
[16:36](https://www.youtube.com/watch?v=twvHm8Fa5jk&t=996s)
Sarah gives several reasons companies publish open source software. Startups may want customers to feel that the technology will outlive the company, and open source can address some security concerns or provide a distribution channel because ML practitioners often search GitHub first. Other projects fit the philosophy of open source more directly because they need extensions, integrations, or community contributions. Sarah advises companies to decide whether users need to see the code, whether the software should be free, and whether the company is ready to accept contributions. Free software without community participation does not require an open source product.

### A startup must treat its open source project as a maintained product
[20:57](https://www.youtube.com/watch?v=twvHm8Fa5jk&t=1257s)
Sarah says an open source project cannot be published and then forgotten. The company must maintain it and respond to community requirements and issues. For an early-stage startup, she often recommends focusing the whole team on the open source project until it has meaningful adoption and engagement. After that, the company can shift attention toward the commercial platform, while still keeping at least a 50-50 split in many cases. Engineers need to interact with users so the product reflects how it is being used. Developer relations can help with applications and best practices, although Sarah says ML makes this role hard to hire for because it requires a baseline understanding of both mathematics and software practices.

### MLOps consolidation will happen where product connections justify switching
[29:41](https://www.youtube.com/watch?v=twvHm8Fa5jk&t=1781s)
Sarah sees a trade-off between flexibility, efficiency, and control in the current MLOps market. Teams often accumulate point products as each new problem appears, then have to maintain integrations and vendor relationships. End-to-end platforms reduce that burden but may fit some workflows poorly. Sarah expects point products to blend in areas such as experiment management with CI/CD, or model monitoring with debugging. She uses a framework from Eric Bernhardsson: two products should be tightly coupled only when the benefits of adopting a unified solution outweigh the cost of switching. Distributed training and model serving may not meet that test if improving one does not materially improve the other.

### Many MLOps problems are data management problems
[37:04](https://www.youtube.com/watch?v=twvHm8Fa5jk&t=2224s)
Sarah says companies are realizing that their ML problems often concern where data is, how it can be accessed, and how data drift affects prediction quality. These are data problems, although generic data engineering tools do not always fit ML use cases. She is especially interested in the gap between structured and unstructured data. Research increasingly applies transformers to text, images, and video, while industry use cases often rely on structured data. For a video dataset, an ML engineer needs ways to explore its contents and understand its variations before choosing a model or preprocessing steps. Better tools for unstructured data could make that work practical.

### Industry and academia need to share better structured benchmarks
[43:03](https://www.youtube.com/watch?v=twvHm8Fa5jk&t=2583s)
Sarah agrees that research models built around very large unstructured datasets can be difficult to use in industry, where teams need to understand model weaknesses, affected classes, and operational risk. She credits Andrew Ng with pointing out the reverse problem: academic researchers often lack access to structured-data benchmarks. Companies are reluctant to share realistic data and the context in which ML operates, while industry professionals criticize academics for optimizing against benchmarks with limited practical meaning. Sarah hopes better access to structured datasets will support useful advances in deep learning for structured data and encourage frameworks aimed at problems that companies actually face.

## Notable quotes
- "I do find that companies are creating open source projects not because anybody really cares about having visibility into the code base, but because they want to be one of the first solutions that ML engineers or data scientists find." (18:54)
- "An open source product is a product. It's one that you need to maintain, it's one where you have to respond to the community's requirements and other issues." (20:34)
- "You can think about two separate point products or two separate parts of the stack and consider, are the benefits of those two components being tightly coupled enough that you can overcome the switching cost?" (31:59)
- "Many companies have realized that their ML problems are actually data management problems." (37:04)
- "If transformers are really the new dimension or the big wave in ML, what we're going to potentially see in MLOps in the coming years are better data management solutions, as well as deployment and engineering solutions for working with unstructured data." (40:36)

## Tools & references mentioned
- Amplify Partners
- Projects to Know
- All Raise
- Mattermark
- Flywheel
- Mesos
- Kubernetes
- Andrew Ng
- Eric Bernhardsson
- Better
- Lyft
- Horovod
- OpenAI
- Wikimedia

## Who should watch
- You are deciding whether an MLOps product should be open source, open core, or commercial, and need a view of the work that community adoption creates.
- Your team is collecting separate MLOps tools and wants a way to judge when integrating or replacing them is worth the switching cost.
- You work with images, text, or video and need better links between those assets and the structured metadata used to operate ML systems.

## Editor's note

Sarah Catanzaro says Mattermark's model and pipeline failures forced the team to read thousands of articles before customers started work. ZenML records each run's steps, inputs, outputs, and code version, so teams can trace an artifact or model back to the data and code that produced it. Unchanged steps can be cached rather than recomputed.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Investing in MLOps](https://mlopstalks.com/talks/investing-in-mlops) (Leigh Marie Braswell, Founders Fund & Davis Treybig, Innovation Endeavors, 48:52)
- [The Current MLOps Landscape](https://mlopstalks.com/talks/the-current-mlops-landscape) (Nathan Benaich, Air Street Capital & Timothy Chen, Essence VC, 58:31)
- [Most Underrated MLOps Topics](https://mlopstalks.com/talks/most-underrated-mlops-topics) (Marian Ignev, CloudStrap.io & SashiDo.io, 53:56)
- [ML Unicorn Start-up Investor Tells-IT-All](https://mlopstalks.com/talks/ml-unicorn-start-up-investor-tells-it-all) (George Mathew, Insight Partners, 51:01)
- [MLOps in Practice: Common Challenges and Lessons Learned](https://mlopstalks.com/talks/mlops-in-practice-common-challenges-and-lessons-learned) (Marouen Hizaoui & Mo Basirati, Machine Learning Reply, 1:03:01)