# MLOps Build or Buy, Startup vs. Enterprise?

Aaron Maurer & Katrina Ni, Slack | MLOps Podcast | Episode 157 | 49:57
Hosted by Jake Noble

Source: https://www.youtube.com/watch?v=IC2uilYf1sc
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/mlops-build-or-buy-startup-vs-enterprise
Published: 2023-05-09
Tags: build-vs-buy, feature-engineering, platform-teams, recommender-systems

## TL;DR
- Slack built a general Recommend API because many teams needed recommendations for users and channels, rather than relying on one large recommendation use case.
- Slack avoids using customer message text as a normal model feature because a model could reveal private business data across customers.
- Slack chose to build some machine learning infrastructure internally when integrating an external service required more work than building a service that fit its own systems.

## Summary
Aaron Maurer and Katrina Ni describe how Slack built a shared recommendation platform for many product teams. The platform exposes a simple API for requesting recommendations, while the ML Services team handles candidate retrieval, feature selection, weighting, model training, and serving behind it. They explain how this interface makes it easier for product managers and engineers to try ML without learning about embeddings or similarity search. Privacy shapes the system. Slack generally avoids building features from customer message text and prefers interaction signals such as relationships between users and channels. For cold starts, the team begins with hand-tuned models, logs the online features, and later trains ML models from that data. The conversation also covers Slack's vertically integrated ML engineering roles and the build-versus-buy decision. Aaron says Slack built internal serving infrastructure because integrating an external service was more difficult than building one that matched Slack's policies and existing systems.

## Key ideas
### Slack needed a shared recommendation system because teams had many smaller use cases
[07:03](https://www.youtube.com/watch?v=IC2uilYf1sc&t=423s)
Aaron Maurer says Slack's recommendation work grew from a pattern that appeared across many products. Instead of one prominent recommender, teams wanted to recommend users or channels in many different contexts. Slack therefore created a generic API that other teams could call, then expanded the system into tools for building recommenders, training models, and producing training data. Search had been Slack's largest historical ML investment, and some data features could be reused, but recommendation had different needs and led to a separate approach.

### A simple API helped product teams adopt machine learning
[10:53](https://www.youtube.com/watch?v=IC2uilYf1sc&t=653s)
The recommendation platform hides concepts such as embeddings, cosine similarity, and nearest-neighbor search behind a basic request and response. Aaron says teams could ask for recommendations by sending a query with context about users or channels, then receive recommendations without handling the underlying ML system. Katrina adds that a new recommender could often be added within a week, with a test page that let product managers inspect results. Seeing useful output made it easier to compare the system with existing heuristics and build trust.

### The platform reuses common features while allowing different candidate sources
[16:10](https://www.youtube.com/watch?v=IC2uilYf1sc&t=970s)
Katrina explains that Slack commonly recommends channels and users, and the team has a shared signal service with features about both. A new use case usually requires deciding which features to use, how to weight them, and how to fetch candidates. Some cases can start with a simple operational database query, such as finding channels a user belongs to. Discovery use cases may need an embedding store to find nearby channels. Slack has reusable components, but the team does not use exactly the same retrieval process for every case.

### Privacy rules out many direct uses of customer message text
[18:51](https://www.youtube.com/watch?v=IC2uilYf1sc&t=1131s)
Aaron says the damage from leaking customer data would outweigh the value of many ML features, since Slack contains business communications. The team therefore usually avoids creating features directly from message text and instead uses interaction patterns, such as which users and channels work together. Training also has to prevent models from learning customer content and reproducing it elsewhere. For autocomplete, Slack can extract likely suggestions from a team's own text and use aggregate signals such as how often a phrase appears, rather than training a generative model directly on team text.

### Hand-tuned models solve cold starts and give teams an understandable baseline
[24:06](https://www.youtube.com/watch?v=IC2uilYf1sc&t=1446s)
Katrina describes starting each recommendation use case with a hand-tuned model. The team can explain a recommendation using familiar signals, such as how often two users interact, and product managers can understand why a feature received a larger weight. The online system logs the features it computes, which later provides training data for an ML model. Jake Noble says this approach also gives the team a benchmark to beat. Aaron adds that Slack keeps the hand-tuned model as a simple fallback if the serving system fails.

### Logging online features avoids reproducing feature logic offline
[26:33](https://www.youtube.com/watch?v=IC2uilYf1sc&t=1593s)
Jake Noble describes Slack's approach as log and weight feature engineering. Features are computed online during serving and logged for later model training. This can avoid reproducing the same feature calculations in an offline pipeline, along with the associated backfills and training-serving mismatches. Jake says the method fits recommendation systems that can collect enough examples within a short training window. Aaron agrees that logging the online values can save substantial engineering time, although reproducing features offline can still be worthwhile in some cases.

### Slack's ML engineers cover the full path from data to production
[32:34](https://www.youtube.com/watch?v=IC2uilYf1sc&t=1954s)
Katrina and Aaron explain that Slack's ML engineers work across data engineering, feature pipelines, model training, model serving, and application integration. People still develop different strengths. Some focus more on infrastructure or GPU systems, while others come from data science and spend more time exploring models. Aaron says the structure lets software engineers grow toward ML and data scientists grow toward software engineering. Pairing people with different strengths helps the team cover a wide technical surface without requiring every engineer to know every part equally well.

### Large companies may build infrastructure when integration costs exceed implementation costs
[36:41](https://www.youtube.com/watch?v=IC2uilYf1sc&t=2201s)
Katrina says Slack replaced an external model-serving service with internal infrastructure after dealing with integration work, company policies, the Salesforce merger, and a long process for ending the external contract. Aaron frames the decision as a comparison between the work required to integrate a service and the work required to build one. He would prefer engineers to spend time on work specific to Slack, but infrastructure can still be specific enough to Slack's systems and constraints that building it is the easier option. Jake says the choice may change as ML infrastructure products become easier to integrate.

## Notable quotes
- Aaron Maurer: "The value we can create with ML would outweigh the value we could destroy if we leak customers' data." (19:11)
- Aaron Maurer: "When we distilled it down to this recommend API, I think it became a lot easier to partner with other teams and deliver stuff." (12:46)
- Katrina Ni: "We can always start with a hand-tuned model, and then why is that? We can gather training data." (24:39)
- Aaron Maurer: "Is it more work to integrate with someone, or is it more work to build it ourselves?" (38:57)
- Aaron Maurer: "We don't even want to hint at the existence of a DM between you and another person through a model." (48:47)

## Tools & references mentioned
- Slack
- Salesforce
- YouTube
- Airbnb
- Parsec
- Tecton
- Google
- AWS
- OpenAI
- GPT
- Kubernetes
- ML Ops

## Who should watch
- You are building ML features for several product teams and need an interface that keeps adoption simple.
- Your recommendation or search system handles sensitive B2B data and you need practical ways to limit what models can learn from customer content.
- You are deciding whether to buy ML infrastructure or build it inside a larger company with existing platforms, policies, and integration requirements.

## Editor's note

Aaron Maurer says Slack built internal model-serving infrastructure when integrating an external service took more work than building one that fit Slack's systems and policies. ZenML lets teams define workflows as Python pipelines and run the same code on different infrastructure through configuration. That gives a platform team a shared workflow without tying every use case to one serving setup.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Build vs Buy an ML Platform](https://mlopstalks.com/talks/build-vs-buy-an-ml-platform) (Diego Oppenheimer, Algorithmia, 57:20)
- [Investing in MLOps](https://mlopstalks.com/talks/investing-in-mlops) (Leigh Marie Braswell, Founders Fund & Davis Treybig, Innovation Endeavors, 48:52)
- [MLOps vs ML-as-a-Service](https://mlopstalks.com/talks/mlops-vs-ml-as-a-service) (Jill Chase & Manmeet Gujral, CapitalG, 11:27)
- [The Shipyard: Lessons Learned While Building an ML Platform](https://mlopstalks.com/talks/the-shipyard-lessons-learned-while-building-an-ml-platform) (Joseph Haaga, Interos, 40:00)
- [Who's MLOps for Anyway?](https://mlopstalks.com/talks/whos-mlops-for-anyway) (Jonathan Rioux, EPAM Systems, 1:10:15)