# Machine Learning at Reasonable Scale

Jacopo Tagliabue, Coveo | MLOps Coffee Sessions | Episode 66 | 1:04:32

Source: https://www.youtube.com/watch?v=qbAo8mo88Ic
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/machine-learning-at-reasonable-scale
Published: 2021-12-08
Tags: build-vs-buy, data-pipelines, data-quality, recommender-systems

## TL;DR
- Most teams should start with a simple, vertically scaled system and add distributed infrastructure only when a real constraint requires it.
- Teams gain more from improving data quality and domain understanding than from making small changes to model architecture at the start.
- Raw company data should be stored as an append-only history, then transformed into replayable tables for analytics and machine learning.

## Summary
Jacopo Tagliabue argues that most machine learning teams are learning from companies operating at a scale they will never need. The practical problem for many practitioners is still getting from a notebook to a production model. At reasonable scale, teams can use managed infrastructure, keep data work close to the business domain, and run many workloads on a single larger machine. Jacopo recommends working backwards from the business goal and constraints, then buying infrastructure that someone else can maintain. He describes a data pattern in which raw events are retained and later transformed into clean snapshots, so mistakes can be corrected without losing history. He also discusses recommender systems, where established collaborative filtering tools cover common cases while neural approaches remain expensive and harder to operate. The conversation ends with his view that the ML ecosystem is becoming easier to enter, while testing, monitoring, bias, and robustness will become more important as more teams reach production.

## Key ideas
### Most ML discussions focus on companies operating far beyond the needs of ordinary teams
[01:59](https://www.youtube.com/watch?v=qbAo8mo88Ic&t=119s)
Jacopo compares the field to people trying to learn tennis while mostly discussing Roger Federer's training. Teams hear about what Google, Facebook, YouTube, Uber, and other large companies do, even though many practitioners are still trying to move their first model from a notebook into production. He defines reasonable scale through both limits and benefits. A smaller team cannot hire Google-sized staff or manage petabytes of data, but it also does not need to serve billions of users. Building a recommender system for an e-commerce company with substantial revenue can be difficult while remaining a manageable engineering problem.

### Managed services can remove distributed computing from many production pipelines
[07:10](https://www.youtube.com/watch?v=qbAo8mo88Ic&t=430s)
Jacopo uses his experience with Spark as an example of infrastructure that can impose more complexity than a smaller team needs. Writing Spark code for aggregations and feature preparation requires unfamiliar patterns, difficult debugging, and many configuration choices. His current setup abstracts that work away and pushes large-data operations into Snowflake. The resulting ML dataset is often small enough to fit on a laptop or a small cloud machine. He says this lets a team build an e-commerce recommender without horizontal scaling, saving time and reducing the expertise needed for infrastructure maintenance.

### Teams should work backwards from the product goal and spend their time on data
[12:54](https://www.youtube.com/watch?v=qbAo8mo88Ic&t=774s)
Jacopo recommends starting with the required predictions, the available input data, the expected traffic, and the business goal. After those constraints are clear, the team can choose services and tools rather than maintaining infrastructure for its own sake. He puts the team's effort into understanding the data, checking that it is correctly formatted, and applying domain knowledge. A plug-in model will not automatically fit a company's use case. He also wants model iteration to be fast and enjoyable, with a simple architecture that works end to end before the team spends time on marginal architectural improvements.

### A managed platform is often cheaper than using engineers to maintain infrastructure
[16:18](https://www.youtube.com/watch?v=qbAo8mo88Ic&t=978s)
Jacopo's advice is to buy almost everything except the parts that affect the company's evaluation or business metric. He considers data work a source of lasting advantage because a team that understands its domain data can create a barrier that competitors must spend time rebuilding. Maintaining Kafka, scaling clusters, or operating Spark does not create that advantage in his view. He also argues that infrastructure costs are easier to predict than the productivity of additional engineers. A service may cost more directly, but engineers who maintain it lose time that could have gone into work that adds value.

### Raw event history should be preserved before transformations are applied
[23:59](https://www.youtube.com/watch?v=qbAo8mo88Ic&t=1439s)
Jacopo describes the data pattern behind his four principles as storing company data in a raw, append-only form and transforming it later with tools such as dbt. The raw history acts like a ledger of what happened inside and outside the company. Clean snapshots can then feed BI systems and ML models. If a transformation is wrong, the team can delete the derived outputs and rebuild them from the retained history. He contrasts this with overwriting a product's price each day, which loses the old value. Keeping each price makes it possible to reconstruct the product's history and treat data events more like versioned code.

### Data checks need both structural rules and knowledge of the domain
[31:41](https://www.youtube.com/watch?v=qbAo8mo88Ic&t=1901s)
For changing transformations and comparing snapshots, Jacopo recommends automated checks wherever possible. Structural checks can verify that columns are not null, values are unique where required, and data types remain valid. Semantic checks examine whether the shape and values of the data still fit reasonable expectations. He mentions Great Expectations as an open-source tool for this kind of validation. The harder part is deciding whether a new output is actually better for the business. That requires understanding the data, and Jacopo accepts that this work cannot be replaced by clever tooling.

### Simple vertical systems should come before distributed or neural systems
[40:50](https://www.youtube.com/watch?v=qbAo8mo88Ic&t=2450s)
Jacopo says teams often choose distributed computing because it was historically the only practical way to process large datasets. Cloud warehouses and current computing options have changed that trade-off, but existing systems and habits can keep teams on the old path. His recommendation is to start with a large machine and a simple approach, such as collaborative filtering for recommendations. A team can later add distributed processing or a transformer model if a specific limitation justifies the cost. Building the simpler version first also develops skills and intuition that transfer to a more complex system.

### Recommender systems have mature common-case tools, while neural systems remain expensive
[43:02](https://www.youtube.com/watch?v=qbAo8mo88Ic&t=2582s)
Jacopo separates recommender systems into two levels. Similar-item, complementary-item, and collaborative-filtering use cases have libraries and practices that can take a team from zero to a working system quickly, including support for cases such as cold starts. Neural recommender systems are more common in very large or highly sophisticated companies, and their practical use remains closer to an art than a settled process. He gives an example where a transformer recommender performed better than a simpler model but cost far more to train. For many companies, that marginal gain may not justify the extra cost and effort.

### Session-based recommendations matter because many shoppers are anonymous
[50:50](https://www.youtube.com/watch?v=qbAo8mo88Ic&t=3050s)
Coveo works on content and commerce recommendations, including complementary products and session-based recommendations. Jacopo points out that complementary recommendations have asymmetric rules. Suggesting an HDMI cable after someone buys a television can make sense, while suggesting a $600 television to someone who only has an HDMI cable in the cart may be a poor recommendation. Session-based systems use what a person is clicking on now rather than relying on a long history tied to a known account. This matters especially for smaller e-commerce sites because many visitors are anonymous and log in only near checkout.

## Notable quotes
- "We're all trying to learn how to play tennis but the only thing we discuss most of the time is Roger Federer training." (02:24)
- "There's no value in the infrastructure at reasonable scale because somebody else is doing that better than you are." (13:41)
- "Store everything as it comes in raw and then transform it with a replayable DAG later." (26:49)
- "Start simple. You can always go build a transformer model down the road." (40:26)
- "It's a fantastic time to be in this field. It's the best moment, really, the present." (59:44)

## Tools & references mentioned
- Coveo
- Tooso
- Roger Federer
- Google
- Facebook
- YouTube
- Uber
- Snowflake
- Spark
- dbt
- Metaflow
- Prefect
- Kafka
- Athena
- Great Expectations
- collaborative filtering
- transformer models
- session-based recommendation
- SIGIR-ecom-data-challenge
- Vicky Boykis
- Luigi Petrone
- Luca
- Patrick
- Andrew
- Andre Enrico

## Who should watch
- You are building your first production ML system and keep comparing your architecture with systems from Google, Facebook, or Uber.
- Your team is spending engineering time maintaining Spark, Kafka, or clusters when the actual ML dataset may fit on a single machine.
- You work on recommendations and need to choose between established collaborative filtering methods and more expensive neural approaches.

## Related talks

- [Building for Small Data Science Teams](https://mlopstalks.com/talks/building-for-small-data-science-teams) (James Lamb, SpotHero, 52:26)
- [A Journey in Scaling ML](https://mlopstalks.com/talks/a-journey-in-scaling-ml) (Gabriel Straub, Ocado Technology, 52:41)
- [Lessons from Studying FAANG ML Systems](https://mlopstalks.com/talks/lessons-from-studying-faang-ml-systems) (Ernest Chan, Duo Security, 45:32)
- [Scaling ML Capabilities in Large Organizations](https://mlopstalks.com/talks/scaling-ml-capabilities-in-large-organizations) (Bertjan Broeksema & Axel Goblet, BigData Republic, 1:02:47)
- [How To Move From Barely Doing BI to Doing AI](https://mlopstalks.com/talks/how-to-move-from-barely-doing-bi-to-doing-ai) (Joe Reis, Ternary Data, 53:47)