# Towards Observability for ML Pipelines

Shreya Shankar, UC Berkeley | MLOps Coffee Sessions | Episode 75 | 57:10
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=V8U1HksRr_k
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/towards-observability-for-ml-pipelines
Published: 2022-01-21
Tags: data-engineering, debugging, monitoring, observability

## TL;DR
- Shreya Shankar argues that ML teams should start observability with one business-related service-level indicator, then use other metrics to explain changes in that indicator.
- She proposes treating ML monitoring as a data management problem, where predictions, labels, identifiers, windows, and joins are handled across delayed and distributed sources.
- Shreya Shankar says startups often build tools in-house because nobody is rewarded for the long process of evaluating and adopting an external tool, while working infrastructure and incentives matter more than specialized hardware.

## Summary
Shreya Shankar describes how her work across Google Brain, an early-stage startup, and UC Berkeley shaped her view of ML systems. She separates infrastructure problems, such as clusters and job scheduling, from the harder work of getting people to collaborate and agree on what to build. The conversation then turns to ML observability. Shreya argues that teams often collect thousands of feature statistics, divergences, and model metrics without knowing what action the data should trigger. They should first define a business-linked service-level indicator, such as daily recall, and use drift and feature statistics to investigate when that indicator changes. She frames the technical challenge as a database problem involving delayed labels, imperfect joins, multiple data sources, and materialized views. Her open source project, mltrace, is intended to provide bolt-on observability and end-to-end traces. She also advises ML engineers to learn databases, data management, scheduling, and cloud architecture before adding more specialized tools.

## Key ideas
### ML infrastructure is easier to build than the collaboration around it
[04:05](https://www.youtube.com/watch?v=V8U1HksRr_k&t=245s)
Shreya separates ML systems problems into infrastructure and collaboration. Modern cloud tools make it relatively easy to set up Kubernetes or Spark clusters and arrange job scheduling. The harder problem is deciding what teams should build and aligning data scientists, ML engineers, and product stakeholders around it. She says Google has strong structures for incorporating ML into products, including career ladders and incentives that reward this work. Smaller companies usually lack those structures. At her startup, it was unclear how people could progress individually, and infrastructure work had to compete with the pressure to get a product working.

### Company incentives shape which ML tools get built
[09:15](https://www.youtube.com/watch?v=V8U1HksRr_k&t=555s)
At Viaduct, Shreya reframed OKRs for infrastructure projects around reducing AWS costs or improving developer productivity. That focus helped the team choose work with a direct operational payoff. She gives feature computation as an example. Refreshing stale features could take four hours, which made the system painful for data scientists and became a developer productivity problem. Shreya also explains why startups often build tools internally. Evaluating Airflow, Argo, Kubeflow, or Prefect can take months, and a small startup may not reward anyone for running that evaluation and winning stakeholder agreement. People are rewarded for having something working, so building it themselves can look like the faster path.

### Monitoring should begin with the business metric people already use
[17:54](https://www.youtube.com/watch?v=V8U1HksRr_k&t=1074s)
Shreya argues that teams should not begin by monitoring KL divergences across tens of thousands of features. They should define the service-level indicator that provides business value, including its metric function and time window. At Viaduct, stakeholders actually looked at a Metabase dashboard showing cumulative recall for a specific problem. The metric was computed daily, and people understood how it mapped to return on investment. By contrast, CloudWatch alerts existed for many feature-table aggregations, but Shreya ignored them because they did not tell her what action to take. A single trusted metric can be the starting point for deciding when investigation is needed.

### Secondary metrics should explain a change in the primary indicator
[19:56](https://www.youtube.com/watch?v=V8U1HksRr_k&t=1196s)
Once a team has a meaningful primary indicator, drift and other diagnostic measurements become useful because they help explain why that indicator is changing. Shreya describes a common failure mode where teams monitor precision, recall, F1, false-positive rate, and true-positive rate over many sliding windows. That can produce 150 numbers on a chart without clarifying what anyone should do. At Viaduct, when the important recall graph fell, Shreya would inspect the other alerts. Often she did not know how to interpret a changed percentile for an unimportant feature, so she retrained on a recent window and hoped it worked. She connects this uncertainty to the lack of a standard on-call process at smaller companies.

### ML monitoring is a database problem because its data arrives late and separately
[28:48](https://www.youtube.com/watch?v=V8U1HksRr_k&t=1728s)
Shreya frames observability as a data management problem. Predictions and feedback may live in different sources, arrive at different times, and require joins over identifiers and time windows. In a recommendation system, for example, the pipeline producing recommendations can be separate from the pipeline collecting user feedback. Some predictions have labels while others do not, yet the team still wants a real-time indicator and confidence interval. The result may also need subgroup breakdowns, such as users in different regions. Shreya imagines a monitoring database interface where systems log outputs, labels, and identifiers, while incremental view maintenance handles the underlying materialized views.

### Pipelines are a better monitoring unit than individual models
[35:03](https://www.youtube.com/watch?v=V8U1HksRr_k&t=2103s)
Shreya says that teams should monitor ML pipelines rather than individual models, especially when systems contain chains of models or embeddings that feed downstream components. When a problem can be fixed without changing the model, that is valuable from a business perspective. Her database-oriented design would put much of the work below the model layer. The system would combine outputs and feedback, account for label lag, and produce a trusted indicator. This approach also fits the range of ML use cases better than assuming that every model has the same monitoring needs. She accepts that robotics, fraud detection, computer vision, and tabular systems may need different measures, while still sharing data management concerns.

### mltrace aims to add open source observability after deployment
[36:25](https://www.youtube.com/watch?v=V8U1HksRr_k&t=2185s)
Shreya introduces mltrace as an open source project for bolt-on observability in ML pipelines. It already includes end-to-end tracing for debugging, while she was still working out the most useful monitoring value proposition. She divides ML data management into three areas: pre-training work such as feature stores and ETL, experiment tracking for choosing models, and everything after deployment. She says the first two areas have strong tools, while post-deployment observability has not made the same progress. She compares the desired simplicity to MLflow and wants a clean, open source format. Her plan was to work with users, integrate the project into real use cases, and learn from forward-deployed engineering.

### ML engineers should learn databases before adding more ML-specific layers
[47:49](https://www.youtube.com/watch?v=V8U1HksRr_k&t=2869s)
For people entering ML engineering, Shreya recommends learning basic database and data management principles. She suggests knowing how to set up a Postgres database, then learning about job queues, execution, and how to let data scientists run jobs. Cloud data architecture matters too. She recalls how a scratch S3 bucket can become an architectural problem once a company has multiple clients and needs isolation, separate buckets, and a clear file-system model. Her preferred style of ML engineering focuses on enabling data scientist productivity and making models easier to debug. She says she is no longer interested in spending her time training models.

## Notable quotes
- Shreya Shankar: "The thing that I think is super lacking and rightfully so because it's a really hard problem is the whole collaboration piece." (04:49)
- Shreya Shankar: "We shouldn't be monitoring KL divergences for tens of thousands of features that doesn't make any sense." (18:34)
- Shreya Shankar: "I think it makes much more sense from a business perspective to monitor pipelines and not individual models." (36:03)
- Shreya Shankar: "Understand databases, data management, the patterns and that." (47:49)
- Shreya Shankar: "We are in the era of the hundred million dollar seed round. I think it's terrible for innovation." (52:45)

## Tools & references mentioned
- Google Brain
- UC Berkeley
- RISE lab
- Viaduct
- Amplify Partners
- Google
- YouTube
- Kubernetes
- Spark
- Airflow
- Argo
- Kubeflow
- Prefect
- Prometheus
- Grafana
- Metabase
- CloudWatch
- S3
- mltrace
- MLflow
- Benedict Evans
- Oliver Sacks
- Postgres

## Who should watch
- You are deciding what ML metrics to put into production and need a way to avoid alert fatigue.
- Your predictions and labels arrive through separate pipelines, with delayed feedback or difficult joins.
- You work at a small ML company and need to choose between adopting infrastructure tools and building them in-house.

## Editor's note

Shreya Shankar argues that teams should start with a business-linked service-level indicator instead of collecting thousands of feature statistics without knowing what action to take. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace a result back to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [DevOps, Security, and Observability in ML](https://mlopstalks.com/talks/devops-security-and-observability-in-ml) (Luke Marsden, MLOps Consulting, 32:46)
- [ML Observability](https://mlopstalks.com/talks/ml-observability) (Aparna Dhinakaran, Arize AI, 55:04)
- [Model Monitoring in Practice: Top Trends](https://mlopstalks.com/talks/model-monitoring-in-practice-top-trends) (Krishnaram Kenthapadi, Fiddler AI, 51:34)
- [The Future of ML and Data Platforms](https://mlopstalks.com/talks/the-future-of-ml-and-data-platforms) (Michael Del Balso, Tecton, 55:17)
- [From Idea to Production ML](https://mlopstalks.com/talks/from-idea-to-production-ml) (Lex Beattie, Spotify, 53:18)