# Watching models in production

352 sessions · follows the tags monitoring, observability, evals
Page: https://mlopstalks.com/threads/watching-models-in-production

How do you tell whether a model still works once people depend on it? Speakers look for answers in drift statistics, production traces and evals, including the disputed practice of asking one model to judge another.

## 2020

27 sessions.

- [High Stakes ML: Active Failures, Latent Factors](https://mlopstalks.com/talks/high-stakes-ml-active-failures-latent-factors) (Flavio Clesio, MyHammer AG). Pushed back: Flavio Clesio rejects using large technology companies as the main reliability benchmark for high-stakes systems. [14:06](https://www.youtube.com/watch?v=9g4deV1uNZo&t=846s)
- [MLOps - The Blind Men and the Elephant](https://mlopstalks.com/talks/mlops-the-blind-men-and-the-elephant) (Saurav Chakravorty, Brillo). Pushed back: Saurav Chakravorty argued that continuous retraining is not necessary in every setup, while continuous monitoring of scoring data is necessary. [29:49](https://www.youtube.com/watch?v=RTBq7e3FhEw&t=1789s)
- [Venture Capital in Machine Learning Startups](https://mlopstalks.com/talks/venture-capital-in-machine-learning-startups) (John Spindler, Capital Enterprise). Pushed back: John Spindler disputes the common preference for deep learning by saying that linear regression is often the better choice when it fits the problem. [18:07](https://www.youtube.com/watch?v=v9OvXxTUBtg&t=1087s)
- [Monitoring the Machine Learning Stack](https://mlopstalks.com/talks/monitoring-the-machine-learning-stack) (Lina Weichbrodt, DKB). Pushed back: Lina Weichbrodt said that real-time response monitoring is needed in addition to offline data-quality checks such as Great Expectations or TensorFlow Data Validation. [21:00](https://www.youtube.com/watch?v=Un30yb1WlpU&t=1260s)

23 more from 2020 on this thread: https://mlopstalks.com/threads/watching-models-in-production/2020

## 2021

38 sessions.

- [Model Watching: Keeping Your Project in Production](https://mlopstalks.com/talks/model-watching-keeping-your-project-in-production) (Ben Wilson, Databricks). Pushed back: Ben Wilson argues that the tools used for drift monitoring matter less than knowing which kinds of drift and statistical behavior to monitor. [18:57](https://www.youtube.com/watch?v=7HJ5x-DglLE&t=1137s)
- [MLOps Investments](https://mlopstalks.com/talks/mlops-investments) (Sarah Catanzaro, Amplify Partners). Pushed back: Sarah Catanzaro says industry and academia both contribute to the gap between research and practical ML because industry rarely provides realistic structured-data benchmarks and context. [43:06](https://www.youtube.com/watch?v=twvHm8Fa5jk&t=2586s)
- [Building ML Blocks with Kubeflow Orchestration with Feature Store](https://mlopstalks.com/talks/building-ml-blocks-with-kubeflow-orchestration-with-feature-store) (Aniruddha Choudhury, Publicis Sapient). Pushed back: Aniruddha Choudhury distinguishes a feature store from a SQL database by emphasizing low-latency online retrieval, feature consistency, and support for batch and streaming ingestion. [53:23](https://www.youtube.com/watch?v=UJLQgxr_Za0&t=3203s)
- [mlctl and Hydrosphere Open Source MLOps Libraries Demo](https://mlopstalks.com/talks/mlctl-and-hydrosphere-open-source-mlops-libraries-demo) (Alex Chung, Intuit). Pushed back: Alex Chung questioned whether Hydrosphere should continue serving models or focus on monitoring and integration with existing tools, and recommended the latter focus. [31:41](https://www.youtube.com/watch?v=muk5NkyGtSQ&t=1901s)

34 more from 2021 on this thread: https://mlopstalks.com/threads/watching-models-in-production/2021

## 2022

31 sessions.

- [Towards Observability for ML Pipelines](https://mlopstalks.com/talks/towards-observability-for-ml-pipelines) (Shreya Shankar, UC Berkeley). Pushed back: Shreya Shankar rejects the common practice of monitoring thousands of feature-level KL divergences as the primary way to operate ML systems. [18:14](https://www.youtube.com/watch?v=V8U1HksRr_k&t=1094s)
- [Platform Thinking: A Lemonade Case Study](https://mlopstalks.com/talks/platform-thinking-a-lemonade-case-study) (Orr Shilon, Lemonade). Pushed back: Automatic model monitoring was not considered suitable for Lemonade's process, so data scientists had to configure monitors manually. [14:09](https://www.youtube.com/watch?v=KemCHs7Xbrs&t=849s)
- [Lessons from Studying FAANG ML Systems](https://mlopstalks.com/talks/lessons-from-studying-faang-ml-systems) (Ernest Chan, Duo Security). Pushed back: Ernest Chan pushed back on the idea that shadow mode would necessarily solve the problem of a model facing major COVID-related data drift, saying it might not help the existing model unless that model were turned off. [36:39](https://www.youtube.com/watch?v=Xqme2sr36RU&t=2199s)
- [DataOps is a Software Engineering Challenge](https://mlopstalks.com/talks/dataops-is-a-software-engineering-challenge) (Micha Kunze, Maersk). Pushed back: Micha Kunze says commercial data-observability tooling was not valuable enough for his team's use case because they needed integrated checks that could stop pipelines, rather than only post hoc metrics. [46:09](https://www.youtube.com/watch?v=iAr6woT1oIQ&t=2769s)

27 more from 2022 on this thread: https://mlopstalks.com/threads/watching-models-in-production/2022

## 2023

65 sessions.

- [Want High Performing LLMs? Hint: It Is All About Your Data](https://mlopstalks.com/talks/want-high-performing-llms-hint-it-is-all-about-your-data) (Vikram Chatterji, Galileo). Pushed back: Vikram Chatterji says there is not yet a broadly adequate metric for evaluating prompts across models, while practitioners often default to BLEU scores and similar measures. [31:56](https://www.youtube.com/watch?v=XpeC1dqfiNo&t=1916s)
- [Embeddings and Retrieval for LLMs: Techniques and Challenges](https://mlopstalks.com/talks/embeddings-and-retrieval-for-llms-techniques-and-challenges) (Anton Troynikov, Chroma). Pushed back: Anton Troynikov argued that heuristics in robotics and retrieval systems are brittle and that models should eventually handle more of the required tasks. [32:54](https://www.youtube.com/watch?v=kZeOPapQ8yM&t=1974s)
- [Evaluating LLM-based Applications](https://mlopstalks.com/talks/evaluating-llm-based-applications) (Josh Tobin, Gantry). Pushed back: Josh Tobin disputes the common view that human evaluation is always the best way to evaluate language models, arguing that automated model evaluation can be useful and that the best approach combines automated and human evaluation. [13:34](https://www.youtube.com/watch?v=r-HUnht-Gns&t=814s)
- [Evaluation](https://mlopstalks.com/talks/evaluation-panel-1-large-language-models-in-production-conference-part-2) (Amrutha Gujjar, Structured & Josh Tobin, Gantry & Sohini Roy, NVIDIA). Pushed back: Josh Tobin disputes relying on public benchmarks for application development, saying they are nearly useless when they do not measure a company's user data and outcomes. [18:17](https://www.youtube.com/watch?v=e0ZLqfus_TY&t=1097s)

61 more from 2023 on this thread: https://mlopstalks.com/threads/watching-models-in-production/2023

## 2024

80 sessions.

- [Language, Graphs, and AI in Industry](https://mlopstalks.com/talks/language-graphs-and-ai-in-industry) (Paco Nathan, Derwen, Inc.). Pushed back: Paco Nathan disputes the model-centric focus on benchmark scores without equal attention to data quality, cost, security and domain-specific evaluations. [33:10](https://www.youtube.com/watch?v=zDSGctGdB2A&t=1990s)
- [Pioneering AI Models for Regional Languages](https://mlopstalks.com/talks/pioneering-ai-models-for-regional-languages) (Aleksa Gordić, OrtusAI). Pushed back: Demetrios Brinkmann suggested that large language models may already beat humans on many exams, while Aleksa Gordić disputed the interpretation because evaluation data may have appeared in training data. [22:54](https://www.youtube.com/watch?v=IQutfmxzZEQ&t=1374s)
- [LLM Evaluation with Arize AI's Aparna Dhinakaran](https://mlopstalks.com/talks/llm-evaluation-with-arize-ais-aparna-dhinakaran) (Arize AI's Aparna Dhinakaran). Pushed back: Aparna Dhinakaran argues that binary or multiclass evaluations are more useful than numeric score evaluations because LLM scores often have no reliable meaning on a spectrum. [36:50](https://www.youtube.com/watch?v=00AQjBDM46Q&t=2210s)
- [The Real E2E RAG Stack](https://mlopstalks.com/talks/the-real-e2e-rag-stack) (Sam Bean, Rewind.ai). Pushed back: Sam Bean disputes the idea that neural networks should be added to search or evaluation because they automatically make systems simpler. [44:18](https://www.youtube.com/watch?v=8uZst7pgOw0&t=2658s)

76 more from 2024 on this thread: https://mlopstalks.com/threads/watching-models-in-production/2024

## 2025

69 sessions.

- [AI Careers Insights from Ex Meta Staff Eng](https://mlopstalks.com/talks/ai-careers-insights-from-ex-meta-staff-eng) (Ilya Reznik, Instructed Machines, LLC). Pushed back: Ilya Reznik disputes the assumption that a high benchmark score demonstrates real-world usefulness. [18:18](https://www.youtube.com/watch?v=YimHT3UfAQ4&t=1098s)
- [AI Agents: The Future of ML Engineering?](https://mlopstalks.com/talks/ai-agents-the-future-of-ml-engineering) (Matt Squire, Fuzzy Labs). Pushed back: The speakers questioned whether the benchmark's contamination analysis adequately established that model performance was not inflated by memorization. [27:50](https://www.youtube.com/watch?v=ButA1OyQAW8&t=1670s)
- [Web Agents: The Cutting Edge of AI is Here?](https://mlopstalks.com/talks/web-agents-the-cutting-edge-of-ai-is-here) (Paul van der Boor & Chiara Caratelli, Prosus Group). Pushed back: The team found that performance on the WebArena benchmark did not translate reliably to the web-agent tasks they cared about. [17:49](https://www.youtube.com/watch?v=QbxN_PN7kZc&t=1069s)
- [AI in Production 2025 | Keynote](https://mlopstalks.com/talks/ai-in-production-2025-keynote) (). Pushed back: The speaker argued that graph RAG does not have a single established best practice and that retrieval strategies require experimentation. [44:10](https://www.youtube.com/watch?v=ioCIpbAKl_M&t=2650s)

65 more from 2025 on this thread: https://mlopstalks.com/threads/watching-models-in-production/2025

## 2026

42 sessions.

- [Enterprise AI Operations: The Missing Piece](https://mlopstalks.com/talks/enterprise-ai-operations-the-missing-piece) (Rani Radhakrishnan, PwC US). Pushed back: The cost of AI should not be calculated simply as replacing human workers, because review, storage, retrieval, and other operating costs must also be included. [28:48](https://www.youtube.com/watch?v=jTmV_jlob5I&t=1728s)
- [Real-Time Voice Agents in Production](https://mlopstalks.com/talks/real-time-voice-agents-in-production) (Panos Stravopodis, Elyos AI). Pushed back: Panos Stravopodis rejected the assumption that human agents necessarily perform better than AI agents and recommended head-to-head benchmarking. [14:10](https://www.youtube.com/watch?v=Zs8onmsPJgw&t=850s)
- [Structured Dissent Patterns for Agentic Production Reliability](https://mlopstalks.com/talks/structured-dissent-patterns-for-agentic-production-reliability) (Phil Stafford, MLOps Community). Pushed back: The swarm's task-aligned evaluation rated the system highly, while the DPFL benchmark treated the approach as overengineered roleplay because it expected ground truth. [15:31](https://www.youtube.com/watch?v=blOifXIJLe4&t=931s)
- [How AI covered a human's paternity leave](https://mlopstalks.com/talks/how-ai-covered-a-humans-paternity-leave) (Quinten Rosseel, Wobby). Pushed back: Quinten Rosseel disputes the common emphasis on text-to-SQL benchmarks as the main measure of agent success, arguing that business context is the real challenge. [4:36](https://www.youtube.com/watch?v=xa47e9hc4RM&t=276s)

38 more from 2026 on this thread: https://mlopstalks.com/threads/watching-models-in-production/2026