# Feedback loops and retraining

A pack of 8 sessions from the MLOps Community YouTube channel, in the order to watch them. 7h 18m of video.
Page: https://mlopstalks.com/packs/feedback-loops-and-retraining

Production errors keep arriving, but the next model trains on the same old examples. Closing that gap takes more than scheduling another training run. Someone must decide which cases deserve review, what the labels mean, and whether the proposed update improves the task. Start with a versioned dataset that receives production examples, then design the selection and annotation process together. Autonomous driving makes the value of rare events concrete; noisy-label analysis prevents corrections from becoming another source of error. Speech-model refreshes connect those findings to new training data. An LLM product example adds caution about interpreting user behavior. Finish with alternatives to automatic retraining and Lyft's fast forecasting updates, where fresh observations change model weights and historical simulation checks the result.

## This pack is for you if

- Your production failures never become reviewed examples in the next training dataset.
- You can afford to label only a small fraction of incoming cases and need to choose them deliberately.
- You retrain on a schedule but cannot explain whether the new data or update cadence helps.

## The talks, in order

### 1. Data-Centric AI Means Centralizing Training Data

Alberto Rizzoli, V7 | 49:01 | MLOps Meetup
Video: https://www.youtube.com/watch?v=DFXVIE8GRF8
Summary: https://mlopstalks.com/talks/data-centric-ai-means-centralizing-training-data.md

Why first: A production example needs a destination before it can improve a model. Rizzoli describes returning low-confidence or pre-labeled items to a versioned training library for human review. His 2021 computer-vision account replaces an informal folder handoff with a deliberate choice of what enters each later dataset.

### 2. Scaling Human-in-the-Loop Machine Learning

Robert Munro | 55:04 | MLOps Meetup
Video: https://www.youtube.com/watch?v=LwbbGsuNpao
Summary: https://mlopstalks.com/talks/scaling-human-in-the-loop-machine-learning.md

Why here: Choosing the next examples and choosing the annotation process are one design problem. Munro combines uncertainty with diversity, then accounts for annotator expertise and disagreement. This prevents the returning-data loop from selecting only familiar uncertain cases while missing confident mistakes or unreliable labels.

### 3. How Sama is Improving ML Models to Make AVs Safer

Duncan Curtis, Sama | 45:35 | MLOps Podcast
Video: https://www.youtube.com/watch?v=UJ6uA46yWzM
Summary: https://mlopstalks.com/talks/how-sama-is-improving-ml-models-to-make-avs-safer.md

Why here: A few unusual seconds can matter more than an uneventful drive. Curtis describes selecting short events, checking coverage and using people to correct automated annotations. The autonomous-vehicle examples make the selection policy tangible: spend review effort on missing conditions and interpretations that the first model pass gets wrong.

### 4. Labeled Datasets that Correct Themselves Automatically

Curtis Northcutt, Cleanlab | 1:06:03 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=IwDGDAHgzAY
Summary: https://mlopstalks.com/talks/labeled-datasets-that-correct-themselves-automatically.md

Why here: Existing labels can be wrong too. Northcutt separates observed labels from likely true labels and uses model disagreement to rank suspicious records. His account explicitly keeps people in the correction process. Add this review queue alongside new-event selection so the update does not merely accumulate errors faster.

### 5. Fixing Your ML Data Blind Spots

Yash Sheth, Galileo | 51:41 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=Z_zt87zhWbE
Summary: https://mlopstalks.com/talks/fixing-your-ml-data-blind-spots.md

Why here: A useful finding should change the next training dataset. Sheth connects missing examples, annotation errors and stale test sets to active-learning refresh workflows. His speech experience also cautions against filling every gap with synthetic data, since a model can learn the generator's artifacts instead of the production variation you meant to add.

### 6. Reliable LLM Products, Fueled by Feedback

Chinar Movsisyan, Feedback Intelligence | 49:17 | MLOps Podcast
Video: https://www.youtube.com/watch?v=1Y_6z6I2g_Y
Summary: https://mlopstalks.com/talks/reliable-llm-products-fueled-by-feedback.md

Why here: User behavior supplies evidence, but not an automatic training label. Movsisyan explains why repeated questions, clicks and abandoned chats need context before teams infer what went wrong. This LLM example qualifies the earlier correction loop: investigate the interaction before turning a user's action into an answer the next model should imitate.

### 7. Continuous Delivery and Automation Pipelines in ML, Part 2

 | 1:07:48 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=bYvntyAXu7g
Summary: https://mlopstalks.com/talks/continuous-delivery-and-automation-pipelines-in-ml-part-2.md

Why here: Fresh evidence does not dictate one update mechanism. This 2020 reading-group discussion considers periodic refitting, updates from existing weights, recent-data weighting and ensembles, alongside leaving a model unchanged. Use those alternatives to choose a response and cadence that fit the observed problem, before the final account shows fast adaptation in practice.

### 8. Real-Time Forecasting Faceoff: Time Series vs. DNNs

Josh Xi, Lyft | 53:42 | MLOps Podcast
Video: https://www.youtube.com/watch?v=zeTnQAKb_2U
Summary: https://mlopstalks.com/talks/real-time-forecasting-faceoff-time-series-vs-dnns.md

Why last: Lyft turns new observations into frequent updates of a small forecasting model. Xi explains online refitting and compares candidate models by walking through historical data as if each prediction were being made then. The concrete finish is a feedback loop with both an update rule and evidence for judging it, rather than retraining simply because new rows arrived.

## Editor's note

Rizzoli returns production examples to versioned datasets, and Munro connects annotation speed to retraining cadence. ZenML records pipeline steps, inputs, outputs and code versions, so a resulting model can be traced to its producing run. That history helps investigate a dataset update while the team defines selection rules, reviews labels and decides whether the new model is useful.

Written by the MLOps Talks editors (the ZenML team), not by any of the speakers.