Production errors keep arriving, but the next model trains on the same old examples. Closing that gap takes more than scheduling another training run. Someone must decide which cases deserve review, what the labels mean, and whether the proposed update improves the task. Start with a versioned dataset that receives production examples, then design the selection and annotation process together. Autonomous driving makes the value of rare events concrete; noisy-label analysis prevents corrections from becoming another source of error. Speech-model refreshes connect those findings to new training data. An LLM product example adds caution about interpreting user behavior. Finish with alternatives to automatic retraining and Lyft's fast forecasting updates, where fresh observations change model weights and historical simulation checks the result.
Why first: A production example needs a destination before it can improve a model. Rizzoli describes returning low-confidence or pre-labeled items to a versioned training library for human review. His 2021 computer-vision account replaces an informal folder handoff with a deliberate choice of what enters each later dataset.
Why here: Choosing the next examples and choosing the annotation process are one design problem. Munro combines uncertainty with diversity, then accounts for annotator expertise and disagreement. This prevents the returning-data loop from selecting only familiar uncertain cases while missing confident mistakes or unreliable labels.
Why here: A few unusual seconds can matter more than an uneventful drive. Curtis describes selecting short events, checking coverage and using people to correct automated annotations. The autonomous-vehicle examples make the selection policy tangible: spend review effort on missing conditions and interpretations that the first model pass gets wrong.
Why here: Existing labels can be wrong too. Northcutt separates observed labels from likely true labels and uses model disagreement to rank suspicious records. His account explicitly keeps people in the correction process. Add this review queue alongside new-event selection so the update does not merely accumulate errors faster.
Why here: A useful finding should change the next training dataset. Sheth connects missing examples, annotation errors and stale test sets to active-learning refresh workflows. His speech experience also cautions against filling every gap with synthetic data, since a model can learn the generator's artifacts instead of the production variation you meant to add.
Why here: User behavior supplies evidence, but not an automatic training label. Movsisyan explains why repeated questions, clicks and abandoned chats need context before teams infer what went wrong. This LLM example qualifies the earlier correction loop: investigate the interaction before turning a user's action into an answer the next model should imitate.
Why here: Fresh evidence does not dictate one update mechanism. This 2020 reading-group discussion considers periodic refitting, updates from existing weights, recent-data weighting and ensembles, alongside leaving a model unchanged. Use those alternatives to choose a response and cadence that fit the observed problem, before the final account shows fast adaptation in practice.
Why last: Lyft turns new observations into frequent updates of a small forecasting model. Xi explains online refitting and compares candidate models by walking through historical data as if each prediction were being made then. The concrete finish is a feedback loop with both an update rule and evidence for judging it, rather than retraining simply because new rows arrived.