# From MVP to Production

Eric Peter, Databricks & Donné Stevenson & Phillip Carter, Honeycomb & Andrew Hoh, Last Mile AI | AI in Production 2024 | 32:53

Source: https://www.youtube.com/watch?v=Smq3Q9r-9cg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/from-mvp-to-production
Published: 2024-04-09
Tags: data-quality, evals, observability

## TL;DR
- Production changes what users do with an AI product, so teams need feedback loops that capture inputs, outputs, user judgments, and domain context.
- Evaluation combines human annotation, heuristic checks, and model-based assessment, with cheaper encoder models often replacing costly LLM judges for classification tasks.
- Keeping generative AI systems current is a version-control problem across data, retrieval infrastructure, processing pipelines, models, and evaluation sets.

## Summary
The panel focuses on the work that begins after an AI demo appears to function. Phillip Carter says production users quickly expose behavior that internal teams did not predict, especially when a product gives users a blank canvas and accepts broad inputs. Donné Stevenson describes staged releases that move from informed testers toward general users. Eric Peter argues that teams need tooling to capture complete interactions, collect thumbs-up or thumbs-down feedback with reasons, and measure performance through several metrics. Andrew Hoh separates evaluation into human annotation, heuristic checks, and LLM-based judging. He says encoder models can make classification-style evaluation much cheaper. The panel also discusses domain-specific definitions of correctness, such as preserving brand names in translation or interpreting industry acronyms. The final section treats freshness as a version-control problem. Data sources, retrieval pipelines, models, and evaluation sets all need versioning so teams can understand performance changes and refresh systems safely.

## Key ideas
### Production users reveal behavior that internal testing cannot predict
[04:19](https://www.youtube.com/watch?v=Smq3Q9r-9cg&t=259s)
Phillip Carter says teams are often arrogant if they think they can predict what users will try once a system reaches production. Users approach an open input field through their own mental models, rather than following the gestures expected by the product team. Generative systems make this harder because inputs can be practically unbounded and outputs can be non-deterministic. Teams therefore need to study what users actually submit, what the system returns, and what users think about those results. Carter says shipping faster only helps if the team also learns from what it ships. Without that loop, the product can remain software that does not do what users need.

### Staged releases give teams a safer evaluation period
[07:34](https://www.youtube.com/watch?v=Smq3Q9r-9cg&t=454s)
Donné Stevenson says her team found that systems which looked good during testing behaved differently with real users. The people who define a problem are not always the people who use the product, and general users may treat an AI system as a black box. Her proposed response is to release in phases. A team can begin with an intermediate group that understands the intended behavior while still representing real use. That group can test the product more deeply and provide feedback before hundreds of general users arrive. The gradual process helps teams learn how the system behaves without releasing a product whose later behavior they do not understand.

### Feedback tooling must connect user judgments to the full interaction
[10:03](https://www.youtube.com/watch?v=Smq3Q9r-9cg&t=603s)
Eric Peter says teams often begin with weak feedback methods, such as asking people to play with a system or record comments in a spreadsheet. Statements such as 'it does not work' do not tell developers which question failed or why. He recommends capturing a full log for every interaction, then giving users a fast way to send a thumbs-up or thumbs-down with a reason. Users may also need to edit the answer and inspect what information was retrieved. Qualitative review is only part of the process. Teams also need metrics, including information-retrieval measures and model-based judgments, so they can combine direct inspection with repeatable signals.

### Evaluation has different costs and different kinds of evidence
[13:16](https://www.youtube.com/watch?v=Smq3Q9r-9cg&t=796s)
Andrew Hoh divides evaluation into human annotation, heuristic checks, and LLM-based evaluation. Human annotation can provide strong evidence, but specialist tasks may require an expert who can judge whether a summary correctly condenses a large source. LLM evaluation costs money because one model assesses another model's output. Hoh says his team has made the most progress with encoder-based models because many evaluator tasks are classification problems, and a smaller classifier can cost much less. No single metric covers every application. Teams may need assertions about output format, checks for faithfulness to retrieved data, and relevance measures, combined according to the needs of a particular industry.

### Correctness depends on domain context
[16:26](https://www.youtube.com/watch?v=Smq3Q9r-9cg&t=986s)
The panel gives several examples of why generic evaluation fails. In a sales system, 'NASA' may mean North America and South America rather than the space agency, so the evaluator needs the company's terminology and acronyms. A translation bot may translate links or brand names that should remain unchanged. Eric Peter says a coding assistant's answer can appear helpful almost every time when GPT-4 is asked to judge it with a vague prompt. The team had to define helpfulness with specific guidelines and examples. Donné Stevenson adds that SQL systems need enough context about a user's data, while also recognizing when missing implicit knowledge means the model should ask for clarification.

### Freshness requires version control across the whole system
[25:10](https://www.youtube.com/watch?v=Smq3Q9r-9cg&t=1510s)
Andrew Hoh describes stale knowledge and changing retrieval data as a large version-control problem. A system can change behavior when its information is updated underneath it, but it can also become stale if the team does not know when to refresh and re-evaluate it. The panel argues for versioning data sources, processing pipelines, retrieval infrastructure, model versions, and evaluation sets. Eric Peter says teams do not usually retrain a language model every night. They instead need reliable data pipelines and a vector database that stays in sync with source systems. The panel connects this work to familiar machine-learning practice: curate ground-truth evaluation sets, define metrics, and track regressions.

## Notable quotes
- Phillip Carter: "You're going to find that, at least in my experience, when you present sort of not even just a blank canvas, but just like a way for someone to input what they actually want to input, you find that they're going to approach your product just differently." (05:12)
- Andrew Hoh: "The first one is you have human annotation, human loop style, right, which is like audit logs or manual experimentation where you're able to manually annotate whether the results are right or wrong." (13:34)
- Andrew Hoh: "It's a massive version control problem." (25:50)
- Eric Peter: "We have to go curate ground truth evaluation set, we're going to decide what metrics we're going to hill climb on." (27:22)

## Tools & references mentioned
- Weights & Biases
- ThursdAI
- Databricks
- scikit-learn
- GPT-4
- Honeycomb
- Last Mile AI
- OpenAI
- MLOps Community
- DataDog
- MLflow
- Spark

## Who should watch
- You are moving an LLM, retrieval system, or AI assistant from an internal demo toward real users and need a practical feedback loop.
- Your team has user feedback but cannot connect it to the exact inputs, outputs, retrieved context, or model version that produced an outcome.
- You need to design evaluations for a domain where correctness depends on specialist terminology, formatting rules, or industry-specific knowledge.

## Editor's note

Andrew Hoh describes stale knowledge and changing retrieval data as a version-control problem that spans data sources, retrieval infrastructure, models, and evaluation sets. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace a result to the data and code that produced it. That gives teams a record for comparing changes and refreshing systems safely.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Extending AI: From Industry to Innovation](https://mlopstalks.com/talks/extending-ai-from-industry-to-innovation) (Sophia Rowland & David Weik, SAS, 1:01:37)
- [GenAI in Production - Challenges and Trends](https://mlopstalks.com/talks/genai-in-production-challenges-and-trends) (Verena Weber, Verena Weber, 48:43)
- [Holistic Evaluation of Generative AI Systems](https://mlopstalks.com/talks/holistic-evaluation-of-generative-ai-systems) (Jineet Doshi, Intuit, 56:04)
- [Everything Hard About Building AI Agents Today](https://mlopstalks.com/talks/everything-hard-about-building-ai-agents-today) (Shreya Shankar & Willem Pienaar, Cleric, 47:03)
- [Productionizing AI: How to Think From the End](https://mlopstalks.com/talks/productionizing-ai-how-to-think-from-the-end) (Annie Condon, 11:11)