# Product Engineering for LLMs

Charles Frye, Full Stack Deep Learning & Sahar Mor, Stripe & Sarah Guo, Conviction & Shyamala Prayaga, NVIDIA & Willem Pienaar | LLMs in Production 2023 | 31:46

Source: https://www.youtube.com/watch?v=jbUGkzHtgOk
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/product-engineering-for-llms
Published: 2023-11-07
Tags: evals, multimodal, product-strategy, voice

## TL;DR
- LLM products move beyond demos when their tasks tolerate errors, automate enough of the workflow, and meet a useful quality threshold.
- Evaluation sets let teams measure whether an application performs as expected and improve it from a general-purpose system toward an expert one.
- Putting an LLM synchronously on a user's critical path creates problems with latency, hallucinations, and review fatigue, so products should often make suggestions or run work asynchronously.

## Summary
The panel discusses how teams can turn LLM capabilities into products that people can use and pay for. Sarah Guo points to legal workflows, code generation, and media creation as areas where the models have crossed a quality threshold. Shyamala Prayaga describes how LLMs reduce the manual work needed to build voice assistants from documents and other knowledge sources. The panel returns several times to evaluation, since teams need data and tests that reflect the performance they expect. Latency and error checking create difficult product choices, especially when an answer appears and then disappears after a safety check. The speakers recommend interfaces where AI suggestions can be ignored, as with GitHub Copilot, and workflows where the system can work asynchronously. They also discuss multimodal systems, accessibility, richer user input, video translation, and robotics. Their view is practical: product design must account for model limits, user attention, and the cost of reviewing generated work.

## Key ideas
### LLM products need to leave the dancing-bear phase
[02:22](https://www.youtube.com/watch?v=jbUGkzHtgOk&t=142s)
Charles Frye frames the challenge as moving past the stage where the impressive fact is that a system can do anything at all. He calls this the "dancing bear phase," borrowing Alan Cooper's description of early technology. Earlier waves such as GUIs, browsers, and mobile required product and design judgment alongside engineering. An LLM product therefore has to be technically sound and useful to a person, rather than stopping at an entertaining demonstration.

### Tasks differ in how safely they can use current models
[03:54](https://www.youtube.com/watch?v=jbUGkzHtgOk&t=234s)
Sarah Guo says the current systems work better in some real-world tasks than others. Two factors matter: how much error a task can tolerate and how much of the work can be automated from start to finish. She points to legal work, where users read documents, summarize them, apply a framework, and produce text, as a strong fit for current models. Code is another form of text generation. She also cites media creation, including Runway, Pika, and Synthesia, as an area that has recently crossed a useful quality threshold.

### Lowering the training barrier lets more people try models
[06:27](https://www.youtube.com/watch?v=jbUGkzHtgOk&t=387s)
Shyamala Prayaga describes hosted model platforms as an important change in the development process. Previously, training or fine-tuning required infrastructure, suitable data, and a long, tedious workflow. New platforms let a user choose a base model, upload data, set hyperparameters, and start a training pipeline. She says this lowers the barrier for people to experiment with model variants and contributes to a larger open-source model ecosystem. The panel also connects current progress to larger, well-curated text datasets such as the data used to train GPT-3.

### Voice assistants are moving from fixed intents toward open-ended knowledge
[10:08](https://www.youtube.com/watch?v=jbUGkzHtgOk&t=608s)
Prayaga contrasts older voice assistants with systems built around large language models. Earlier systems relied on speech recognition, intent classification, rules, and manually collected utterances. Small variations such as asking about the weather in Seattle or asking whether an umbrella is needed had to be mapped into the same intent. LLMs can interpret these variations without every utterance being entered by hand. Prayaga says retrieval-augmented generation also makes it easier to turn a large source such as a car owner's manual into a conversational assistant.

### Evaluation is how a general model becomes a dependable application
[14:49](https://www.youtube.com/watch?v=jbUGkzHtgOk&t=889s)
Willem Pienaar says teams that build strong applications usually maintain an evaluation set and a dataset for ongoing monitoring. These checks show whether the system is performing as expected and give the team a basis for iteration. The panel uses this process to explain how an application can move from a general capability toward something closer to an expert system. Evaluation also matters when a product must balance answer quality against latency, safety checks, and the user's experience.

### Synchronous AI on the critical path creates user and review fatigue
[20:47](https://www.youtube.com/watch?v=jbUGkzHtgOk&t=1247s)
The panel identifies latency as especially difficult for open-ended, user-facing systems. A generated answer may need another model call to check whether it is harmful or biased, but showing the answer and then deleting it can feel dystopian. Sarah Guo argues that systems should avoid forcing users to be good at prompt writing and should instead gather missing detail through a workflow. The speakers also warn that generated code can increase the amount of code humans must review. An assistant that creates ten times as much code can create ten times as much review work.

### Suggestions and asynchronous work give users room to reject the model
[21:07](https://www.youtube.com/watch?v=jbUGkzHtgOk&t=1267s)
Willem Pienaar says LLMs are most annoying when they are placed synchronously in front of a familiar task. GitHub Copilot is a better pattern because its suggestions appear beside the user's work and can be dismissed quickly. Sarah Guo extends this idea to engineering agents that can run a job, test code, gather information, and return later with a result or a question. Shyamala Prayaga applies the same product principle to voice and multimodal assistants through suitable audio cues, visual cues, and notifications.

### Multimodal products should reduce work and improve access
[25:13](https://www.youtube.com/watch?v=jbUGkzHtgOk&t=1513s)
Prayaga describes NVIDIA's NeMo Vision Assistant as a system that accepts images and text. It can describe an image or write a story based on its subject or background, while also working as a text-to-text model. She focuses on accessibility: a multimodal model could generate a detailed description for someone using a screen reader instead of relying on a short manually written alt tag. The panel also discusses video translation, generated video, editing workflows, and robotics as areas where combining modalities could make difficult tasks easier.

## Notable quotes
- Charles Frye: "The exciting thing is not how good the product is, but the fact that it can do anything at all." (02:22)
- Sarah Guo: "The pattern I think is kind of interesting is companies figuring out what is the workflow to be predictive." (18:26)
- Willem Pienaar: "Where do LLMs really annoy people and suck? It's to put them on the critical path and you put them synchronously in front of the user on a job that they're familiar doing." (20:47)
- Sarah Guo: "If you 10x the amount of code to review, you 10x the amount of code review to do." (22:03)
- Shyamala Prayaga: "I think about it more on the utility standpoint, like how it is helping the users." (26:32)

## Tools & references mentioned
- Charles Frye
- Alan Cooper
- GPT-3
- Runway
- Pika
- Synthesia
- GitHub Copilot
- ChatGPT
- Sydney
- Stripe Docs
- Seek AI
- Alex Graveley
- Inflection
- Character.AI
- NVIDIA NeMo Vision Assistant
- NVIDIA AI Playground
- GPT-4V
- Google's Search Generative Experience
- ElevenLabs
- Suno
- Photoshop
- Full Stack Deep Learning

## Who should watch
- You are building an LLM feature and need product patterns that account for latency, hallucinations, evaluation, and user review work.
- Your team is turning voice, document, or multimodal capabilities into an assistant and needs examples of where current systems reduce manual effort.
- You are deciding whether an AI system should interrupt a user's workflow, make suggestions, or run work in the background.

## Related talks

- [Evaluating LLM-based Applications](https://mlopstalks.com/talks/evaluating-llm-based-applications) (Josh Tobin, Gantry, 49:50)
- [From Building Self-driving Cars to Building LLM Applications](https://mlopstalks.com/talks/from-building-self-driving-cars-to-building-llm-applications) (Effy Zhang, Baserun, 10:45)
- [Building Products](https://mlopstalks.com/talks/building-products) (Sam Charrington, TWIML AI Podcast & George Mathew, Insight Partners & Asmitha Rathis, PromptOps & Natalia Burina, Meta & Sahar Mor, Stripe, 45:18)
- [Evaluating Quality and Improving LLM Products at Scale](https://mlopstalks.com/talks/evaluating-quality-and-improving-llm-products-at-scale) (Austin Bell, Slack, 15:11)
- [How Product Metrics Become LLM Evaluations](https://mlopstalks.com/talks/how-product-metrics-become-llm-evaluations) (Raza Habib, Humanloop, 53:07)