LLM products move beyond demos when their tasks tolerate errors, automate enough of the workflow, and meet a useful quality threshold.
2
Evaluation sets let teams measure whether an application performs as expected and improve it from a general-purpose system toward an expert one.
3
Putting an LLM synchronously on a user's critical path creates problems with latency, hallucinations, and review fatigue, so products should often make suggestions or run work asynchronously.
Summary
The panel discusses how teams can turn LLM capabilities into products that people can use and pay for. Sarah Guo points to legal workflows, code generation, and media creation as areas where the models have crossed a quality threshold. Shyamala Prayaga describes how LLMs reduce the manual work needed to build voice assistants from documents and other knowledge sources. The panel returns several times to evaluation, since teams need data and tests that reflect the performance they expect. Latency and error checking create difficult product choices, especially when an answer appears and then disappears after a safety check. The speakers recommend interfaces where AI suggestions can be ignored, as with GitHub Copilot, and workflows where the system can work asynchronously. They also discuss multimodal systems, accessibility, richer user input, video translation, and robotics. Their view is practical: product design must account for model limits, user attention, and the cost of reviewing generated work.
Charles Frye frames the challenge as moving past the stage where the impressive fact is that a system can do anything at all. He calls this the "dancing bear phase," borrowing Alan Cooper's description of early technology. Earlier waves such as GUIs, browsers, and mobile required product and design judgment alongside engineering. An LLM product therefore has to be technically sound and useful to a person, rather than stopping at an entertaining demonstration.
Tasks differ in how safely they can use current models
Sarah Guo says the current systems work better in some real-world tasks than others. Two factors matter: how much error a task can tolerate and how much of the work can be automated from start to finish. She points to legal work, where users read documents, summarize them, apply a framework, and produce text, as a strong fit for current models. Code is another form of text generation. She also cites media creation, including Runway, Pika, and Synthesia, as an area that has recently crossed a useful quality threshold.
Lowering the training barrier lets more people try models
Shyamala Prayaga describes hosted model platforms as an important change in the development process. Previously, training or fine-tuning required infrastructure, suitable data, and a long, tedious workflow. New platforms let a user choose a base model, upload data, set hyperparameters, and start a training pipeline. She says this lowers the barrier for people to experiment with model variants and contributes to a larger open-source model ecosystem. The panel also connects current progress to larger, well-curated text datasets such as the data used to train GPT-3.
Voice assistants are moving from fixed intents toward open-ended knowledge
Prayaga contrasts older voice assistants with systems built around large language models. Earlier systems relied on speech recognition, intent classification, rules, and manually collected utterances. Small variations such as asking about the weather in Seattle or asking whether an umbrella is needed had to be mapped into the same intent. LLMs can interpret these variations without every utterance being entered by hand. Prayaga says retrieval-augmented generation also makes it easier to turn a large source such as a car owner's manual into a conversational assistant.
Evaluation is how a general model becomes a dependable application
Willem Pienaar says teams that build strong applications usually maintain an evaluation set and a dataset for ongoing monitoring. These checks show whether the system is performing as expected and give the team a basis for iteration. The panel uses this process to explain how an application can move from a general capability toward something closer to an expert system. Evaluation also matters when a product must balance answer quality against latency, safety checks, and the user's experience.
Synchronous AI on the critical path creates user and review fatigue
The panel identifies latency as especially difficult for open-ended, user-facing systems. A generated answer may need another model call to check whether it is harmful or biased, but showing the answer and then deleting it can feel dystopian. Sarah Guo argues that systems should avoid forcing users to be good at prompt writing and should instead gather missing detail through a workflow. The speakers also warn that generated code can increase the amount of code humans must review. An assistant that creates ten times as much code can create ten times as much review work.
Suggestions and asynchronous work give users room to reject the model
Willem Pienaar says LLMs are most annoying when they are placed synchronously in front of a familiar task. GitHub Copilot is a better pattern because its suggestions appear beside the user's work and can be dismissed quickly. Sarah Guo extends this idea to engineering agents that can run a job, test code, gather information, and return later with a result or a question. Shyamala Prayaga applies the same product principle to voice and multimodal assistants through suitable audio cues, visual cues, and notifications.
Multimodal products should reduce work and improve access
Prayaga describes NVIDIA's NeMo Vision Assistant as a system that accepts images and text. It can describe an image or write a story based on its subject or background, while also working as a text-to-text model. She focuses on accessibility: a multimodal model could generate a detailed description for someone using a screen reader instead of relying on a short manually written alt tag. The panel also discusses video translation, generated video, editing workflows, and robotics as areas where combining modalities could make difficult tasks easier.
"Where do LLMs really annoy people and suck? It's to put them on the critical path and you put them synchronously in front of the user on a job that they're familiar doing."Willem Pienaar20:47
Who should watch
You are building an LLM feature and need product patterns that account for latency, hallucinations, evaluation, and user review work.
Your team is turning voice, document, or multimodal capabilities into an assistant and needs examples of where current systems reduce manual effort.
You are deciding whether an AI system should interrupt a user's workflow, make suggestions, or run work in the background.
