# Shipping LLMs: Buckle Up & Enjoy the Ride

Rex Harris | AI in Production 2024 | 12:51

Source: https://www.youtube.com/watch?v=zI7JL69Vrhw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/shipping-llms-buckle-up-enjoy-the-ride
Published: 2024-04-24
Tags: evals, latency, product-strategy, prompt-engineering

## TL;DR
- LLMs should solve customer problems that already exist instead of being added because they are the latest technology.
- Shipping LLM features gives product teams faster ways to design, test, learn from users, and iterate.
- Production LLM systems bring new risks, including prompt injections, hallucinations, inference costs, privacy, latency, and evaluation.

## Summary
Rex Harris frames LLM work as a progression from prompt experiments to proof-of-concept applications and then production systems. His advice starts with finding a real customer problem before choosing the technology. He gives knowledge distillation, content creation, conversational interfaces, and AI assistants as examples of areas where LLMs can fit. Harris also describes the practical rewards of building with LLMs. Product managers can work more closely on conversational UX, test ideas quickly, inspect customer logs, and learn from both positive and painful user interactions. The risks are substantial. Teams must handle prompt injections, hallucinations, inference costs, privacy, latency, and evaluation. Harris argues that models are getting better, faster, and cheaper, while the excitement around the technology can help teams build momentum. His closing advice is to learn through the challenges and enjoy the work, because that enthusiasm can show in the product.

## Key ideas
### LLM work moves from street driving to the racetrack
[02:01](https://www.youtube.com/watch?v=zI7JL69Vrhw&t=121s)
Harris describes three stages of LLM proficiency. Prompt engineering is like driving on city streets while getting your bearings. A proof of concept is like moving onto the highway, where there is more to manage and more risk. Production is the racetrack, where teams need to tune many parts of the system before launch. The production stage carries more risk, but it also offers more reward. This framing makes clear that a successful prompt experiment is only an early step toward a customer-facing system.

### A real customer problem should come before the model
[03:07](https://www.youtube.com/watch?v=zI7JL69Vrhw&t=187s)
Harris warns against having an LLM solution looking for a problem. He calls this the pattern of powerful technology being applied without a clear use case. He points to knowledge distillation and summarization, content creation, and AI assistants as areas where LLMs can help. His example from sleep technology used an AI sleep coach for a smart mattress. He also mentions Slack's ability to distill information from threads and channels, and Gamma, which he used to create the presentation.

### LLMs give product teams more control over conversational design
[05:07](https://www.youtube.com/watch?v=zI7JL69Vrhw&t=307s)
Harris says product managers can become much closer to the user experience when the product is conversational. They can design how the system responds, test ideas rapidly, and make changes for internal or external users. He points to Gradio while discussing the tooling that makes this work easier. The feedback loop resembles ordinary product development, but the speed of testing and iteration can be much faster.

### Non-deterministic outputs make building frustrating and rewarding
[06:09](https://www.youtube.com/watch?v=zI7JL69Vrhw&t=369s)
LLM outputs do not always behave the same way, so teams have to work through trial and error. Harris compares the experience to a variable reward: an output can be frustrating when it misses the mark, then exciting when it produces something unexpectedly good. He presents this unpredictability as a condition builders need to accept rather than a reason to avoid the technology.

### Conversation logs expose what users need
[07:07](https://www.youtube.com/watch?v=zI7JL69Vrhw&t=427s)
Conversational products can reveal more about customer needs because users explain what they want in their interactions. Harris says teams can inspect logs to learn about requests for a specific feature as well as needs across the product. These logs also include moments when users are happy and moments when they struggle. He mentions Context AI as a way to analyse sentiment and insights from LLM logs.

### Production LLMs add operational problems
[08:25](https://www.youtube.com/watch?v=zI7JL69Vrhw&t=505s)
Harris lists prompt injections, hallucinations, inference costs, privacy, latency, and evaluations as challenges that teams must handle when shipping LLMs. He does not present the technology as simple to deploy. His advice is to start with awareness that these issues will create real work, then plan for them as part of building the product.

### Faster, cheaper models and team excitement can help teams move
[08:47](https://www.youtube.com/watch?v=zI7JL69Vrhw&t=527s)
Harris identifies two sources of momentum. Models are continuing to improve while becoming faster and cheaper, and open source work is making the model ecosystem more active. He also says teams can use the excitement around generative AI to create momentum inside a company. That enthusiasm only makes sense when the product has a sound reason for using LLMs.

### Enjoyment is part of the product-building process
[09:49](https://www.youtube.com/watch?v=zI7JL69Vrhw&t=589s)
Harris closes with three pieces of advice: find the reason for using LLMs, accept that learning will bring challenges, and enjoy the work. In the follow-up discussion, he explains that hype usually has a negative connotation, but building a useful LLM feature can give a team genuine excitement. When the reason is right, that excitement can create momentum and show up in the product.

## Notable quotes
- "Find your why, don't just apply LLMs because it's the latest technology." (04:25)
- "You can really get intimate with the customer needs." (07:07)
- "There are these kind of LLM wrenches looking for nuts." (03:07)
- "Models continue to get better faster and cheaper." (08:47)
- "When you are having fun and creating something you love, it really shows in the product." (10:10)

## Tools & references mentioned
- Precision Nutrition
- Slack
- Gamma
- Figma
- Gradio
- Ricky Bobby
- Context AI
- Sora
- Mistral
- Tom Ford
- Vin Diesel

## Who should watch
- You are deciding whether an LLM belongs in an existing product and need a way to separate customer value from technology enthusiasm.
- You are a product manager or designer working on a conversational feature and want to use faster testing and user feedback loops.
- Your team is moving from an LLM prototype toward production and needs a reminder about prompt injections, hallucinations, cost, privacy, latency, and evaluation.

## Related talks

- [All the Hard Stuff with LLMs in Product Development](https://mlopstalks.com/talks/all-the-hard-stuff-with-llms-in-product-development) (Phillip Carter, Honeycomb, 1:01:04)
- [Product Engineering for LLMs](https://mlopstalks.com/talks/product-engineering-for-llms) (Charles Frye, Full Stack Deep Learning & Sahar Mor, Stripe & Sarah Guo, Conviction & Shyamala Prayaga, NVIDIA & Willem Pienaar, 31:46)
- [Product Strategy for LLM Features When LLMs Aren't Your Product](https://mlopstalks.com/talks/product-strategy-for-llm-features-when-llms-arent-your-product) (Harini Kannan, 22:59)
- [Using LLMs to Punch Above Your Weight!](https://mlopstalks.com/talks/using-llms-to-punch-above-your-weight) (Cameron Feenstra, Anzen, 35:49)
- [Evaluating Quality and Improving LLM Products at Scale](https://mlopstalks.com/talks/evaluating-quality-and-improving-llm-products-at-scale) (Austin Bell, Slack, 15:11)