# From Building Self-driving Cars to Building LLM Applications Effy Zhang, Baserun | LLMs in Production 2023 | 10:45 Source: https://www.youtube.com/watch?v=uXA7cfSWFng Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/from-building-self-driving-cars-to-building-llm-applications Published: 2023-11-02 Tags: evals, human-in-the-loop, monitoring, testing ## TL;DR - LLM applications need deeper evaluation than a pass-or-fail unit test because their outputs and workflows are more complex. - Changing a model can require evaluating the whole application again, while users may also find unexpected ways to misuse it. - Teams should combine staged testing, user-focused metrics, feedback loops, and collaboration between engineers and operational or product roles. ## Summary Effy Zhang applies lessons from Cruise's self-driving car systems to LLM applications. She explains that both systems produce complex outputs that need more analysis than a simple pass-or-fail test. Model changes can affect an application end to end, and users may expose unexpected behaviors through jailbreaks or unfamiliar interactions. Zhang recommends testing at three levels: individual model calls, separate workflows, and the complete user journey. Teams should connect these tests to a top-level product metric that reflects user experience, then break that metric into measures for each testing stage. She also argues for a closer feedback loop between production data, monitoring, analysis, feature priorities, testing, and iteration. Product managers and other operational roles can review interactions, annotate results, test prompts, and take part in experiments. The talk is practical and honest about the limits of optimizing only cost and latency. ## Key ideas ### LLM outputs need analysis beyond pass-or-fail tests [00:28](https://www.youtube.com/watch?v=uXA7cfSWFng&t=28s) Zhang compares LLM applications with the visualization and analysis tools Cruise built for self-driving cars. A vehicle system combines data from many sensors, so its behavior cannot be described by a simple pass-or-fail result. LLM applications also produce outputs that need deeper analysis. Their surrounding tooling becomes more complex as teams process more data and evaluate responses in context. ### Changing the model can require end-to-end reevaluation [01:14](https://www.youtube.com/watch?v=uXA7cfSWFng&t=74s) At Cruise, Zhang worked on a transition from one vehicle design to another, which required research and substantial system changes. She compares this with moving from GPT to another model such as Llama Index. A model change can affect the whole application, so teams may need to reevaluate performance from the model call through the complete user experience rather than checking only the new model in isolation. ### Users expose failure modes that developers did not plan for [01:55](https://www.youtube.com/watch?v=uXA7cfSWFng&t=115s) Cruise had to account for people trying to trick its cars into stopping, such as placing an object on top of the vehicle. Zhang connects this with LLM users who may jailbreak an application or try to make it reveal information that should remain private. The product therefore needs monitoring for misuse and unexpected interactions as features are shipped. ### New environments reveal gaps in the system's assumptions [02:40](https://www.youtube.com/watch?v=uXA7cfSWFng&t=160s) When Cruise expanded into Austin, its cars encountered police officers riding horses, something the team had not seen in San Francisco. The unfamiliar object exposed a gap in the car's ability to identify its surroundings. Zhang says LLM teams face a similar problem because they cannot fully predict how end users will interact with an application. Collecting feedback after release becomes part of improving the product. ### User experience includes more than latency and cost [03:24](https://www.youtube.com/watch?v=uXA7cfSWFng&t=204s) Zhang uses Clay as an example of an application that makes waiting feel acceptable through interface design and animation. She says developers often focus on latency and cost, while the delivered feature is the complete user experience. Product design can help users work with slow model responses, so teams should evaluate the feature as a whole rather than treating model speed as the only measure. ### LLM development needs a continuous feedback and iteration cycle [04:30](https://www.youtube.com/watch?v=uXA7cfSWFng&t=270s) Zhang describes a Cruise loop in which data from vehicles is monitored, analyzed, and turned into feature requirements before returning to testing and iteration. She applies the same pattern to LLM applications. Production interactions should feed monitoring and analysis, which should influence priorities, testing, and later releases. The development process continues after the initial feature launch. ### Testing should cover calls, workflows, and the complete user journey [06:26](https://www.youtube.com/watch?v=uXA7cfSWFng&t=386s) Zhang proposes three testing phases for increasingly complex LLM systems. Unit tests check whether an individual model call behaves as expected. Subsystem tests evaluate a distinct workflow before it is combined with the rest of the application. End-to-end tests measure the complete user journey, including chained model calls, third-party APIs, and custom functions. ### Top-level product metrics should connect the testing stages [07:50](https://www.youtube.com/watch?v=uXA7cfSWFng&t=470s) Cruise used take-over rate as a measure related to comfort, safety, and autonomous driving. Zhang recommends that LLM teams choose a top-level metric tied to the customer experience, such as whether a user completes a task or gives a positive interaction rating. That metric can then be broken into submetrics for each testing phase, giving product owners and developers a shared view of application performance. ### Operational and product roles should participate in evaluation [09:20](https://www.youtube.com/watch?v=uXA7cfSWFng&t=560s) Zhang says product managers and other operational roles increasingly monitor daily interactions, summarize what users do, and bring findings back into development. They may also write prompts and run tests. Teams should let these non-technical contributors annotate monitoring and test results, influence prompts, run A/B tests, and help tune models. Their involvement can make the feedback loop more useful. ## Notable quotes - "A lot of times large language model output need more deep analysis and the tooling around it is getting a lot more complex." (00:54) - "When you switch the model you need to re-evaluate your whole app performance end to end." (01:46) - "End of day we are trying to build a feature for the users." (04:11) - "One way to think about testing you can structure test as three phase." (06:47) - "Think about how do you bridge the gap to make that data pipeline is more streamlined." (09:40) ## Tools & references mentioned - Baserun - Cruise - Clay - GPT - Llama Index ## Who should watch - You are building an LLM feature with chained model calls, APIs, or custom functions and need a testing structure. - Your team tracks latency and cost but has not connected those measures to whether users complete their tasks. - Product managers or operations staff review user interactions and could take part in prompt testing and evaluation. ## Related talks - [Evaluating LLM-based Applications](https://mlopstalks.com/talks/evaluating-llm-based-applications) (Josh Tobin, Gantry, 49:50) - [Systematically Test and Evaluate Your LLMs Apps](https://mlopstalks.com/talks/systematically-test-and-evaluate-your-llms-apps) (Gideon Mendels, Comet, 1:01:43) - [Evaluation](https://mlopstalks.com/talks/evaluation-panel-1-large-language-models-in-production-conference-part-2) (Abi Aryan, Independent Consultant & Amrutha Gujjar, Structured & Josh Tobin, Gantry & Sohini Roy, NVIDIA, 38:19) - [Fireside Chat - The Future of LLMs](https://mlopstalks.com/talks/fireside-chat-the-future-of-llms) (David Hershey, Unusual Ventures & Daniel Jeffries, AI Infrastructure Alliance, 36:07) - [Evaluating Quality and Improving LLM Products at Scale](https://mlopstalks.com/talks/evaluating-quality-and-improving-llm-products-at-scale) (Austin Bell, Slack, 15:11)