# Lessons Learned Productionising LLMs for Stripe Support Sophie Daly, Stripe | LLMs in Production 2023 | 11:33 Source: https://www.youtube.com/watch?v=uW4MUatKINs Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/lessons-learned-productionising-llms-for-stripe-support Published: 2023-07-28 Tags: data-quality, evals, fine-tuning, monitoring ## TL;DR - LLMs can produce plausible but incorrect answers, so Stripe broke support answering into classification, topic identification, response generation, and tone adjustment. - Offline evaluations did not predict agent adoption, which made online feedback, proxy metrics, and early user experience work necessary. - Better labeled data improved performance more than newer GPT engines, so Stripe moved toward classification, weak supervision, and a continuously maintained support dataset. ## Summary Sophie Daly describes Stripe's first LLM application for support agents. The system did not answer customers directly. It prompted agents with responses that were meant to be accurate, concise, and consistent with Stripe's tone. Stripe found that an out-of-the-box GPT model often gave plausible but wrong answers, so the team split the task into several fine-tuned stages. These stages filtered questions, identified topics, generated answers from relevant context, and adjusted the tone. Offline testing looked strong, but agents used the prompts less often than expected. Stripe lacked direct online accuracy labels, so the team built a match-rate heuristic by comparing generated responses with the responses agents actually sent. Daly argues that production LLM work needs monitoring and online feedback from the start. She also says data quality mattered more than moving to newer models. For the next iteration, Stripe replaced generative output with simpler classification methods and planned to use weak supervision and subject matter experts to keep the training data current. ## Key ideas ### Out-of-the-box GPT answers can sound right while being wrong [02:13](https://www.youtube.com/watch?v=uW4MUatKINs&t=133s) Stripe tested DaVinci GPT with a question such as, "How can I pause payouts?" The answer sounded plausible, but it was incorrect. Daly says this happened across much of Stripe's support space because the model's pre-training could be outdated, incomplete, or mixed with instructions from another payments company. Prompt engineering could improve individual answers, but the support domain was too broad and complex for prompts alone to solve it. ### Breaking the task into smaller models gave Stripe more control [02:57](https://www.youtube.com/watch?v=uW4MUatKINs&t=177s) Stripe split support answering into sequential steps. The system first decided whether an input was a valid support question, then identified its topic, generated an answer using topic-relevant context, and adjusted the result to Stripe's friendly but succinct tone. The first classification steps filtered questions before they reached the response model. Daly says fine-tuning removed hallucinations in their case and required about 500 labels per class, which let the team rely on expert agent annotations for its pilot. ### Offline quality did not predict whether agents would use the prompts [04:36](https://www.youtube.com/watch?v=uW4MUatKINs&t=276s) The team used labeled datasets to evaluate classification models and asked expert agents to review generated responses. It also worked with agents on user testing and training-data collection. These offline results looked strong before launch. After shipping, however, agents used the generated prompts far less than expected. They were accustomed to their existing way of solving cases, and the team had little visibility into whether online answers were accurate. ### A proxy match rate provided a useful online signal [05:55](https://www.youtube.com/watch?v=uW4MUatKINs&t=355s) Because labeling online cases at the required scale was not feasible, Stripe compared generated responses with the responses agents actually sent to users. Daly describes this match rate as a crude lower-bound measure of expected accuracy. It did not provide a complete evaluation, but it gave the team directional information about model behavior in production instead of leaving them unable to tell whether offline and online performance differed. ### Agent behavior made user experience work part of the ML problem [06:35](https://www.youtube.com/watch?v=uW4MUatKINs&t=395s) Positive user testing and good online model trends did not translate into efficiency gains because agents often ignored the prompts. Daly says the team needed a much larger user experience effort to increase adoption. Her practical lesson is to ask early whether human behavior can affect the business problem and to involve the UX team before the system is ready to ship. ### Every stage should reach shadow mode before the full system ships [07:17](https://www.youtube.com/watch?v=uW4MUatKINs&t=437s) Daly recommends shipping each stage of a multi-step framework in shadow mode as soon as it is ready. This lets the team debug components while development continues, check that each stage works, and validate expectations one step at a time. Waiting for one large end-to-end launch made it harder to find where the system or adoption process was failing. ### Monitoring belongs in the definition of a shipped model [07:36](https://www.youtube.com/watch?v=uW4MUatKINs&t=456s) Stripe initially treated monitoring as something a small, resource-constrained team might add later. The production experience changed that view. Daly says a model is not shipped unless it has full monitoring in a dashboard, because online feedback shows whether the system is solving the intended problem. Proxy metrics can be imperfect, but they are still better than having no production signal. ### Data work mattered more than adopting a newer model [08:14](https://www.youtube.com/watch?v=uW4MUatKINs&t=494s) Daly says writing the LLM framework took days or weeks, while improving the training dataset took much longer. Changes to label quality produced larger performance gains than switching to more advanced GPT engines. The observed errors were usually tied to specific Stripe support issues rather than broad language understanding failures. Adding better or more representative samples usually addressed the gap. ### Stripe moved toward simpler models and living support data [09:09](https://www.youtube.com/watch?v=uW4MUatKINs&t=549s) Collecting labels for generative fine-tuning added substantial complexity, so Stripe's second iteration replaced the generative component with simpler classification. The team planned to use weak supervision through Snorkel and embedding classification to label data at scale without explicit labels for every example. Subject matter experts would maintain the dataset as Stripe's products and support space changed, keeping labels current. ## Notable quotes - "LLMs are not oracles." (02:13) - "Directional feedback using heuristics is a million times better than being completely in the dark." (06:56) - "A model is not shipped unless it has full monitoring in a dashboard." (07:55) - "Writing the code for this LLM framework took a matter of days or weeks, whereas iterating on the data set to train these models took months." (08:33) - "A good data strategy will outweigh any fancy LLM architecture, especially for solving business problems that require deep domain expertise at scale." (10:34) ## Tools & references mentioned - Stripe - GPT - DaVinci GPT - Snorkel ## Who should watch - You are building an LLM feature for an internal operations team and need to measure whether people actually use it after launch. - Your offline evaluation looks strong, but you lack reliable production labels or visibility into model behavior. - You are deciding whether to invest in a newer model or improve domain data, labeling, and subject matter expert review. ## Related talks - [MLOps at Stripe](https://mlopstalks.com/talks/mlops-at-stripe) (Emmanuel Ameisen, Stripe, 44:36) - [Want High Performing LLMs? Hint: It Is All About Your Data](https://mlopstalks.com/talks/want-high-performing-llms-hint-it-is-all-about-your-data) (Vikram Chatterji, Galileo, 33:56) - [Pitfalls and Best Practices: 5 Lessons from LLMs in Production](https://mlopstalks.com/talks/pitfalls-and-best-practices-5-lessons-from-llms-in-production) (Raza Habib, Humanloop, 30:27) - [Evaluating Quality and Improving LLM Products at Scale](https://mlopstalks.com/talks/evaluating-quality-and-improving-llm-products-at-scale) (Austin Bell, Slack, 15:11) - [Designing Human in the Loop Experiences for LLMs](https://mlopstalks.com/talks/designing-human-in-the-loop-experiences-for-llms) (Alberto Rizzoli, V7, 11:40)