LLMs can produce plausible but incorrect answers, so Stripe broke support answering into classification, topic identification, response generation, and tone adjustment.
2
Offline evaluations did not predict agent adoption, which made online feedback, proxy metrics, and early user experience work necessary.
3
Better labeled data improved performance more than newer GPT engines, so Stripe moved toward classification, weak supervision, and a continuously maintained support dataset.
Summary
Sophie Daly describes Stripe's first LLM application for support agents. The system did not answer customers directly. It prompted agents with responses that were meant to be accurate, concise, and consistent with Stripe's tone. Stripe found that an out-of-the-box GPT model often gave plausible but wrong answers, so the team split the task into several fine-tuned stages. These stages filtered questions, identified topics, generated answers from relevant context, and adjusted the tone. Offline testing looked strong, but agents used the prompts less often than expected. Stripe lacked direct online accuracy labels, so the team built a match-rate heuristic by comparing generated responses with the responses agents actually sent. Daly argues that production LLM work needs monitoring and online feedback from the start. She also says data quality mattered more than moving to newer models. For the next iteration, Stripe replaced generative output with simpler classification methods and planned to use weak supervision and subject matter experts to keep the training data current.
Out-of-the-box GPT answers can sound right while being wrong
Stripe tested DaVinci GPT with a question such as, "How can I pause payouts?" The answer sounded plausible, but it was incorrect. Daly says this happened across much of Stripe's support space because the model's pre-training could be outdated, incomplete, or mixed with instructions from another payments company. Prompt engineering could improve individual answers, but the support domain was too broad and complex for prompts alone to solve it.
Breaking the task into smaller models gave Stripe more control
Stripe split support answering into sequential steps. The system first decided whether an input was a valid support question, then identified its topic, generated an answer using topic-relevant context, and adjusted the result to Stripe's friendly but succinct tone. The first classification steps filtered questions before they reached the response model. Daly says fine-tuning removed hallucinations in their case and required about 500 labels per class, which let the team rely on expert agent annotations for its pilot.
Offline quality did not predict whether agents would use the prompts
The team used labeled datasets to evaluate classification models and asked expert agents to review generated responses. It also worked with agents on user testing and training-data collection. These offline results looked strong before launch. After shipping, however, agents used the generated prompts far less than expected. They were accustomed to their existing way of solving cases, and the team had little visibility into whether online answers were accurate.
A proxy match rate provided a useful online signal
Because labeling online cases at the required scale was not feasible, Stripe compared generated responses with the responses agents actually sent to users. Daly describes this match rate as a crude lower-bound measure of expected accuracy. It did not provide a complete evaluation, but it gave the team directional information about model behavior in production instead of leaving them unable to tell whether offline and online performance differed.
Agent behavior made user experience work part of the ML problem
Positive user testing and good online model trends did not translate into efficiency gains because agents often ignored the prompts. Daly says the team needed a much larger user experience effort to increase adoption. Her practical lesson is to ask early whether human behavior can affect the business problem and to involve the UX team before the system is ready to ship.
Every stage should reach shadow mode before the full system ships
Daly recommends shipping each stage of a multi-step framework in shadow mode as soon as it is ready. This lets the team debug components while development continues, check that each stage works, and validate expectations one step at a time. Waiting for one large end-to-end launch made it harder to find where the system or adoption process was failing.
Monitoring belongs in the definition of a shipped model
Stripe initially treated monitoring as something a small, resource-constrained team might add later. The production experience changed that view. Daly says a model is not shipped unless it has full monitoring in a dashboard, because online feedback shows whether the system is solving the intended problem. Proxy metrics can be imperfect, but they are still better than having no production signal.
Data work mattered more than adopting a newer model
Daly says writing the LLM framework took days or weeks, while improving the training dataset took much longer. Changes to label quality produced larger performance gains than switching to more advanced GPT engines. The observed errors were usually tied to specific Stripe support issues rather than broad language understanding failures. Adding better or more representative samples usually addressed the gap.
Stripe moved toward simpler models and living support data
Collecting labels for generative fine-tuning added substantial complexity, so Stripe's second iteration replaced the generative component with simpler classification. The team planned to use weak supervision through Snorkel and embedding classification to label data at scale without explicit labels for every example. Subject matter experts would maintain the dataset as Stripe's products and support space changed, keeping labels current.
"A good data strategy will outweigh any fancy LLM architecture, especially for solving business problems that require deep domain expertise at scale."10:34
Who should watch
You are building an LLM feature for an internal operations team and need to measure whether people actually use it after launch.
Your offline evaluation looks strong, but you lack reliable production labels or visibility into model behavior.
You are deciding whether to invest in a newer model or improve domain data, labeling, and subject matter expert review.
