High-stakes applications need explicit controls for failure modes such as distribution shifts, bias, poor performance in low-resource settings, and speech recognition problems for accented users.
2
Human review, smaller subtasks, structured prompts, prompt databases, fine-grained evaluation, and model ensembles can reduce the risks of using LLMs.
3
Teams should evaluate models across user cohorts, measure calibration and robustness, and avoid using an LLM when a simpler, more interpretable method is enough.
Summary
Yada Pruksachatkun explains why LLM applications in medicine, law, therapy, and other high-stakes settings need more than strong headline results. She uses a therapy bot as an example, showing how controllability, structured conversation flows, bias, fairness, and speech recognition errors can affect users. Earlier approaches, such as intent and entity systems with explicit dialogue states, provide useful design ideas. She recommends human review, breaking large tasks into smaller ones, reducing the input and output space, maintaining a prompt database, and building fine-grained evaluation sets. Ensembles can combine several models, but teams should use regex or traditional models when those are sufficient. Owning model weights can provide more control and access to confidence scores. Evaluation should cover user cohorts, robustness, and calibration. Pruksachatkun also warns that results from public benchmarks may not transfer to a specific domain, especially when privacy and other constraints differ.
High-stakes systems need explicit controls for predictable failure modes
Pruksachatkun says LLMs can be useful in medicine, law, and similar areas, but they have many failure cases. These include robustness problems under distribution shifts, degradation from equivalent perturbations, and weak performance in low-resource settings. Her therapy bot example adds practical concerns: the bot needs a defined framework such as CBT or family dynamics, and it needs controllability, bias checks, and fairness checks. A speech-to-text component can perform worse for people with accents, creating downstream problems in the user experience.
Older dialogue systems offer useful patterns for controlling LLM applications
For controllability, Pruksachatkun recommends looking at practices from earlier machine learning systems. She points to the period before language models, when dialogue systems used intents, entities, conversation trees, and branches. These systems also tracked dialogue state. In a therapy bot, an intake flow could record social history and emergency contacts. The point is to make important parts of the interaction explicit instead of leaving every decision to an open-ended language model.
Human review should cover ordinary use and dangerous edge cases
Pruksachatkun calls a human in the loop a standard practice for high-stakes applications. When a domain expert, such as a doctor, uses the product, the system can show its output for that person to check. When the end user is not an expert, people with relevant expertise can monitor alerts and potential fatal cases in the background. This gives the system a way to catch serious errors that automated evaluation may miss.
Smaller tasks and smaller answer spaces are easier to manage
One way to reduce difficulty is to divide a large task into smaller tasks. For information retrieval, a system might assess each paragraph for relevance instead of trying to handle an entire document at once. Pruksachatkun also says classification is usually easier than generation, and that reducing the output space helps. Choosing among ten classes is easier than choosing among a thousand. Reducing input variability can also make a task easier for the model.
Prompt work needs a database, retrieval, and detailed tests
For applications using off-the-shelf models, Pruksachatkun recommends keeping a prompt database. The system can retrieve examples with embeddings, and those embeddings may need fine-tuning for the particular use case. She also recommends a structured process for building prompts and test sets. Fine-grained evaluation suites help teams identify exactly where the model fails instead of relying on one overall score.
Ensembles can combine information from several models
Pruksachatkun connects model ensembles with older data science practice. Self-consistency and similar methods use multiple samples or predictions, and the same idea can extend to language models. Teams can combine black-box APIs with their own fine-tuned models, or use several models they control. The goal is to bring more information into a prediction rather than depending on one model response.
Her advice for high-stakes systems is to avoid LLMs when they are not necessary. Regex, logistic regression, and random forests can be useful alternatives, even though they also have limitations. Traditional models may make their weights or behavior easier to inspect. Pruksachatkun also warns that factuality remains a problem as the input, context window, and grounding material grow. Human involvement is still needed for emotional intelligence.
Evaluation must cover cohorts, calibration, and domain differences
Pruksachatkun recommends evaluating performance across the different cohorts who may use the product. Teams should measure robustness and calibration, meaning the relationship between a model's confidence and its correctness. Dialogue systems can also use user simulators. Public claims should be treated carefully: teams need to ask how similar the reported task is to their own task and how similar the reported domain is to theirs. Privacy and robustness constraints can make published results fail to transfer.
