# Reliable Hallucination Detection in Large Language Models

Jiaxin Zhang, Intuit AI Research | AI in Production 2024 | 35:24

Source: https://www.youtube.com/watch?v=G5tBnPEyjLc
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/reliable-hallucination-detection-in-large-language-models
Published: 2024-04-25
Tags: cost, evals, guardrails, structured-outputs

## TL;DR
- Self-consistency checks can miss answers that are consistently wrong or mark correct answers as inconsistent.
- SAC3 checks consistency across semantically equivalent questions and across different language models.
- Consistency-based mitigation can combine model responses or split long-context comparisons into sentence-level checks.

## Summary
Jiaxin Zhang explains why hallucination detection matters for products such as Intuit's financial assistants, where incorrect answers can damage trust. He reviews factuality hallucinations, where an answer is wrong, and faithfulness hallucinations, where a generated answer does not stay grounded in its source. Zhang then examines self-consistency methods such as SelfCheckGPT. These methods can fail when a model repeatedly gives the same wrong answer, or when sampling makes a correct answer appear inconsistent. His proposed SAC3 method adds semantically equivalent question perturbations and cross-model checks. He also presents two mitigation methods: response fusion across models, and divide-and-conquer reasoning for long text comparisons. The talk includes benchmark results, sampling and latency considerations, and open problems involving long-form generation, multimodal models, model cost, knowledge boundaries, and the balance between creativity and factuality.

## Key ideas
### Hallucinations can be factual errors or failures to stay grounded in a source
[04:36](https://www.youtube.com/watch?v=G5tBnPEyjLc&t=276s)
Zhang separates hallucinations into factuality and faithfulness problems. A factuality hallucination gives an incorrect answer to a question, such as identifying the wrong first person to walk on the moon. A faithfulness hallucination appears when a model summarizes a supplied article but adds information that the article does not contain. He also describes intrinsic hallucinations, which conflict with the source or introduce logical inconsistency, and extrinsic hallucinations, which add unsupported information without directly contradicting the source. Detection methods therefore need to account for both outside factual correctness and consistency with the provided context.

### Self-consistency can miss answers that are consistently wrong
[11:29](https://www.youtube.com/watch?v=G5tBnPEyjLc&t=689s)
SelfCheckGPT samples several answers to the same question and treats agreement as evidence that the model knows the answer. Zhang shows why this assumption is incomplete. When asked whether pi is smaller than 0.2, a model may repeatedly give the same wrong answer. The responses are consistent, but they are not factual. The opposite problem can also occur: a deterministic answer may be correct, while higher-temperature samples disagree with it. A simple consistency score could then classify the correct answer as hallucinated. Zhang uses these cases to motivate checks beyond repeated sampling of one unchanged question.

### SAC3 adds question perturbations and cross-model checking
[13:21](https://www.youtube.com/watch?v=G5tBnPEyjLc&t=801s)
SAC3, or semantic-aware cross-check consistency, expands self-consistency in two directions. First, it creates semantically equivalent versions of the user question. If the model understands the question and has the relevant knowledge, its answers should remain aligned after these changes. Second, it asks other verifier models to answer the question, rather than relying on one model such as GPT-4 or GPT-3.5 Turbo. SAC3 combines self consistency, cross-question consistency, cross-model consistency, and cross-model question consistency into an overall score. The method can use black-box or open-source verifier models.

### The SAC3 evaluation uses both classification and generation tasks
[16:36](https://www.youtube.com/watch?v=G5tBnPEyjLc&t=996s)
Zhang evaluates SAC3 on question-answering settings that include prime-number classification and checking whether a person graduated from a specified school. He compares the full method with a naive self-consistency baseline and reports a significant improvement when question perturbations, model perturbations, and cross-checking are included. He also evaluates generation tasks such as HellaSwag and open-domain question answering. Sampling does not have to be large: Zhang says three to five samples can provide competitive performance while reducing cost compared with larger sample sets.

### Parallel execution can reduce the cost of sampling-based checks
[17:56](https://www.youtube.com/watch?v=G5tBnPEyjLc&t=1076s)
Because SAC3 samples several responses and performs several consistency checks, runtime and cost are practical concerns. Zhang describes a package that parallelizes the sampling paths and the consistency checks. In his example, the time for a question-answering query falls from roughly 15 seconds to around two or three seconds. The design also allows teams to choose their own verifier models. Smaller models such as Falcon can be included, and the method can assign different weights to models when a particular verifier is considered more reliable for the task.

### Response fusion can use agreement across several models to mitigate hallucinations
[19:30](https://www.youtube.com/watch?v=G5tBnPEyjLc&t=1170s)
Zhang's first mitigation method is a black-box, zero-resource approach called model check and fusion. The input is sent to several models, which can be black-box or open-source. A consistency procedure compares the responses, ranks them, and identifies the most consistent information. A fusion step then combines the responses, using the strengths of each model while reducing weaknesses in individual outputs. Zhang gives an example that combines responses from three models, including GPT-4 and PaLM 2, and reports better performance than using a single model for that dataset.

### Divide-and-conquer reasoning checks long summaries sentence by sentence
[21:31](https://www.youtube.com/watch?v=G5tBnPEyjLc&t=1291s)
Long-context consistency is difficult when a reference paragraph is compared directly with a short summary. Zhang's divide-and-conquer reasoning method breaks the reference and candidate into sentence-level checks. One evaluator decides whether each sentence is consistent and provides a reason. A second component converts these decisions and reasons into a numerical score. A third agent uses the reasons for inconsistency to revise the candidate so it better matches the reference. Zhang reports that this approach improves consistency evaluation on semantic consistency and summarization benchmarks, and that it reduces output inconsistency by nearly 90 percent in his experiments.

### Hallucination-free generation remains out of reach for current systems
[26:48](https://www.youtube.com/watch?v=G5tBnPEyjLc&t=1608s)
Zhang is direct about the remaining limits. Long-form text and document-level generation can still contain hallucinations, even when retrieval is used. Multimodal models create additional detection problems beyond text-only systems. He also points to a cost tradeoff when mitigation uses several models and multiple improvement rounds. Other open questions concern whether models can identify the boundaries of their knowledge and how to balance creativity with factuality. The acceptable balance depends on the application: a wrong number in a financial or healthcare setting can cause serious harm, while a creative writing task may tolerate or even benefit from invention.

## Notable quotes
- "Self checking the step consistency of LMs is not sufficient for detecting factuality." (11:34)
- "If the model has knowledge of a given concept, sampled responses are likely to be similar and contain consistent facts." (09:59)
- "Zero hallucination is impossible at the current stage." (27:46)
- "Consistency is pretty essential and helpful for detecting, evaluating, and mitigating hallucinations." (27:14)

## Tools & references mentioned
- Intuit
- Intuit AI Research
- TurboTax
- QuickBooks
- SelfCheckGPT
- SAC3
- GPT-4
- GPT-3.5 Turbo
- Falcon
- PaLM 2
- Llama
- HellaSwag
- StarEval
- GPTScore
- RAG
- RM Blender
- DCR

## Who should watch
- You are building an LLM feature where incorrect or unsupported answers need to be detected without access to model internals.
- Your current evaluation relies on repeated samples from one model, and you want to test question perturbations or multiple verifier models.
- You work on summarization or long-context generation and need a way to score and revise sentence-level inconsistencies.

## Related talks

- [Stopping Hallucinations From Hurting Your LLMs](https://mlopstalks.com/talks/stopping-hallucinations-from-hurting-your-llms) (Atindriyo Sanyal, Galileo, 15:09)
- [Controlled and Compliant AI Applications](https://mlopstalks.com/talks/controlled-and-compliant-ai-applications) (Daniel Whitenack, Prediction Guard, 25:13)
- [Evaluating the Effectiveness of Large Language Models](https://mlopstalks.com/talks/evaluating-the-effectiveness-of-large-language-models) (Aniket Singh, Ultium Cells, 35:41)
- [Model Blind Spot Discovery for Better Models](https://mlopstalks.com/talks/model-blind-spot-discovery-for-better-models) (Pavol Bielik, LatticeFlow & David Garnitz, VectorFlow, 55:35)
- [Incorporating LLMs in High-stake Use Cases](https://mlopstalks.com/talks/incorporating-llms-in-high-stake-use-cases) (Yada Pruksachatkun, Moonhub, 11:01)