Self-consistency checks can miss answers that are consistently wrong or mark correct answers as inconsistent.
2
SAC3 checks consistency across semantically equivalent questions and across different language models.
3
Consistency-based mitigation can combine model responses or split long-context comparisons into sentence-level checks.
Summary
Jiaxin Zhang explains why hallucination detection matters for products such as Intuit's financial assistants, where incorrect answers can damage trust. He reviews factuality hallucinations, where an answer is wrong, and faithfulness hallucinations, where a generated answer does not stay grounded in its source. Zhang then examines self-consistency methods such as SelfCheckGPT. These methods can fail when a model repeatedly gives the same wrong answer, or when sampling makes a correct answer appear inconsistent. His proposed SAC3 method adds semantically equivalent question perturbations and cross-model checks. He also presents two mitigation methods: response fusion across models, and divide-and-conquer reasoning for long text comparisons. The talk includes benchmark results, sampling and latency considerations, and open problems involving long-form generation, multimodal models, model cost, knowledge boundaries, and the balance between creativity and factuality.
Hallucinations can be factual errors or failures to stay grounded in a source
Zhang separates hallucinations into factuality and faithfulness problems. A factuality hallucination gives an incorrect answer to a question, such as identifying the wrong first person to walk on the moon. A faithfulness hallucination appears when a model summarizes a supplied article but adds information that the article does not contain. He also describes intrinsic hallucinations, which conflict with the source or introduce logical inconsistency, and extrinsic hallucinations, which add unsupported information without directly contradicting the source. Detection methods therefore need to account for both outside factual correctness and consistency with the provided context.
Self-consistency can miss answers that are consistently wrong
SelfCheckGPT samples several answers to the same question and treats agreement as evidence that the model knows the answer. Zhang shows why this assumption is incomplete. When asked whether pi is smaller than 0.2, a model may repeatedly give the same wrong answer. The responses are consistent, but they are not factual. The opposite problem can also occur: a deterministic answer may be correct, while higher-temperature samples disagree with it. A simple consistency score could then classify the correct answer as hallucinated. Zhang uses these cases to motivate checks beyond repeated sampling of one unchanged question.
SAC3 adds question perturbations and cross-model checking
SAC3, or semantic-aware cross-check consistency, expands self-consistency in two directions. First, it creates semantically equivalent versions of the user question. If the model understands the question and has the relevant knowledge, its answers should remain aligned after these changes. Second, it asks other verifier models to answer the question, rather than relying on one model such as GPT-4 or GPT-3.5 Turbo. SAC3 combines self consistency, cross-question consistency, cross-model consistency, and cross-model question consistency into an overall score. The method can use black-box or open-source verifier models.
The SAC3 evaluation uses both classification and generation tasks
Zhang evaluates SAC3 on question-answering settings that include prime-number classification and checking whether a person graduated from a specified school. He compares the full method with a naive self-consistency baseline and reports a significant improvement when question perturbations, model perturbations, and cross-checking are included. He also evaluates generation tasks such as HellaSwag and open-domain question answering. Sampling does not have to be large: Zhang says three to five samples can provide competitive performance while reducing cost compared with larger sample sets.
Parallel execution can reduce the cost of sampling-based checks
Because SAC3 samples several responses and performs several consistency checks, runtime and cost are practical concerns. Zhang describes a package that parallelizes the sampling paths and the consistency checks. In his example, the time for a question-answering query falls from roughly 15 seconds to around two or three seconds. The design also allows teams to choose their own verifier models. Smaller models such as Falcon can be included, and the method can assign different weights to models when a particular verifier is considered more reliable for the task.
Response fusion can use agreement across several models to mitigate hallucinations
Zhang's first mitigation method is a black-box, zero-resource approach called model check and fusion. The input is sent to several models, which can be black-box or open-source. A consistency procedure compares the responses, ranks them, and identifies the most consistent information. A fusion step then combines the responses, using the strengths of each model while reducing weaknesses in individual outputs. Zhang gives an example that combines responses from three models, including GPT-4 and PaLM 2, and reports better performance than using a single model for that dataset.
Divide-and-conquer reasoning checks long summaries sentence by sentence
Long-context consistency is difficult when a reference paragraph is compared directly with a short summary. Zhang's divide-and-conquer reasoning method breaks the reference and candidate into sentence-level checks. One evaluator decides whether each sentence is consistent and provides a reason. A second component converts these decisions and reasons into a numerical score. A third agent uses the reasons for inconsistency to revise the candidate so it better matches the reference. Zhang reports that this approach improves consistency evaluation on semantic consistency and summarization benchmarks, and that it reduces output inconsistency by nearly 90 percent in his experiments.
Hallucination-free generation remains out of reach for current systems
Zhang is direct about the remaining limits. Long-form text and document-level generation can still contain hallucinations, even when retrieval is used. Multimodal models create additional detection problems beyond text-only systems. He also points to a cost tradeoff when mitigation uses several models and multiple improvement rounds. Other open questions concern whether models can identify the boundaries of their knowledge and how to balance creativity with factuality. The acceptable balance depends on the application: a wrong number in a financial or healthcare setting can cause serious harm, while a creative writing task may tolerate or even benefit from invention.
