Hallucinations include factual mistakes and misleading text that sounds linguistically correct, and they often come from data, validation, encoding, and prompting problems.
2
Token-level probabilities can provide signals about whether an LLM is likely to hallucinate, although completion models often do not expose those probabilities through their APIs.
3
Galileo tested entropy-based metrics and agreement between repeated model outputs to detect hallucinations at the level of individual sentences or text segments.
Summary
Atindriyo Sanyal explains hallucinations as a data quality problem in LLM systems. They include factual errors and misleading answers that sound plausible. He traces them to issues such as overfitting, class imbalance, missing training coverage, weak validation data, encoding mistakes, and poor prompts. His team's approach uses token probabilities, external probability models, and repeated completions to estimate whether an answer contains hallucinated text. The method is designed to work at subtext level rather than assigning one label to a whole response. Sanyal describes experiments using datasets including SelfCheckGPT WikiBio, Self-Instruct Human Evaluation, and OpenAssistant. He discusses log probabilities, PPL5, pseudo-entropy, and agreement between multiple outputs. He reports that pseudo-entropy reached 69% accuracy in one experiment and shows examples of fabricated Shakespeare quotations and nonexistent URLs. Galileo had incorporated these metrics into its products for prompt inspection, model evaluation, and training-data analysis.
Hallucinations are plausible-looking errors with several data-related causes
Sanyal defines hallucinations as factual mistakes and misleading text that may look linguistically correct. He connects them to problems that occur before and during model training, including overfitting, hidden class imbalance, insufficient examples of a particular type, weak validation coverage, encoding mistakes, and poor prompting. He frames hallucination as a data quality issue that can prevent practical LLM systems from reaching production. The problem is especially difficult because a response can read naturally while still being wrong. His examples include invented facts, fabricated text, and other outputs that appear credible until checked.
Token probabilities provide a model-level signal for uncertainty
Sanyal reduces an LLM to a next-token prediction machine for the purposes of evaluation. At each step, the model chooses a token from a probability distribution. He argues that these token-level probabilities reveal information about how the model views its own output. The distribution is not a direct proof that a statement is true or false, but it provides a useful signal for hallucination detection. The approach examines probabilities across a completion instead of relying only on the final text. Sanyal uses this simple view of the model to motivate the metrics tested later in the talk.
The proposed metric needs to detect hallucinations inside a response
Galileo's experiments sought a metric that could automatically quantify hallucinations across different LLM responses. Sanyal says the metric must work at subtext level because a single generated blob may contain only one or two hallucinated sentences. The team used open-ended text generation, collected inputs and model outputs, examined token probabilities, and passed outputs through third-party models for additional signals. They also wanted the metric to support diverse tasks rather than one narrow response format. The evaluation assumed state-of-the-art models such as GPT-3.5 and newer, since older research focused on failure modes that newer models may handle better.
Completion and probability models may need to be separated
The evaluation uses a two-model setup. The completion model produces the response being evaluated, while the probability model supplies token probabilities. In an ideal case they would be the same model, but proprietary model APIs often provide completions without enough probability information. Sanyal describes experiments comparing different combinations of completion and probability models to find useful signals. This separation allows the team to evaluate outputs from models that do not expose the required internal information. It also creates a practical constraint: the quality of the hallucination metric depends partly on how well the chosen probability model reflects the output under examination.
The experiments used both standard datasets and open-ended assistant data
The team explored existing datasets and created some of its own because hallucination evaluation was still a new area. Sanyal names SelfCheckGPT WikiBio, which contains Wikipedia biographies, and Self-Instruct Human Evaluation, which covers open-ended text generation. OpenAssistant gave the team the most promising and difficult test case among the newer models they evaluated. These datasets let the researchers compare metrics across different forms of generated content. The goal was to avoid building a detector that worked only for biographies or only for one fixed question-and-answer format.
Entropy and probability metrics aggregate token signals into response-level scores
The baseline metrics included log probabilities, PPL5, and pseudo-entropy. Log probability takes the log of the probability for each token in a completion. PPL5 measures the entropy of the probability distribution among the top five tokens. Pseudo-entropy is Galileo's heuristic built on Shannon entropy, also using the top five token responses. Because these measures operate at token level while an LLM produces a longer block of text, the team compared average and minimum aggregation. Sanyal describes this as a broad set of experiments across metrics, datasets, and aggregation choices.
Agreement across repeated outputs can reveal hallucination
Galileo also tested multi-model baselines that use a separate model for extra information. One approach asked GPT-3.5 to generate a question and answer, then used the same model as a grader, but Sanyal calls its results mediocre. Another method used token information to reduce the effect of uncertainty in the first few tokens. The strongest result came from ChatGPT agreement, also called ChatGPT friend. GPT-3.5 reran the base model's task several times, and the system compared the outputs. The underlying intuition is that a hallucinated answer is more likely to vary substantially across runs than an answer the model is confident about.
Galileo applied the metrics to prompts, training data, and model outputs
Sanyal reports a result of 69% accuracy for detecting hallucinations with pseudo-entropy in one experiment using DaVinci as both completion and probability model. He also gives a 63% precision figure for one of the tested configurations. The examples included fabricated quotations attributed to Shakespeare's The Tempest and URLs that did not exist. Galileo incorporated hallucination detection alongside its data error potential metric. Sanyal names Prompt Inspector for creating and managing prompts, and describes using depth and the hallucination metric to expose noise in fine-tuning data, evaluation data, and model outputs. He closes by introducing LLM Studio.
"These are outputs of state-of-the-art OpenAI models, quotations from Shakespeare's The Tempest which were completely made up, non-existent books, as well as URLs that do not exist on the web."13:30
Who should watch
You are evaluating an LLM application and need a signal for detecting plausible but false outputs before users see them.
Your model API gives you completions but limited probability information, so you want to understand external probability models and repeated-run checks.
You are building prompts or fine-tuning a model and need to inspect noise in training data, evaluation data, or generated responses.
