Key takeaways
- Sampling-based detection like SelfCheckGPT resamples the model 5-20 times and flags claims that change across generations, but it misses wrong answers the model states consistently.
- NLI entailment scoring is a cheap, deterministic per-claim check that over-flags paraphrases and under-flags subtle numeric errors, so neutral labels need careful handling.
- Retrieval-grounded verification is the only method that can contradict a document, but it inherits stale-index and bad-retrieval failure modes.
- Run detection as a cascade: NLI screening first, sampling on borderline scores, retrieval on claims that matter, with two thresholds routing the band between them to human review.
- No detector replaces sampled human review; every score is a routing signal for which borderline answers deserve a human, not proof the rest are correct.
LLM hallucination detection catches answers whose confidence outruns their justification before they reach your users or your LLM judge. The three practical method families are sampling-based consistency checks, entailment scoring with NLI models, and retrieval-grounded verification. This guide compares how each works, what it costs in latency and tokens, and which fits your pipeline.
What is LLM hallucination detection and why does it matter?
Hallucination detection is the process of scoring whether an LLM output is actually supported by its own reasoning, its training distribution, or an external source, rather than whether it reads well. A multi-step agent can carry a confident-sounding fabrication through four tool calls before anything checks it, and downstream evals only see the final answer.
It differs from LLM-as-a-judge quality scoring: a judge grades against a rubric (tone, completeness, policy fit), while hallucination detection asks a narrower question, whether the claims hold. The two miss different things, which is why a judge can pass a response a human reviewer flags.
The common sampling-based approach, SelfCheckGPT, resamples the model several times and measures consistency across answers: a claim the model states differently on each pass is a hallucination candidate.
How do sampling-based methods like SelfCheckGPT detect hallucinations?
Sampling-based detection treats a hallucination as an inconsistency problem: if a model genuinely knows a fact, restating the same prompt should produce roughly the same answer. SelfCheckGPT samples N additional responses (typically 5-20) and scores each claim in the original answer against the samples, using an LLM prompt, BERTScore, QA overlap, or NLI to measure agreement. Low agreement across samples flags the claim as unsupported. The core assumption is that stochastic sampling exposes what the model actually "knows" versus what it confidently fabricated once.
Consistency sampling vs token-probability signals
A cheaper variant skips extra generations entirely and reads the model's own token logprobs. If the tokens making up a factual claim carry low probability, or if asking the model to rate its own answer yields low confidence, that's a signal. The trade-off: logprobs are a single forward pass, so they're fast, but they measure local token surprise, not whether the claim survives independent restatement. A fabricated name can appear with high probability if the model committed to it early in the generation. Sampling consistency is slower but tests something closer to the claim itself. In practice, teams use logprobs as a pre-filter and reserve sampling for claims that look borderline, which cuts the cost of running 10 extra generations on every answer in your eval samples.
Cost and latency trade-offs
The math is unforgiving at pipeline scale. Sampling 10 completions per answer multiplies your inference bill roughly tenfold, and consistency checks add several seconds of latency, which rules them out for synchronous user-facing paths. That's why the pattern that works is asynchronous: run sampling-based checks on a sample of production traffic or on borderline answers your judge already flagged, not on every token of every response. Token-probability checks fit inline; consistency sampling fits your offline eval loop.
Where sampling-based methods fail is structural: a model can be consistently wrong if the error is baked into its weights, so high agreement across samples tells you the answer is stable, not that it's true. This is the divergence problem in miniature: the signal measures self-consistency, and a confident, wrong answer is perfectly self-consistent. It also degrades on multi-hop claims, where one bad intermediate step poisons agreement scoring for the whole chain.
How do entailment and NLI models score factual consistency?
Entailment-based detection treats hallucination scoring as a natural language inference problem: given a source text and a claim, an NLI model outputs entailment, contradiction, or neutral. You split the LLM's answer into atomic claims, score each against the source passage, and aggregate the labels into a consistency score. Because the model runs once per claim on fixed inputs, results are deterministic and cheap compared with sampling several full generations, which is what methods like SelfCheckGPT require.
Claim decomposition and per-claim scoring
Decomposition quality decides ceiling and floor. A claim like "Acme was founded in 2011 and is headquartered in Berlin" must be split before scoring, or one false fact can drag an otherwise supported sentence to neutral. Per-claim scores also give you actionable output: you can flag the specific sentence that failed instead of rejecting the whole answer, which fits rubric-based judge pipelines that already score at the criterion level.
Where NLI models over- and under-flag
NLI models over-flag paraphrases and under-flag subtle numeric or temporal errors. Neutral is the failure mode to watch: a claim the model cannot verify reads as "not contradicted," and naive pipelines count that as passing. Calibrate against your own eval samples, and expect divergence between NLI labels and human reviewers on borderline tone and policy issues, the same gap you see with LLM judges.
When should you use retrieval-grounded verification instead of model self-evaluation?
Retrieval-grounded verification beats self-evaluation when the answer cites facts a pipeline can check: names, numbers, dates, policy clauses. Sampling-based methods catch answers the model itself is shaky on, but a fluent answer generated from a confidently wrong internal belief passes every self-consistency test. Grounding against a source of truth is the only method that can say "this contradicts the document" rather than "the model seems unsure."
The cost is infrastructure: a retrieval layer, a chunking strategy, an entailment or comparison step per claim, and added latency on every response. You also inherit retrieval's failure modes. If your index is stale or your retriever pulls the same wrong page the training data echoed, verification certifies the error with higher confidence than the raw answer had. Garbage retrieved is garbage verified. A practical split: use retrieval-grounding for answers tied to documents you control (RAG outputs, support replies, policy questions), and self-evaluation for open-ended generation where no canonical source exists. If your LLM judge and human reviewers diverge on borderline safety calls, grounding the rubric in retrieved evidence gives both sides the same artifact to argue about, which is where ClearRun focuses: scoring whether an answer's stated confidence is justified by its cited reasoning, with a signed record when the checkpoint fails.
Which detection method fits your pipeline? Side-by-side comparison
| Sampling-based | Entailment/NLI | Retrieval-grounded | |
|---|---|---|---|
| Latency | High (multiple generations) | Low (one classifier pass) | Highest (retrieval + verification) |
| Marginal cost | Several LLM calls per answer | Cheap model inference | LLM calls plus retrieval infra |
| Infrastructure | None beyond your model | A hosted NLI checkpoint | Vector store, retrieval, source corpus |
| Failure modes | Misses consistent-but-wrong answers; noisy on borderline samples | Entailment errors on long or implicit claims | Bad retrieval or stale corpus looks like a hallucination fix |
Match the family to your failure surface. Sampling-based fits closed-book generation where you have no trusted corpus. NLI scoring works when you need cheap per-claim consistency checks; the SummaC results showed entailment models transfer reasonably across summarization tasks (SummaC paper). Retrieval-grounded verification is the only option when external facts decide correctness.
Combining methods in one checkpoint
Run them as a cascade, not a vote: NLI first for cheap screening, sampling only on borderline scores, retrieval on claims that matter. Honest note on our own product: ClearRun scores confidence-versus-justification divergence but does no retrieval grounding, so it complements rather than replaces the third column.
Which open-source hallucination detection tools are on GitHub?
SelfCheckGPT is the reference implementation for sampling-based detection: it generates multiple stochastic samples from the same model, then scores each sentence by consistency across samples using NLI or similarity measures. The repo ships the full pipeline, so you can run it against your own model rather than a hosted API. Its main cost is latency, since every check requires extra generations.
Beyond SelfCheckGPT, most open-source work lives inside eval frameworks. DeepEval (Confident AI) includes a faithfulness and hallucination metric you can run in CI, and Patronus AI publishes eval tooling with hallucination-focused scorers. If you want to bolt detection onto an existing LLM judge rather than replace it, these are the shortest paths.
Commercial platforms with hallucination checks
Braintrust, EvalGuard, and Patronus AI all position hallucination checks as part of a broader eval platform: scorers plus experiment tracking plus datasets, accessed through SDKs and dashboards. The tradeoff versus rolling your own is control. If you need to inspect or modify the scoring logic, open source wins; if you need managed scoring alongside the rest of your eval stack, the platforms win. ClearRun sits closer to the open-source end: CLI, SDK, and MCP server with no dashboard, focused specifically on flagging answers whose confidence outruns their reasoning.
How do you integrate hallucination detection into a production LLM pipeline?
A workable pattern has five stages: sample the answer (k generations or a paraphrase pass), score consistency with your chosen method, apply a threshold, route borderline answers, and log a signed record for anything that ships. Keep the scorer out of band so a scorer outage degrades to logging, not to dropped traffic. If you use retrieval-grounded scoring, RAGAS gives you faithfulness and answer-relevance metrics you can wire into this loop as an open-source evaluation library.
Thresholding and routing borderline answers
Set two thresholds, not one. Above the high threshold, ship. Below the low threshold, regenerate or fall back to retrieval. The band between them goes to a human reviewer or a second judge pass. Calibrate the band against your own labeled eval samples; a judge that overflags money mentions but misses coercion (a divergence pattern seen in real reviewer workflows) needs separate thresholds per failure mode, not one global cutoff.
Auditing and signed records
For decisions that matter, log the answer, the score, the sampled evidence, and the threshold version, then sign the record. ClearRun does this: it flags answers whose confidence outruns their reasoning and emits a signed record via CLI, SDK, or hosted API. The honest gap: a signed record proves what you checked, not that the answer is correct. If your downstream consumers need end-to-end verification rather than a checkpoint, a retrieval-grounding tool may fit better.
What are the limitations and failure modes of current detection methods?
Every method above fails the same way in one case: a wrong answer stated consistently. SelfCheckGPT-style sampling sees low variance because the model genuinely believes the claim. NLI sees entailment because the passage and answer were generated together. Retrieval grounding helps only if the corpus contains the correction, and it can pass an answer that cites a real but misread source.
The second failure is judge/human divergence. Your LLM judge can be right on the literal rubric and still miss what a human reviewer catches: a policy violation expressed politely, or a correct answer that's wrong for the context. That's a rubric gap, not a scoring bug, and no detector closes it for you.
Third is calibration drift. A judge calibrated against last quarter's prompts overflags new patterns and misses old ones; the fix is periodic re-scoring against sampled human labels, not a better prompt.
Which is the honest limit: no detection method replaces sampled human review. Treat every score, including ClearRun's confidence-versus-reasoning check, as a routing signal that tells you which borderline answers deserve a human, not which ones don't.
FAQ
Can LLM hallucination detection be fully automated, or do humans still need to review?
No method replaces sampled human review. Every detector, from consistency sampling to retrieval grounding, misses the same case: a wrong answer stated consistently. Treat scores as routing signals that decide which borderline answers deserve a human, not as proof the rest are correct. Reviewers also stay in the loop for calibration and rubric gaps.
How much does adding hallucination detection increase latency and cost per response?
It depends on the method. Sampling 10 completions per answer multiplies your inference bill roughly tenfold and adds seconds of latency, so it belongs in offline eval loops. NLI scoring is a single cheap classifier pass per claim and fits inline. Retrieval grounding adds the most latency, requiring a retrieval step plus verification on every response.
Is a hallucination score the same thing as an honesty score?
No. A hallucination score measures whether claims are supported by evidence or consistency checks. An honesty score would measure whether the model communicates its own uncertainty, which is a different property. An answer can be fully supported and still overconfident, or hedged and still wrong. The two signals correlate but measure different failure surfaces.
Do I need RAG before hallucination detection works?
Only for retrieval-grounded verification. Sampling-based and NLI methods work on closed-book generation with no trusted corpus, though they can only flag instability or unsupported claims, never contradict a document. If your answers cite facts from documents you control, grounding against that corpus is the only method that can say a claim is false, not just shaky.
How do I calibrate my hallucination threshold against human reviewer labels?
Re-score a set of your own labeled eval samples, then set two thresholds: ship above the high one, regenerate below the low one, and route the band between to human review. Calibrate per failure mode rather than with one global cutoff, and re-run the calibration periodically, since a judge calibrated on old prompts drifts on new patterns.