Vectara Factual Consistency

In plain words · How often its summaries stick to what the document actually says

What does it measure?

How faithfully a document summary sticks to the source. It is close to the hallucination rate flipped around (roughly 100% − hallucination rate) — the same information viewed as 'higher is better'.

Who evaluates it, and how?

The search company Vectara has AIs summarise more than 7,700 documents spanning news, technology, medicine, law and more, and its own judge model HHEM (currently version 2.3) checks whether the summary contains facts absent from the source. The document set is private, so it cannot be learned in advance. This table takes the values from the public leaderboard; an AI that has since been removed or renamed there is listed separately below the table with its last published value, unranked.

Reading the score

0–100%, higher is better. It nearly mirrors the hallucination rate, but the two are published separately and can differ slightly in the decimals. It measures summarisation faithfulness only and says nothing about general knowledge accuracy.

Last updated: 2026-09-22