Five frontier LLMs disagree on 67% of 1k real-world fact-check claims | Hacker News

A Hacker News thread questions a study on LLM fact-checking that found five frontier models disagreed on 67% of 1,000 claims.

Commenters say the result may reflect vague verdict labels like “mostly true” and “misleading,” not just model inconsistency.

They also note many claims can plausibly get multiple answers, especially without an “unknown” or abstain option.

The discussion raises doubts about whether the benchmark measures real factual disagreement or simply weak evaluation design.