BullshitBench v2

Citation frequency 1/3
What does it measure?
A fun benchmark that checks whether an AI clearly points out that a question is built on a nonsense premise instead of confidently answering it anyway.
Who evaluates it, and how?
It asks 100 nonsense questions (13 techniques) across software, finance, legal, medical and physics. Three AI judges (Claude Sonnet 4.6, GPT-5.2 and Gemini 3.1 Pro Preview) sort each answer into clear pushback, partial challenge or accepted nonsense, and the score is the share of questions with clear pushback. The same model is measured separately at each reasoning setting.
Reading the score
Scores run from 0 to 100%. Higher is better. Measured by Peter Gostev (BullshitBench, MIT). Answers are graded by AI judges, so the judges' tendencies affect the score. It does not measure how often a model wrongly rejects a valid question. It is a score for fun and is not used in our use-case rankings.
Top score95.0%
Models tested143
Scoring basisHigher is better
Last updated2026.11.19
Model release date
51 models without a release date are not plotted; they appear only in the table below.
  1. 1
    Anthropic2026.05.27 · none
    95.0%
  2. 1
    Alibaba
    qwen/qwen3.8-max
    minimal
    95.0%
  3. 3
    Anthropic2026.02.17 · high
    91.0%
  4. 4
    Google2026.11.12 · xhigh
    90.0%
  5. 4
    Alibaba2026.11.16 · max
    90.0%
  6. 4
    Anthropic2025.11.24 · high
    90.0%
  7. 7
    Anthropic2026.02.04 · high
    87.0%
  8. 8
    Google2026.11.18 · xhigh
    84.0%
  9. 9
    Anthropic2026.04.16 · none
    83.0%
  10. 10
    Anthropic2026.06.30 · low
    80.0%

The "harness" is the agent program the AI used to carry out the task. The same model can score very differently depending on its harness and reasoning effort.