MMLU-Pro

In plain words · How many college-level questions across 14 subjects it gets right

What does it measure?

Artificial Analysis now classifies this as 'legacy' and rarely measures new models, so recent ones may be missing. A harder version of MMLU, the exam built from university-level subjects. It tests knowledge and reasoning across a broad range of fields — law, medicine, engineering, history and more.

Who evaluates it, and how?

Researchers at the University of Waterloo and elsewhere expanded MMLU from four to ten answer options, removed questions that were too easy or flawed, and rebuilt it into about 12,000 questions across 14 fields. Scored as accuracy; Artificial Analysis runs it independently.

Reading the score

0–100%. With ten options, guessing gives 10%. It is designed to score 16–33 points lower than the original MMLU, so do not compare it directly with MMLU numbers elsewhere.