Q How strong is it across many tests combined? AA Intelligence Index · Bottom 30%#83
Q How strong is it across many tests combined? AA Intelligence Index · Bottom 30%#83
Bottom 30%#83 of 117 20.4 About AA Intelligence Index More about AA Intelligence Index
Q Can it handle customer service by the rules? TAU2 · Bottom 50%#40
Q Can it handle customer service by the rules? TAU2 · Bottom 50%#40
Bottom 50%#40 of 79 88.3% About TAU2 More about TAU2
Q1 Can it pull scattered facts from several long documents into one answer? Long Context Reasoning · Bottom 17%#98 Q1 Can it pull scattered facts from several long documents into one answer? Long Context Reasoning · Bottom 17%#98 Q2 Can it read a contract hundreds of pages long and find exactly what's needed? CorpFin v2 · Top 37%#23 Q3 Can it find company filings and do financial analysis? Finance Agent v1.1 · Top 44%#16
Q1 Bottom 17%#98 of 116 63.7% Q2 65.3% Q3 54.5% About Long Context Reasoning More about Long Context Reasoning
Q1 Does it avoid making things up when summarizing? Vectara Hallucination Rate · Bottom 37%#24 Q1 Does it avoid making things up when summarizing? Vectara Hallucination Rate · Bottom 37%#24 Q2 Does it avoid making things up when summarizing? Vectara Factual Consistency · Bottom 37%#24
Q1 Bottom 37%#24 of 36 10.7% Q2 89.3% About Vectara Hallucination Rate More about Vectara Hallucination Rate
Q1 Can it operate a computer itself to crack the toughest development jobs? TerminalBench Hard · Bottom 21%#64 Q1 Can it operate a computer itself to crack the toughest development jobs? TerminalBench Hard · Bottom 21%#64 Q2 Can it write code for scientific research? SciCode · Bottom 27%#87
Q1 Bottom 21%#64 of 79 21.2% Q2 40.5% About TerminalBench Hard More about TerminalBench Hard
Q Does it follow tricky rules like format and length without missing any? IFBench · Bottom 17%#69
Q Does it follow tricky rules like format and length without missing any? IFBench · Bottom 17%#69
Bottom 17%#69 of 81 51.2% About IFBench More about IFBench
Q1 Can it solve problems that stump experts across fields? Humanity's Last Exam · Bottom 21%#94 Q1 Can it solve problems that stump experts across fields? Humanity's Last Exam · Bottom 21%#94 Q2 Can it solve graduate-level science problems? GPQA Diamond · Bottom 34%#70
Q1 Bottom 21%#94 of 117 14.9% Q2 82.6% About Humanity's Last Exam More about Humanity's Last Exam
Q MRCR v2 (8-needle) MRCR v2 · Bottom 38%#34
Q MRCR v2 (8-needle) MRCR v2 · Bottom 38%#34
Bottom 38%#34 of 53 50.1% Reasoning effort: Thinking About MRCR v2 More about MRCR v2