For the last few years, the AI community has treated benchmarks like simple exams: a static test of intelligence that could be graded cheaply. However, as models evolve into agents—systems capable of using tools, navigating the web, and performing long-term planning—the cost of grading them has skyrocketed. We have reached a point where verifying an AI's performance is no longer a trivial overhead but a significant compute-heavy operation, effectively becoming a new bottleneck for the field.

Consider the Holistic Agent Leaderboard (HAL). It recently spent $40,000 just to test a handful of models on agent-based benchmarks. Unlike older methods where you could simply feed a few thousand questions into an API and count the correct answers, evaluating an agent requires watching it perform a multi-step task, often involving error-prone 'scaffolding'—the software wrappers that help the AI interact with the outside world. This process is sensitive and expensive, with costs sometimes varying by orders of magnitude for the same task.

The fundamental problem is that static benchmarks, which test a model’s raw prediction capabilities, allowed for clever shortcuts. Researchers found they could compress these tests by 100x or 200x without losing accurate rankings of which model is best. But agent evaluations are different; because they involve agents performing dynamic, multi-turn interactions, researchers cannot simply subsample the data. Furthermore, some modern benchmarks go a step further, requiring 'training-in-the-loop' where the model must actively learn or optimize during the evaluation process itself. This makes the evaluation process behave less like an exam and more like an experiment that requires massive GPU resources to complete.

This shift creates a troubling 'accountability barrier' for independent researchers and academic groups. If a single rigorous evaluation run costs as much as an annual student travel budget—or upwards of $10,000 for a comprehensive test—independent verification of AI claims becomes impossible for anyone without industry-scale funding. The current standard of 'reporting accuracy' based on a single, non-repeated test is dangerously insufficient, yet scaling this to statistically sound reliability testing requires multiplying those costs by eight or more. As we move deeper into the era of agentic AI, we risk a future where only well-funded corporations can afford to prove what their systems are actually capable of doing.

For the last few years, the AI community has treated benchmarks like simple exams: a static test of intelligence that could be graded cheaply. However, as models evolve into agents—systems capable of using tools, navigating the web, and performing long-term planning—the cost of grading them has skyrocketed. We have reached a point where verifying an AI's performance is no longer a trivial overhead but a significant compute-heavy operation, effectively becoming a new bottleneck for the field.

Consider the Holistic Agent Leaderboard (HAL). It recently spent $40,000 just to test a handful of models on agent-based benchmarks. Unlike older methods where you could simply feed a few thousand questions into an API and count the correct answers, evaluating an agent requires watching it perform a multi-step task, often involving error-prone 'scaffolding'—the software wrappers that help the AI interact with the outside world. This process is sensitive and expensive, with costs sometimes varying by orders of magnitude for the same task.

The fundamental problem is that static benchmarks, which test a model’s raw prediction capabilities, allowed for clever shortcuts. Researchers found they could compress these tests by 100x or 200x without losing accurate rankings of which model is best. But agent evaluations are different; because they involve agents performing dynamic, multi-turn interactions, researchers cannot simply subsample the data. Furthermore, some modern benchmarks go a step further, requiring 'training-in-the-loop' where the model must actively learn or optimize during the evaluation process itself. This makes the evaluation process behave less like an exam and more like an experiment that requires massive GPU resources to complete.

This shift creates a troubling 'accountability barrier' for independent researchers and academic groups. If a single rigorous evaluation run costs as much as an annual student travel budget—or upwards of $10,000 for a comprehensive test—independent verification of AI claims becomes impossible for anyone without industry-scale funding. The current standard of 'reporting accuracy' based on a single, non-repeated test is dangerously insufficient, yet scaling this to statistically sound reliability testing requires multiplying those costs by eight or more. As we move deeper into the era of agentic AI, we risk a future where only well-funded corporations can afford to prove what their systems are actually capable of doing.