Can Benchmarks Alone Tell Us How Capable an AI Agent Is?
AI is beginning to read screens, operate applications, and carry out work. The question we ask about its performance is changing too. Beyond what it knows, we need to establish how much of a task it can actually finish.
Yesterday, I had a video meeting with OSWorld co-author Tianbao Xie. We discussed the difference between a model's score and a product's ability to do work, and why measuring performance becomes harder as agents advance.
OSWorld puts real computer work to the test
OSWorld is an evaluation environment that asks AI to complete tasks using real operating systems and applications. The main evaluation in the original 2024 paper comprised 369 tasks in an Ubuntu environment. Agents change settings, edit files, and move between materials in applications such as Chrome, GIMP, and LibreOffice.
Each test begins with initial files and settings. After the agent finishes, evaluation code checks the actual files and settings. It scores the outcome, not the AI's claim that the work is done.
Observation conditions include screenshots and accessibility trees containing the structure and text of on-screen elements. Results change with the information provided and the number of actions allowed. Original OSWorld paper
Long workflows expose the difficulty of finishing
OSWorld 2.0 expanded the evaluation to 108 long-horizon workflows. The median time for a human to complete a task is about 1.6 hours.
An expense-reimbursement example involves cross-checking a policy PDF, emailed receipts, bank records, and previous reports. If a new email changes the requirements, the agent must revise its plan. If supporting documents conflict, it must ask the user. The task requires managing both the consistency of information and the state of the work.
The evaluation uses an average of 27.25 checkpoints per task. It distinguishes full completion, where all requirements are met, from a partial score for the requirements satisfied in the final state. Under the 2.0 paper's June 24, 2026 release conditions, with screenshot observations, 500 steps, and batched actions, Claude Opus 4.8 max achieved a completion rate of 20.6% and a partial score of 54.8%. OSWorld 2.0 paper
Keeping benchmarks in step with AI progress
Tests that use real applications are affected by environmental changes. Websites change, applications start slowly, and initial file states drift. An evaluator may even reject a spreadsheet formula that is functionally equivalent to the expected one.
In 2025, OSWorld-Verified reviewed more than 300 pieces of feedback and fixed problems like these. Changes in scores can reflect changes to the test environment and scoring, not just the model. OSWorld-Verified
Tianbao's assessment was that, as progress at leading labs accelerates, V2 is losing its ability to distinguish performance faster than V1 did. He was also considering whether simply adding harder tasks could preserve the evaluation's value over time.
Saturation here means a weakening ability to distinguish a particular leading group, not that every model has solved every real-world task. An evaluation that finds the next research frontier and one that predicts performance on users' work require different tasks.
How the harness changes the same model's performance
I asked whether we should distinguish a model's own performance from the capabilities of an actual agent product. Tianbao explained that we need to include the “harness.”
A harness is the execution code, tool connections, and observation-and-action system that let a model work. It determines what the model sees, which tools it receives, and how its chosen actions are executed. We also discussed how Codex's execution code is part of a system refined through repeated improvements.
To compare models, we should align the surrounding conditions as closely as possible. To compare products, however, we need to include the tools and execution methods each product actually offers. Removing integrations to standardize conditions can erase the strengths of the product the user is buying.
An evaluation therefore needs to disclose the model-and-harness combination and the execution conditions together.
What an agent sees and how it acts
An agent given screenshots must find buttons in an image. An agent given an accessibility tree can use structured elements and text.
Tianbao explained that sufficient accessibility information can let even a model with limited visual capabilities perform tasks. Yet missing information and latency also made purely vision-based approaches important in his long-term view.
The way an agent acts changes performance too. Tianbao used the example of a form with 100 input fields. Calling the model for each field and using code to batch several inputs require different numbers of round trips and introduce different delays. If the screen changes midway through, the agent must observe it again, so the scope of each batch also needs to be designed.
Even “500 steps” can allow different amounts of work per step. Products that use APIs or the command line operate under different conditions from those that click through screens. The basis for comparison depends on whether we are testing screen-operation skills or the ability to finish work using permitted means.
Local service connections can decide whether work gets done
In the meeting, we also discussed how separate authentication apps, approval procedures, and restrictions on external connections can make purchases or sign-ups difficult to complete in Korea and Japan. Tianbao saw support for local services as requiring not only models, but also integrations, partnerships, and operational preparation.
When work stops, we need to distinguish the cause. Failing to understand a request, failing to operate a screen, lacking an integration, and waiting for user approval are different problems.
Removing authentication and connection constraints from a test can make real-world work look too easy. We should also evaluate whether an agent asks for necessary confirmation and resumes work afterward. Regional evaluations need to reflect actual services and workflows, not just translated instructions.
The challenge of evaluating AI that learns about its user
Continual learning came up later in the conversation. Tianbao expected that, as an agent develops a deeper understanding of a user's circumstances and preferences, the cost of transferring that context to another service will grow.
Applying this to evaluation raises a new problem. Resetting the agent every time misses the value of accumulated context. Keeping each agent's existing history makes like-for-like comparisons difficult. This raises the question of whether first-use and long-term-use conditions should be evaluated separately.
Storing and retrieving memories is also different from continual learning that updates a model's weights. If performance improves with use, we need to explain what changed.
Beyond scores: measuring the ability to do real work
An agent's performance report should specify the model and harness, task version, observation inputs, permitted tools, and execution budget. Results should distinguish partial progress from full completion, while also examining time, cost, user intervention, and safe execution.
I believe AIB needs to preserve these distinctions when making technical evaluations accessible to general users. What did the agent see, which tools did it use, in what environment, with whose help, and how much did it finish? Those should be the criteria for explaining an AI agent's capabilities.