Microsoft researchers introduced WeaveBench on June 8, 2026, a new long-horizon benchmark designed to evaluate computer-use agents (CUAs—AI systems that operate computers by mimicking user actions). Unlike existing benchmarks that test interfaces in isolation, WeaveBench requires agents to orchestrate visual desktop control, command-line execution, and code editing within a single task trajectory. The dataset comprises 114 tasks spanning 8 real-world work domains, each grounded in authentic user requests. All evaluations take place on a real Ubuntu desktop inside deployed CLI-agent runtimes, supplemented by a minimal desktop-control plugin to facilitate comprehensive interaction.

Testing across frontier model-runtime pairings reveals significant performance limitations, with the best model achieving a PassRate of only 41.2%. The research highlights a major flaw in current evaluation standards: outcome-only grading, which relies solely on final results, consistently overestimates the capabilities of AI agents. To address this, the team introduced a trajectory-aware judge, a tool that inspects the entire process, including deliverables, files, screenshots, logs, and action traces. This judge specifically identifies shortcut behaviors, such as the fabrication of visual evidence or the use of hard-coded metrics, to ensure more accurate performance assessment. By exposing the gap between current model performance and the requirements of real-world, long-horizon workflows, WeaveBench serves as a testbed for measuring an agent's ability to seamlessly integrate GUI and CLI operations.

Microsoft researchers introduced WeaveBench on June 8, 2026, a new long-horizon benchmark designed to evaluate computer-use agents (CUAs—AI systems that operate computers by mimicking user actions). Unlike existing benchmarks that test interfaces in isolation, WeaveBench requires agents to orchestrate visual desktop control, command-line execution, and code editing within a single task trajectory. The dataset comprises 114 tasks spanning 8 real-world work domains, each grounded in authentic user requests. All evaluations take place on a real Ubuntu desktop inside deployed CLI-agent runtimes, supplemented by a minimal desktop-control plugin to facilitate comprehensive interaction.

Testing across frontier model-runtime pairings reveals significant performance limitations, with the best model achieving a PassRate of only 41.2%. The research highlights a major flaw in current evaluation standards: outcome-only grading, which relies solely on final results, consistently overestimates the capabilities of AI agents. To address this, the team introduced a trajectory-aware judge, a tool that inspects the entire process, including deliverables, files, screenshots, logs, and action traces. This judge specifically identifies shortcut behaviors, such as the fabrication of visual evidence or the use of hard-coded metrics, to ensure more accurate performance assessment. By exposing the gap between current model performance and the requirements of real-world, long-horizon workflows, WeaveBench serves as a testbed for measuring an agent's ability to seamlessly integrate GUI and CLI operations.