SWE-CoWork Tests Coding Agents at Work
- •SWE-CoWork evaluates coding agents on 101 assignments across real repositories and simulated organizations
- •The two leading models passed approximately 63% of fail-to-pass tests but completed fewer than 8% of assignments
- •Complete requirements raised fail-to-pass results for two models by approximately 16 percentage points without improving task completion
Researchers introduced SWE-CoWork, a benchmark for testing coding agents on software assignments in simulated organizations, in a paper published on November 15. It contains 101 assignments from 101 real repositories, spanning 11 programming languages and 12 application domains. For each assignment, an agent starts with a brief handoff and must gather requirements from 26 simulated colleagues and workplace records, resolve conflicting information, implement changes, and pass private executable tests.
The researchers evaluated 11 models. The two leading models passed approximately 63% of fail-to-pass tests, which check whether a previously failing test now passes, but completed fewer than 8% of assignments. Giving Seed Evolving and GPT-6 Astra complete requirements upfront raised their fail-to-pass results by approximately 16 percentage points, while complete assignment success remained unchanged.
A closer analysis found problems across the delivery process. Claude-Opus-5.5 directed a mean of 93% of its first questions about queried cards to the responsible colleagues across runs, but retrieved only 64% of the requirements held by colleagues. Agents failed to follow up on deferred questions, and some implementations remained incomplete even after agents obtained requirements. In addition, 31% of requirements agents marked complete failed their tests. The authors identify sustained requirement tracking from clarification through implementation, verification, and handoff as a key challenge for reliable autonomous software engineering.
The fastest, most reliable AI news, straight to your inbox
