Coding Agent Harnesses Show Up to 3x Cost Gap

Researchers from Shandong University in China and the University of California, Berkeley reported in a study posted to an online archive that three coding agent harnesses differed in cost per task by up to 3x, while their code-writing success rates remained within statistical error. A harness is an execution environment that helps an AI model edit code and call tools. The comparison covered Anthropic’s Claude Code, the open-source tool OpenCode, and the minimal Mini-SWE-Agent.

The researchers tested five models: the open-weight Qwen3.6-35B-A3B and Qwen3.8-27B, plus the API-based DeepSeek-V4-Flash, GLM-5.3-Flash, and HY4-Preview. Claude Opus 5 running through Claude Code served as a reference model. They evaluated performance on 492 offline tasks from SWE-bench Verified, which measures code-editing ability, separating 45 difficult tasks that take people at least 1 to 4 hours to solve from the other 447.

On the harder tasks, switching harnesses changed the outcome of about 13% of problems between success and failure. Similar variation appeared when the same harness was run repeatedly. The researchers said much of the apparent performance gap between harnesses may reflect random variation between runs. Harnesses built by model makers for their own models also did not consistently produce higher success rates. DeepSeek’s official command-line harness had a success rate within the range of the other harnesses, but the highest cost per task among those compared.

OpenCode scored up to 9 percentage points lower than other environments on larger tasks. About half of the gap came from runs that hit the 32,000-token output limit and stopped before writing a code patch. Claude Code had a recovery feature that automatically asked the model to continue generating output when it reached the same limit.

Cost differences began with the first API call. Fixed context in system prompts and tool definitions exceeded 10,000 tokens in Claude Code, compared with just over 800 tokens in Mini-SWE-Agent. The gap increased costs as tasks required more steps. One exception appeared with DeepSeek models: Mini-SWE-Agent made more calls, reversing the cost ranking. The researchers said cost efficiency depended on the combination of model and workflow. They recommended repeated testing on enough tasks instead of judging a complex harness from a single run. They also proposed simplifying prompts and tool definitions, repeatedly measuring cost and performance on the model intended for use, and keeping only features shown to improve results.

Researchers from Shandong University in China and the University of California, Berkeley reported in a study posted to an online archive that three coding agent harnesses differed in cost per task by up to 3x, while their code-writing success rates remained within statistical error. A harness is an execution environment that helps an AI model edit code and call tools. The comparison covered Anthropic’s Claude Code, the open-source tool OpenCode, and the minimal Mini-SWE-Agent.

The researchers tested five models: the open-weight Qwen3.6-35B-A3B and Qwen3.8-27B, plus the API-based DeepSeek-V4-Flash, GLM-5.3-Flash, and HY4-Preview. Claude Opus 5 running through Claude Code served as a reference model. They evaluated performance on 492 offline tasks from SWE-bench Verified, which measures code-editing ability, separating 45 difficult tasks that take people at least 1 to 4 hours to solve from the other 447.

On the harder tasks, switching harnesses changed the outcome of about 13% of problems between success and failure. Similar variation appeared when the same harness was run repeatedly. The researchers said much of the apparent performance gap between harnesses may reflect random variation between runs. Harnesses built by model makers for their own models also did not consistently produce higher success rates. DeepSeek’s official command-line harness had a success rate within the range of the other harnesses, but the highest cost per task among those compared.

OpenCode scored up to 9 percentage points lower than other environments on larger tasks. About half of the gap came from runs that hit the 32,000-token output limit and stopped before writing a code patch. Claude Code had a recovery feature that automatically asked the model to continue generating output when it reached the same limit.

Cost differences began with the first API call. Fixed context in system prompts and tool definitions exceeded 10,000 tokens in Claude Code, compared with just over 800 tokens in Mini-SWE-Agent. The gap increased costs as tasks required more steps. One exception appeared with DeepSeek models: Mini-SWE-Agent made more calls, reversing the cost ranking. The researchers said cost efficiency depended on the combination of model and workflow. They recommended repeated testing on enough tasks instead of judging a complex harness from a single run. They also proposed simplifying prompts and tool definitions, repeatedly measuring cost and performance on the model intended for use, and keeping only features shown to improve results.