ToolSpace Builds Harder Agent Benchmarks

Researchers from Renmin University of China introduced ToolSpace in a paper published on Nov 3, proposing a tool-centric framework for building realistic environments and benchmarks for long-horizon agents. The system targets agents that combine tools across MCP services into executable workflows.

Existing methods first generate a database and scenario, then derive tools and tasks. The authors say this can produce tool interfaces that do not match real services, tool combinations unsupported by actual execution, and tasks that rarely cross environments. ToolSpace reverses that process: it mines data, state, and retry dependencies from adjacent calls in real MCP service traces and open agent corpora, using deterministic rules rather than language-model judgment. It groups the resulting weighted tool-composition graph into environments.

ToolSpace creates databases only for environments where tools change state; query-only and stateless tools replay recorded outputs. It then explores tool chains through state-aware random walks with rollback, converts each chain into a natural-language task with assertion-based checks, and keeps only tasks that replay successfully from the initial state.

Applied to AutomationBench and τ³-Bench Banking, the pipeline produced Pro versions where seven frontier models scored up to 38.2 points lower. Across seven long-horizon tool-use benchmarks, Qwen3.6-35B-A3B rose from a 23.0 average score to 41.0, or 43.2 with reinforcement learning (training from feedback); GLM-5.2 rose from 44.6 to 56.6, matching Claude Opus 5. Ablations found that execution-witnessed tool links outperformed links judged by language models, and scores continued to improve as data and environment scale increased.

Researchers from Renmin University of China introduced ToolSpace in a paper published on Nov 3, proposing a tool-centric framework for building realistic environments and benchmarks for long-horizon agents. The system targets agents that combine tools across MCP services into executable workflows.

Existing methods first generate a database and scenario, then derive tools and tasks. The authors say this can produce tool interfaces that do not match real services, tool combinations unsupported by actual execution, and tasks that rarely cross environments. ToolSpace reverses that process: it mines data, state, and retry dependencies from adjacent calls in real MCP service traces and open agent corpora, using deterministic rules rather than language-model judgment. It groups the resulting weighted tool-composition graph into environments.

ToolSpace creates databases only for environments where tools change state; query-only and stateless tools replay recorded outputs. It then explores tool chains through state-aware random walks with rollback, converts each chain into a natural-language task with assertion-based checks, and keeps only tasks that replay successfully from the initial state.

Applied to AutomationBench and τ³-Bench Banking, the pipeline produced Pro versions where seven frontier models scored up to 38.2 points lower. Across seven long-horizon tool-use benchmarks, Qwen3.6-35B-A3B rose from a 23.0 average score to 41.0, or 43.2 with reinforcement learning (training from feedback); GLM-5.2 rose from 44.6 to 56.6, matching Claude Opus 5. Ablations found that execution-witnessed tool links outperformed links judged by language models, and scores continued to improve as data and environment scale increased.