Δτ-Bench Tests Agent Adaptation

Researchers at University College London presented Δτ-Bench, an executable benchmark for testing whether AI agents adapt when the tools available to them change. The paper was published on Oct 25 and submitted by Shuofei Qiao on Oct 27. It models tool ecosystems in which services evolve independently: familiar tools can disappear or be replaced, more efficient alternatives can emerge, and documentation can be noisy. Agents are tested on whether they find effective ways to use tools and update procedural knowledge as the available capabilities shift.

Δτ-Bench contains 1,588 validated multi-turn tasks across 42 domains and six successive ecosystem states. The states are generated through capability-preserving transformations. The strongest evaluated model or harness achieved 42.3% task success. The authors report that gains from existing skill-learning methods did not consistently translate into better adaptation as the ecosystem evolved.

The researchers also propose Steward, an ecosystem manager that maintains tools and procedural knowledge through incremental skill maintenance and dynamic provisioning during task execution. Their experiments found that Steward sustained higher task success through ecosystem changes with lower skill-maintenance overhead, and improved recovery from disrupted workflows and adoption of more efficient alternatives.

Researchers at University College London presented Δτ-Bench, an executable benchmark for testing whether AI agents adapt when the tools available to them change. The paper was published on Oct 25 and submitted by Shuofei Qiao on Oct 27. It models tool ecosystems in which services evolve independently: familiar tools can disappear or be replaced, more efficient alternatives can emerge, and documentation can be noisy. Agents are tested on whether they find effective ways to use tools and update procedural knowledge as the available capabilities shift.

Δτ-Bench contains 1,588 validated multi-turn tasks across 42 domains and six successive ecosystem states. The states are generated through capability-preserving transformations. The strongest evaluated model or harness achieved 42.3% task success. The authors report that gains from existing skill-learning methods did not consistently translate into better adaptation as the ecosystem evolved.

The researchers also propose Steward, an ecosystem manager that maintains tools and procedural knowledge through incremental skill maintenance and dynamic provisioning during task execution. Their experiments found that Steward sustained higher task success through ecosystem changes with lower skill-maintenance overhead, and improved recovery from disrupted workflows and adoption of more efficient alternatives.