Researchers from the University of Illinois at Urbana-Champaign introduced JevAgentHarness, a system that delegates some tool choices by AI agents to smaller decision models. Their paper, published on October 15, presents a benchmark and a 4B model trained on the states agents reach after making mistakes. The approach aims to reduce the cost and latency of long-running tasks while preserving success.

The researchers created AgentDecisionBench to test decision models on isolated questions and while controlling an agent, using public benchmarks and their DecisionGym environment. A model that was most accurate on isolated questions completed only 69.5% of agent tasks. Delegating tool choices without a fitted confidence threshold reduced task success by 19.0 points. JevAgentHarness fits a threshold for each choice using the agent’s own trajectories and leaves decisions below that threshold to the large language model.

The team trained the open AgentDecision-4B model on states reached after agent errors, using Jev’s answers as labels. On tasks unlike its training data, agents using AgentDecision-4B solved 91.5%, compared with 82.0% using Jev. On tasks like the training data, 4B models trained this way made agents 81.1–89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The authors say they will release the models, data and harness.

Researchers from the University of Illinois at Urbana-Champaign introduced JevAgentHarness, a system that delegates some tool choices by AI agents to smaller decision models. Their paper, published on October 15, presents a benchmark and a 4B model trained on the states agents reach after making mistakes. The approach aims to reduce the cost and latency of long-running tasks while preserving success.

The researchers created AgentDecisionBench to test decision models on isolated questions and while controlling an agent, using public benchmarks and their DecisionGym environment. A model that was most accurate on isolated questions completed only 69.5% of agent tasks. Delegating tool choices without a fitted confidence threshold reduced task success by 19.0 points. JevAgentHarness fits a threshold for each choice using the agent’s own trajectories and leaves decisions below that threshold to the large language model.

The team trained the open AgentDecision-4B model on states reached after agent errors, using Jev’s answers as labels. On tasks unlike its training data, agents using AgentDecision-4B solved 91.5%, compared with 82.0% using Jev. On tasks like the training data, 4B models trained this way made agents 81.1–89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The authors say they will release the models, data and harness.