Researchers at the University of Virginia, Microsoft Research and Tsinghua University developed forecasting software that helps AI agents assess proposed changes to their own code and instructions before testing them. The approach, described in a paper posted to arXiv, was evaluated on programming, paper-acceptance prediction, simulated-robot reward design and mathematical grading tasks. Initial results suggest forecasts can help agents improve more efficiently.

The method extends HyperAgents, which pairs a task-performing agent with an agent that modifies the code and instructions used by both; earlier versions and results are kept in an archive. The researchers added an Improvement Dynamics Model (IDM), which examines a proposed edit, the agent being changed and past results. It predicts task-level outcomes and distinguishes wrong answers from attempts that do not finish, since those failures may call for different fixes. When a bias-adjusted forecast indicates worse performance than the current agent’s measured performance, the modifying component gets the forecast and one extra chance to revise. The resulting agent is tested regardless of the final forecast. Its software and instructions can change, but the underlying language-model weights stay fixed.

In one example, an edit put instructions outside quotation marks and broke the program. The modifying agent spotted the problem but ran out of tool calls before fixing it. The forecaster predicted unfinished tasks, leading to another revision; after correcting the quotation marks and checking syntax, the agent completed 11 tasks.

For each experimental condition, the team ran five trials of 40 improvement rounds. On 166 programming tasks excluded from improvement and selection, selected agents had a median success rate of about 30%, versus 22% for HyperAgents and 16% for the shared starting agent. Success meant producing a software patch that passed task-specific tests. With matching language-model token budgets, the forecasting approach retained an advantage; the accounting included tokens for modification, forecasting, forecast updates and task evaluation, but not total computing cost.

The forecasting software itself proposes updates every five rounds and checks predictions against later outcomes before adopting them. To test transfer, the researchers used a forecaster developed jointly on paper-acceptance prediction and simulated-robot reward design for mathematical grading, starting with fresh task-solving and modifying programs. Across 100 unseen grading tasks, selected agents matched expert grades with median accuracy of 55%, compared with 51% for agents using a fresh forecaster and 47% for HyperAgents. The findings do not establish that agents can replace human reviewers, graders or programmers; they suggest forecasting may help identify useful software changes and transfer experience to another task.

Researchers at the University of Virginia, Microsoft Research and Tsinghua University developed forecasting software that helps AI agents assess proposed changes to their own code and instructions before testing them. The approach, described in a paper posted to arXiv, was evaluated on programming, paper-acceptance prediction, simulated-robot reward design and mathematical grading tasks. Initial results suggest forecasts can help agents improve more efficiently.

The method extends HyperAgents, which pairs a task-performing agent with an agent that modifies the code and instructions used by both; earlier versions and results are kept in an archive. The researchers added an Improvement Dynamics Model (IDM), which examines a proposed edit, the agent being changed and past results. It predicts task-level outcomes and distinguishes wrong answers from attempts that do not finish, since those failures may call for different fixes. When a bias-adjusted forecast indicates worse performance than the current agent’s measured performance, the modifying component gets the forecast and one extra chance to revise. The resulting agent is tested regardless of the final forecast. Its software and instructions can change, but the underlying language-model weights stay fixed.

In one example, an edit put instructions outside quotation marks and broke the program. The modifying agent spotted the problem but ran out of tool calls before fixing it. The forecaster predicted unfinished tasks, leading to another revision; after correcting the quotation marks and checking syntax, the agent completed 11 tasks.

For each experimental condition, the team ran five trials of 40 improvement rounds. On 166 programming tasks excluded from improvement and selection, selected agents had a median success rate of about 30%, versus 22% for HyperAgents and 16% for the shared starting agent. Success meant producing a software patch that passed task-specific tests. With matching language-model token budgets, the forecasting approach retained an advantage; the accounting included tokens for modification, forecasting, forecast updates and task evaluation, but not total computing cost.

The forecasting software itself proposes updates every five rounds and checks predictions against later outcomes before adopting them. To test transfer, the researchers used a forecaster developed jointly on paper-acceptance prediction and simulated-robot reward design for mathematical grading, starting with fresh task-solving and modifying programs. Across 100 unseen grading tasks, selected agents matched expert grades with median accuracy of 55%, compared with 51% for agents using a fresh forecaster and 47% for HyperAgents. The findings do not establish that agents can replace human reviewers, graders or programmers; they suggest forecasting may help identify useful software changes and transfer experience to another task.