Paper Proposes Multiscale Agent Training Method

A paper titled “Multiscale Policy Optimization for Long-Horizon LLM Agents” was published on October 26 and submitted to Hugging Face on October 28. Its authors propose Multiscale Policy Optimization (MPO), a reinforcement-learning framework for training language-model agents that act over extended sequences of interactions. The paper reports experiments on search, tool-use and automation tasks using 4B and 30B-A3B models, with consistent gains over strong reinforcement-learning baselines.

The authors say agents generate actions token by token but interact with their environments through complete assistant turns. They observed that critic values (estimates of how promising a state is) vary mainly between turns, motivating policy optimization at the turn level while assigning credit more finely among tokens. MPO combines Multiscale Advantage Estimation (MAE) with turn-level importance sampling to align credit assignment with policy updates at the turn scale. The paper’s summary does not provide numerical results or identify the baselines.

A commenter raised questions about whether turn-level importance ratios remain reliable when turns differ greatly in length, giving examples of a 40-token function call and a 6,000-token file read. The commenter also suggested that a critic value that stays nearly constant within a turn might serve as a per-turn anomaly score during inference, and asked whether that value stays calibrated as the policy changes. These are commenter questions and suggestions; the paper summary does not report answers or evaluate that proposed use.

A paper titled “Multiscale Policy Optimization for Long-Horizon LLM Agents” was published on October 26 and submitted to Hugging Face on October 28. Its authors propose Multiscale Policy Optimization (MPO), a reinforcement-learning framework for training language-model agents that act over extended sequences of interactions. The paper reports experiments on search, tool-use and automation tasks using 4B and 30B-A3B models, with consistent gains over strong reinforcement-learning baselines.

The authors say agents generate actions token by token but interact with their environments through complete assistant turns. They observed that critic values (estimates of how promising a state is) vary mainly between turns, motivating policy optimization at the turn level while assigning credit more finely among tokens. MPO combines Multiscale Advantage Estimation (MAE) with turn-level importance sampling to align credit assignment with policy updates at the turn scale. The paper’s summary does not provide numerical results or identify the baselines.

A commenter raised questions about whether turn-level importance ratios remain reliable when turns differ greatly in length, giving examples of a 40-token function call and a 6,000-token file read. The commenter also suggested that a critic value that stays nearly constant within a turn might serve as a per-turn anomaly score during inference, and asked whether that value stays calibrated as the policy changes. These are commenter questions and suggestions; the paper summary does not report answers or evaluate that proposed use.