Researchers introduced TreeOPSD, a critic-free framework for training language-model agents on long-horizon tasks with reinforcement learning and verifiable rewards. The paper was published on arXiv on October 16 and submitted to Hugging Face Papers on October 19. It reports tests across Multi-Hop QA and Deep Research benchmarks, using models from 1.7B to 30B-A3B parameters.

The authors identify two training problems: terminal rewards arrive only at the end of an interaction, making it difficult to assign credit to individual turns, and sequential mini-batch updates can make the current policy drift from the behavior policy that generated the training data. Existing methods use learned critics, process annotations, or stronger external teachers to provide denser feedback; these add components or require annotations. Importance-ratio clipping can limit policy changes, but the paper says it does not define a finite target for sequential updates.

TreeOPSD treats each interaction turn as a node in a tree, branches alternative continuations from shared prefixes, and recursively propagates terminal rewards back to estimate turn-level credit. It converts that credit into a bounded teacher distribution, then trains the student policy by minimizing Forward KL divergence from the teacher. The authors say this recovers the tree policy-gradient direction at the behavior policy while keeping updates bounded toward a finite target across sequential mini-batches.

Across matched rollout budgets, TreeOPSD outperformed GRPO, GSPO, and fixed-credit tree policy gradients on Multi-Hop QA and Deep Research benchmarks. On Deep Research, it raised the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO. The paper says gains were especially notable on long-horizon tasks.

Researchers introduced TreeOPSD, a critic-free framework for training language-model agents on long-horizon tasks with reinforcement learning and verifiable rewards. The paper was published on arXiv on October 16 and submitted to Hugging Face Papers on October 19. It reports tests across Multi-Hop QA and Deep Research benchmarks, using models from 1.7B to 30B-A3B parameters.

The authors identify two training problems: terminal rewards arrive only at the end of an interaction, making it difficult to assign credit to individual turns, and sequential mini-batch updates can make the current policy drift from the behavior policy that generated the training data. Existing methods use learned critics, process annotations, or stronger external teachers to provide denser feedback; these add components or require annotations. Importance-ratio clipping can limit policy changes, but the paper says it does not define a finite target for sequential updates.

TreeOPSD treats each interaction turn as a node in a tree, branches alternative continuations from shared prefixes, and recursively propagates terminal rewards back to estimate turn-level credit. It converts that credit into a bounded teacher distribution, then trains the student policy by minimizing Forward KL divergence from the teacher. The authors say this recovers the tree policy-gradient direction at the behavior policy while keeping updates bounded toward a finite target across sequential mini-batches.

Across matched rollout budgets, TreeOPSD outperformed GRPO, GSPO, and fixed-credit tree policy gradients on Multi-Hop QA and Deep Research benchmarks. On Deep Research, it raised the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO. The paper says gains were especially notable on long-horizon tasks.