FewStepRL Adapts Reinforcement Learning for Diffusion

Researchers at Tencent Hunyuan published FewStepRL, a method for adapting reinforcement learning objectives to few-step diffusion models, in a paper published on November 6. The work addresses a training-cost problem: existing objectives often run the current model at every training iteration, and these rollouts can dominate costs when using many-step samplers. Few-step models reduce that cost, but predict an average velocity over an interval, while existing objectives are designed for instantaneous velocity in many-step sampling.

The researchers adapt objectives in three phases. First, an x-space transformation expresses objectives through a shared policy prediction, usually at the clean output and at a marked level for transition or flow-map objectives. This makes objectives defined over densities, velocities, or rewards comparable. Second, gradient decomposition separates each objective’s gradient into self-regression toward a reference and reward augmentation, making the method’s anchors and weights the key choices. Third, anchor adaptation calculates anchors from the few-step model at the end of its transport or along its tangent; the model or an auxiliary network supplies instantaneous velocity.

The approach produces few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD, and ReFL in one modular codebase. In experiments, reward-gradient methods performed best when rewards were differentiable and their gradients affordable. In other cases, DiffusionNFT-style methods generally reached high rewards in relatively few iterations, though each iteration could cost more. Adding inference steps brought no systematic aggregate benefit, but some outputs visibly improved when sampling used two to four times as many steps as training.

Researchers at Tencent Hunyuan published FewStepRL, a method for adapting reinforcement learning objectives to few-step diffusion models, in a paper published on November 6. The work addresses a training-cost problem: existing objectives often run the current model at every training iteration, and these rollouts can dominate costs when using many-step samplers. Few-step models reduce that cost, but predict an average velocity over an interval, while existing objectives are designed for instantaneous velocity in many-step sampling.

The researchers adapt objectives in three phases. First, an x-space transformation expresses objectives through a shared policy prediction, usually at the clean output and at a marked level for transition or flow-map objectives. This makes objectives defined over densities, velocities, or rewards comparable. Second, gradient decomposition separates each objective’s gradient into self-regression toward a reference and reward augmentation, making the method’s anchors and weights the key choices. Third, anchor adaptation calculates anchors from the few-step model at the end of its transport or along its tangent; the model or an auxiliary network supplies instantaneous velocity.

The approach produces few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD, and ReFL in one modular codebase. In experiments, reward-gradient methods performed best when rewards were differentiable and their gradients affordable. In other cases, DiffusionNFT-style methods generally reached high rewards in relatively few iterations, though each iteration could cost more. Adding inference steps brought no systematic aggregate benefit, but some outputs visibly improved when sampling used two to four times as many steps as training.