A paper published on Hugging Face proposes ReMaRk, a reinforcement learning method for large language model recommenders that scores generated recommendation lists against all valid replies jointly. The authors argue that training should treat the original request, rather than each answer, as the unit: for example, a Reddit request may receive several replies that name different titles. ReMaRk compares a generated list with the union of titles in those replies and samples each request equally.

The paper says the existing Rank-GRPO approach selects one reply per training example, so which reply was selected accounts for 85% of the variation in its advantage score, rather than the quality of the generated list. ReMaRk reduces training cost while matching or exceeding earlier performance. Across seven backbones and three datasets, it reaches the validation recall of 2,000 per-reply reinforcement-learning steps using 25 to 53% as many steps, and leads at that budget.

When both methods are trained until neither improves, ReMaRk is never significantly behind Rank-GRPO; in four of seven comparisons, it reaches Rank-GRPO's best result with fewer GPU-hours. The same reward method also helps on ProtoQA, where about 100 people answer each question. The authors say the findings point to jointly rewarding multiple valid human answers when reinforcement learning is trained on tasks with several possible responses.

A paper published on Hugging Face proposes ReMaRk, a reinforcement learning method for large language model recommenders that scores generated recommendation lists against all valid replies jointly. The authors argue that training should treat the original request, rather than each answer, as the unit: for example, a Reddit request may receive several replies that name different titles. ReMaRk compares a generated list with the union of titles in those replies and samples each request equally.

The paper says the existing Rank-GRPO approach selects one reply per training example, so which reply was selected accounts for 85% of the variation in its advantage score, rather than the quality of the generated list. ReMaRk reduces training cost while matching or exceeding earlier performance. Across seven backbones and three datasets, it reaches the validation recall of 2,000 per-reply reinforcement-learning steps using 25 to 53% as many steps, and leads at that budget.

When both methods are trained until neither improves, ReMaRk is never significantly behind Rank-GRPO; in four of seven comparisons, it reaches Rank-GRPO's best result with fewer GPU-hours. The same reward method also helps on ProtoQA, where about 100 people answer each question. The authors say the findings point to jointly rewarding multiple valid human answers when reinforcement learning is trained on tasks with several possible responses.