Researchers introduced CriticOPD, a method for training language models, to improve mathematical reasoning by using a teacher model’s critique to correct errors in a student model’s solutions. The paper, “Recovering Reliable Supervision via Critique-Guided On-Policy Distillation,” was published on Hugging Face on Oct 14 and submitted on Oct 15 by Kaixiang Zhao and coauthors.

On-policy distillation trains a student on its own generated solutions, with a teacher supplying guidance for each token. The method assumes the teacher can produce a correct answer from any partial solution, but the researchers found that assumption fails when the student’s solution contains errors: the teacher may recognize an error and still continue along the flawed path. They call the teacher’s ability to recover a correct answer from a student’s partial solution “recoverability.”

With CriticOPD, the teacher critiques an incorrect solution, and the student rewrites the remaining steps using that feedback. The teacher scores the rewritten continuation while the critique is visible; the student then trains on the revised solution without seeing the critique. The paper says the critique acts as privileged information during scoring.

Tests on two Qwen3 student models and six mathematical reasoning benchmarks found that CriticOPD improved average accuracy over standard on-policy distillation by 4.57 percentage points for Qwen3-1.7B and 4.06 percentage points for Qwen3-0.6B. The researchers reported the largest gains on competition problems.

Researchers introduced CriticOPD, a method for training language models, to improve mathematical reasoning by using a teacher model’s critique to correct errors in a student model’s solutions. The paper, “Recovering Reliable Supervision via Critique-Guided On-Policy Distillation,” was published on Hugging Face on Oct 14 and submitted on Oct 15 by Kaixiang Zhao and coauthors.

On-policy distillation trains a student on its own generated solutions, with a teacher supplying guidance for each token. The method assumes the teacher can produce a correct answer from any partial solution, but the researchers found that assumption fails when the student’s solution contains errors: the teacher may recognize an error and still continue along the flawed path. They call the teacher’s ability to recover a correct answer from a student’s partial solution “recoverability.”

With CriticOPD, the teacher critiques an incorrect solution, and the student rewrites the remaining steps using that feedback. The teacher scores the rewritten continuation while the critique is visible; the student then trains on the revised solution without seeing the critique. The paper says the critique acts as privileged information during scoring.

Tests on two Qwen3 student models and six mathematical reasoning benchmarks found that CriticOPD improved average accuracy over standard on-policy distillation by 4.57 percentage points for Qwen3-1.7B and 4.06 percentage points for Qwen3-0.6B. The researchers reported the largest gains on competition problems.