Study Measures Distillation’s Mismatch Tolerance

Researchers led by Cong Yu published an arXiv paper on November 8 examining how much training-inference mismatch on-policy distillation (OPD) can tolerate. OPD trains a student language model on its own responses while a fixed teacher guides it. Quantized inference and asynchronous rollouts can make the rollout engine differ from the model being trained, creating mismatch that may destabilize reinforcement learning (RL). The researchers measured quantization across three student-teacher pairs and tested weight staleness and injected noise on one pair.

The paper reports that OPD can maintain training quality despite substantial mismatch, but performance degrades beyond a model-specific limit the authors call a mismatch budget. In a 7B student, quality held at more than 200× the mismatch of a standard BF16 engine and failed at about twice that level. Two RL methods broke at about 12× and 60× under the same engines. The amount of mismatch that can be tolerated varies by model, and some layers are more sensitive to reduced precision.

The authors also found that weight noise can change which prefixes (generated text sequences that condition later output) the model reaches, causing errors that per-token corrections cannot fix. They propose a lightweight admission check that screens rollout configurations against a calibrated budget in minutes on one GPU, without a full training run for each candidate. The check compares token probabilities from a quantized model with those from a full-precision model; it needs neither a teacher nor training.

Model-specific FP8 recipes preserve training quality while allowing KV caches to hold at least 1.93× as many tokens as BF16. Combining FP8 with asynchronous rollouts reached the same training quality 1.62× sooner than synchronous BF16 training. In the authors’ tests, benchmark accuracy stayed within 1.4 points of normal training below the mismatch limit. They say the limit can tighten during training: one configuration passed at step 100 but failed by step 200. Their discussion notes that the limits differed between 1.5B and 7B students, and the paper’s tests covered math prompts; applying the check to code or tool-use prompts would require rerunning it for that task.

Researchers led by Cong Yu published an arXiv paper on November 8 examining how much training-inference mismatch on-policy distillation (OPD) can tolerate. OPD trains a student language model on its own responses while a fixed teacher guides it. Quantized inference and asynchronous rollouts can make the rollout engine differ from the model being trained, creating mismatch that may destabilize reinforcement learning (RL). The researchers measured quantization across three student-teacher pairs and tested weight staleness and injected noise on one pair.

The paper reports that OPD can maintain training quality despite substantial mismatch, but performance degrades beyond a model-specific limit the authors call a mismatch budget. In a 7B student, quality held at more than 200× the mismatch of a standard BF16 engine and failed at about twice that level. Two RL methods broke at about 12× and 60× under the same engines. The amount of mismatch that can be tolerated varies by model, and some layers are more sensitive to reduced precision.

The authors also found that weight noise can change which prefixes (generated text sequences that condition later output) the model reaches, causing errors that per-token corrections cannot fix. They propose a lightweight admission check that screens rollout configurations against a calibrated budget in minutes on one GPU, without a full training run for each candidate. The check compares token probabilities from a quantized model with those from a full-precision model; it needs neither a teacher nor training.

Model-specific FP8 recipes preserve training quality while allowing KV caches to hold at least 1.93× as many tokens as BF16. Combining FP8 with asynchronous rollouts reached the same training quality 1.62× sooner than synchronous BF16 training. In the authors’ tests, benchmark accuracy stayed within 1.4 points of normal training below the mismatch limit. They say the limit can tighten during training: one configuration passed at step 100 but failed by step 200. Their discussion notes that the limits differed between 1.5B and 7B students, and the paper’s tests covered math prompts; applying the check to code or tool-use prompts would require rerunning it for that task.