2026-10-15 · ArXiv
arXiv:2605.12483v5 Announce Type: replace-cross
Abstract: In settings where labeled verifiable training data is the binding constraint, each checked example should be allocated to the model and reward density where it is most informative. We identify a reward-density principle: sparse sequence-level reward is most useful on models that can explore and discover better behavior, while dense token-level teacher supervision is better suited for compressing that behavior into a smaller deployment model; accordingly, scarce labeled data should be used upstream on the strongest available teacher, then transferred downstream as dense supervision. We evaluate this rule through a four-stage workflow:teacher RL, forward-KL warmup, on-policy distillation, optional post-bridge student RL on verifiable math and code generation with Qwen3, Llama, and Qwen2.5 models. At fixed Qwen3-1.7B deployment-student size, an RL-improved 8B teacher distilled through the dense bridge outperforms direct GRPO on the same student ($79.3\%$ vs.\ $75.9\%$ on MATH; $25.2\%$ vs.\ $19.8\%$ on AIME~2024), while transfer from the same teacher before RL underperforms. A component ablation confirms that each stage is load-bearing: the RL-improved teacher adds $7.8$ absolute points on MATH over a raw teacher of the same size, the forward-KL warmup adds $1.7$, and on-policy distillation adds $3.3$. The teacher-quality ordering: raw-teacher transfer $
原文链接：https://arxiv.org/abs/2605.12483
查看更多资讯请见 AI 行业资讯。