A new scaling-law paper fits a power law that predicts the peak accuracy of a distilled student to within 0.7 of a point at scales it never trained on. The same paper, in an appendix titled “Lessons learned and what did not work”, records that it tried to fit a law for when that peak arrives and gave up.

Both facts are in Scaling Properties of Same-Family On-Policy Distillation, posted 26 September by a team at Zhejiang University and Kuaishou Technology and the highest-voted paper in this week’s Hugging Face daily listings. It is a careful piece of work, and its headline holds. What repays reading is how much narrower the claim is than “distillation outcomes are predictable”, and that the narrowing is done by the authors rather than by a critic.

On-policy distillation here means a fixed teacher scoring every token of rollouts sampled from the student, rather than the student imitating pre-recorded teacher output. The study crosses five Qwen2.5 base sizes (0.5B to 14B) as teachers and as students, 25 pairs in all, on a mixed GSM8K and MATH split of 14.8K training problems scored against 6.3K held-out ones.

Four quantities, two claimed laws

Across the paper the authors attempt to characterise four things about an on-policy distillation run. It is worth separating them, because they fare very differently and only two are claimed in the abstract.

The peak gold score. Fitted as a joint power law in student size, capped teacher size and the teacher’s own remaining error. This one works. Withholding the largest student or teacher scale and re-fitting, the joint law lands “within one accuracy point”; the authors’ project page puts the extrapolation error at 0.7 points. Against ablated versions of itself the joint form is clearly better: leave-one-scale-out RMSE of 1.66 accuracy points, against 3.43 for scales alone and 2.55 for teacher score alone.

The sharpest check is not the RMSE table but a single held-out comparison. They take a 3B teacher checkpoint mid-RL whose score (66.0) slightly exceeds the 1.5B RL endpoint (63.6), and distil a 7B student from each. The higher-scoring teacher produces the worse student: 73.8 against 77.5. Only the joint law orders those two correctly. That is a genuine out-of-sample prediction of a counterintuitive result, and it is the strongest thing in the paper.

The useful-transfer rate — how fast accuracy climbs early on. Also fitted as a power law, and the authors downgrade it themselves. The project page says to “Read the rate law as an interpretable summary rather than a precise predictor”, citing log-space R² of 0.59 and 0.76 for the two objectives. The paper’s limitations appendix goes further than the project page does, and the sentence is worth quoting exactly: “the transfer-rate power law does not outperform the raw-affine baseline under leave-one-scale-out RMSE”. A fitted law that does not beat an affine model on the same covariates is a description, not a predictor.

The useful-transfer extent — how long the regular phase lasts before dynamics turn noisy. “The transfer extent resists a comparable law.” Observed and censored values span a √KL distance of 0.20 to 0.36 with medians of 0.30 and 0.29, so the authors report it as “an approximately scale-free KL budget rather than a scaling target”. That is a useful constant and an honest non-result.

The location of the peak. This one appears in neither the abstract nor the project page. From Appendix K: “We attempted to model the location… of the gold-score maximum as a function of student and teacher scale. The observed locations are non-monotone across the grid… and no simple parametric family survived identifiability checks.” So the peak value is predictable and the moment it arrives is not, which matters precisely because the thing you have to decide during a run is when to stop.

Counting them: one predictor, one law its authors tell you not to use as one, and two quantities with no law at all. The abstract claims exactly the first two, so the paper does not overclaim. The distance is between what the paper says and what a reader skimming “power laws predict the peak before training” will take away.

What the grid settles anyway

Set the laws aside and the 25-cell grid produces three results that need no curve fitting, all of which are more immediately actionable than the peak law.

In all ten weak-to-strong pairs the student’s peak exceeds its teacher’s own score, so one small RL expert lifts every larger sibling. The authors are careful about what that does and does not buy: “Direct RL on the student itself remains above every weak-teacher variant”, so the case for weak-to-strong distillation is amortising one expensive expert across a family, not beating RL.

An off-policy SFT cold start — the conventional way to align a student to a teacher before distilling — costs 6.6, 15.7 and 19.4 accuracy points for 3B, 7B and 14B students of the 0.5B expert. The mechanism is stated plainly: one epoch of SFT on the weak teacher’s rollouts “pins every student near the teacher’s own score”, erasing 32 points of the 14B student’s initialisation, and the subsequent distillation phase does not recover it.

And bootstrapping up the family — 0.5B to 1.5B to 3B to 7B to 14B, each stage teaching the next — loses to distilling directly from the 0.5B expert at every size (76.8 against 77.7 at 14B). This reverses the bootstrapping gain reported by Burns et al. for weak-to-strong fine-tuning. The dissociation is sharp: the 1.5B distillation product scores 53.0% against the 0.5B RL expert’s 39.8%, and still teaches worse.

Two things the appendices carry that the summaries do not

Every run in the study, 25 Vanilla cells plus 17 Delta cells plus teachers and baselines, uses a single random seed and no configuration is repeated. The authors defend this on compute grounds and by precedent, and note that scaling-law work typically derives its statistical support from regularity across configurations rather than replication within them. Fair, and worth knowing before treating a 0.9-point gap between two cells as a result.

The one I would not have expected: the same recipes showed no post-peak regression on an FSDP backend with BF16 optimizer state, and “regressed reliably” on Megatron with FP32. The authors suspect FP32 tracks a small late-stage gradient signal that BF16 rounds away, fixed the study to one backend, and wrote it down. They also record that their first implementation of the Delta objective used a KL estimator whose value was correct while “its gradient with respect to the policy is the gradient of the forward KL, so training regularized the wrong divergence”.

Neither of those is in the project page, and the second is the kind of thing most groups quietly fix. A paper whose central claim is an empirical regularity, and which then tells you that the regularity’s tail behaviour moved when it changed optimizer precision, has given you the one fact you need to decide whether to believe it. That is the opposite of a problem with this paper.