ReOrder-OPD: Reliability-Aware Prompt Ordering for On-Policy Distillation

High-$R$ prompts are more likely to provide teacher supervision whose continuation reaches a correct answer from student-visited states. Placing them earlier gives the initial OPD updates more reliable teacher targets. At update $k$, both the sampled trajectory and the teacher targets are conditioned on the current student:

where $\mathcal{U}$ denotes one OPD update. Presenting prompt $a$ before $b$ changes the policy that generates the trajectory for $b$; in general, $\mathcal{U}_{b}(\mathcal{U}_{a}(\pi))$ and $\mathcal{U}_{a}(\mathcal{U}_{b}(\pi))$ need not coincide. ReOrder-OPD uses this path dependence to visit higher coarse reliability strata first. Early reliable updates alter the policy that generates later trajectories and teacher targets, propagating the effect of prompt order through training.

Our main teacher is Qwen3-30B-A3B-Instruct-2507, paired with Qwen3-1.7B, 4B, and 8B students (Yang et al. 2025). We also pair Gemma4-26B-A4B-it with Gemma4-E2B and E4B students (Team et al. 2026).

We use DeepMath-1K for reliability and ordering studies, DeepMath-5K for the main comparison, and DeepMath-17K for data scaling. All sets are derived from DeepMath-103K (He et al. 2025). For each prompt, the teacher library contains up to 16 verifier-correct responses. Every student–seed pair independently generates one rollout per prompt for proxy scoring. On DeepMath-5K, 4,720 of 5,000 prompts have a nonempty library; the remaining 280 form the unscored tail and remain in the training pool. Code-domain validation uses a 5,000-prompt subset of the code split of Eurus-2-RL-Data (Cui et al. 2025).

On DeepMath-1K, $\widehat{R}$ uses $K_{s}=16$ student trajectories, five uniformly spaced nonterminal prefixes per trajectory (at $1/6,\ldots,5/6$ of the response), and $M=16$ teacher continuations per prefix: 80 tested prefixes and 1,280 continuations per prompt. Initial ReOrder scoring instead draws 16 teacher responses per prompt, verifies them to retain correct references, and adds one independently sampled student scoring rollout per prompt. This rollout is excluded from the $K_{s}=16$ trajectories used to estimate $\widehat{R}$ and is never reused for OPD training. These preprocessing passes are absent from vanilla OPD; matched comparisons equalize prompt visits and update counts, not total generation and verification cost. The teacher library is reusable across students, seeds, OPD objectives, and refreshes for a fixed teacher–dataset pair; changing either requires rebuilding it. Relative to the diagnostic $\widehat{R}$ estimate, initial scoring uses $80\times$ fewer teacher samples and $16\times$ fewer student samples in generated-sequence counts.

We evaluate AIME24/25/26 and HMMT25-Feb, HMMT25-Nov, and HMMT26-Feb. For each problem, mean@16 is verifier correctness averaged over 16 sampled responses. The six-set score averages problem-level mean@16 over all 183 problems. Standard deviations are computed across three training seeds. We use mean@16 as the primary metric because it estimates expected correctness under the fixed sampling configuration using all generated responses. For code generation, we evaluate HumanEval+ (164 tasks) and MBPP+ (378 tasks) with EvalPlus (Liu et al. 2023), and LiveCodeBench V6 (175 tasks) (Jain et al. 2024). Code Avg. is their unweighted mean.

The 1K, 5K, and 17K comparisons use batch sizes 16, 64, and 64 for 63, 78, and 266 updates, respectively, giving approximately one pass over each prompt pool; Code-5K likewise uses batch size 64 for 78 updates. Vanilla OPD and ReOrder use a learning rate of $10^{-6}$, one on-policy student rollout per prompt visit, and a maximum response length of 16,384 tokens. Evaluation samples 16 responses with temperature 1.0 and top-$p$ 1.0. Matched conditions share the prompt pool, update budget, and all other settings; only prompt order differs. Three-seed studies use seeds 42–44.

Table 1 compares ReOrder-OPD with matched vanilla OPD across Qwen3 and Gemma4 teachers, five student configurations, six mathematics benchmarks, and three training seeds. Within each student, the teacher, initialization, prompt pool, OPD objective, update budget, and evaluation protocol are held fixed, so the comparison isolates prompt order. ReOrder raises the Six-set aggregate for all five students, with gains of 1.09–2.58 percentage points. All 15 model–seed aggregate paired differences are positive, and 27 of the 30 model–benchmark means improve. The limited benchmark-level regressions are localized rather than systematic across a model family or benchmark group. The consistent aggregate gains across two model families and five student scales show that reliability-aware ordering improves performance under a fixed OPD update budget.

Table 2 tests the same ordering intervention on a different class of verifiable generation tasks under matched models, prompt pools, objectives, and update budgets. Code Avg. improves for both Qwen3-1.7B and Qwen3-4B, and all six student–seed paired differences are positive. The mean direction is also positive on HumanEval+, MBPP+, and LiveCodeBench V6 for both students. Prompt ordering benefits code generation without changing the underlying OPD objective, extending the scheduling principle beyond mathematical reasoning.

FiRe-OPD and ExOPD (Li et al. 2026c; Yang et al. 2026a) determine how supervision is applied within an on-policy rollout, whereas ReOrder determines when the prompt is visited before that rollout is generated. Table 3 composes the two intervention levels while retaining each trajectory-level objective, teacher, prompt pool, and update budget. At seed 42, adding ReOrder improves all six combinations spanning three student scales and both objectives. The gains appear for both FiRe-OPD and ExOPD at every tested scale, rather than depending on one particular trajectory-level loss. Improvements in all six tested combinations demonstrate that prompt scheduling is compatible with both trajectory-level objectives across the three student scales.

The first block of Table 4 compares equal-size subsets drawn from the same initial student state, with the subset size and 21-update training budget held fixed. High-$R$ prompts obtain the strongest average result and exceed low-$R$ prompts by 1.25 points on the combined score. Middle-$R$ and random are close, so the useful distinction is primarily between coarse reliability strata rather than a finely resolved ranking of individual prompt gains. These results show that $R$ separates high- and low-utility prompt groups at the coarse resolution needed for scheduling.

The full-pool block of Table 4 changes only access order: descending $R$ is 1.94 points above vanilla, ascending $R$ is 1.94 below it, and all three seed-level differences favor descending $R$. Figure 1(b) holds the prompt pool, objective, and update budget fixed. Student-correct-first, teacher-success-first, and annotated easy-first sort prompts respectively by the base student’s empirical correctness rate, the teacher’s verified success rate, and the dataset difficulty annotation; ReOrder outperforms all three orders. $R$ summarizes the interaction among the prompt, student-induced prefix, and teacher continuation, and is the strongest scheduling signal among the tested alternatives.

We evaluate whether the inexpensive score used by ReOrder preserves the coarse reliability structure exposed by the oracle experiments. On the same 905 prompts, the actual one-rollout ROUGE-5 score in Figure 3 partitions prompts into deciles whose mean $R$ rises from 0.29 to 0.98, with all nine adjacent changes positive. The local signals in Figure 1(a) have weak prompt-level associations with continuation success, and their decile curves reverse direction multiple times. ROUGE-5 thus serves as an ordinal scheduling score that separates coarse reliability strata, rather than a calibrated estimator of $R$. Maximum aggregation accounts for multiple valid solution paths by requiring the student rollout to resemble at least one verifier-correct teacher trajectory.

Table 5 compares proxy variants on the same 905 prompts, fixed scoring rollouts, and teacher trajectories. ROUGE-3 and ROUGE-5 both produce nine increasing decile transitions, whereas ROUGE-1 produces eight. Their agreement indicates that the coarse ranking is not specific to one $n$-gram order. With ROUGE-5 fixed, maximum aggregation preserves all nine increases and produces a larger high–low reliability separation than either mean or random-teacher aggregation on the shared rollouts. Maximum assigns high priority when the student resembles any one correct solution, which is appropriate for prompts with multiple valid paths.

Oracle $R$ identifies the useful access direction, and one scoring rollout recovers it at the coarse resolution needed for scheduling. Training still draws fresh on-policy trajectories and applies the original OPD objective.

Table 6 tests whether a static ordering remains useful as the DeepMath prompt pool grows. Within each scale, Vanilla and ReOrder use the same data, student, objective, and matched number of updates. ReOrder improves the AIME25/26 score at 1K, 5K, and 17K, showing that the benefit persists beyond the small-pool setting. The gain decreases from 3.33 to 1.15 points as the pool and training trajectory grow. The shrinking gain suggests that a static ranking becomes less informative over a longer training trajectory, motivating priority refresh.

The scale trend motivates refreshing priorities as the student evolves. Table 7 compares Vanilla, Static ReOrder, and Dynamic ReOrder on DeepMath-17K under the same teacher library, prompt pool, and 266-update budget. Dynamic refresh changes only scores and order for unvisited prompts. For every student, Dynamic has the highest combined score, followed by Static and then Vanilla. The individual columns contain two exceptions: Static is highest on AIME26 for Qwen3-4B, and Vanilla is highest there for Qwen3-8B. The aggregate pattern matches the policy dependence of $q(x;\pi_{s})$: refreshed priorities track the student generating the remaining on-policy trajectories. Dynamic ReOrder adds student-side scoring but no teacher generation or OPD updates at refresh time, preserving the matched OPD update budget.