AdaCuRL: Adaptive Curriculum Reinforcement Learning with Invalid Sample Mitigation and Historical Revisiting

基本信息

• 标题: AdaCuRL: Adaptive Curriculum Reinforcement Learning with Invalid Sample Mitigation and Historical Revisiting
• 第一作者: Renda Li (AMAP, Alibaba Group)
• 研究团队: amap
• 会议/期刊: arXiv:2511.09478v1 [cs.LG], 2025
• 代码: Not explicitly mentioned
• PDF 文件: [AdaCuRL](file:///C:/Users/admin/.openclaw/workspace/attachment/papers/20260708_adacurl_adaptive_curriculum_rl.pdf)

研究摘要

In the rapidly evolving landscape of large language model reasoning, the pursuit of enhanced complex reasoning capabilities has emerged as one of the most consequential research frontiers. While supervised fine-tuning (SFT) has traditionally served as the cornerstone for distilling expert reasoning trajectories into models, the recent breakthrough of reinforcement learning-based approaches—most notably Group Relative Policy Optimization (GRPO) as demonstrated by DeepSeek-R1—has illuminated a compelling alternative: models can potentially self-improve their reasoning abilities without dependence on expensive, labor-intensive high-quality Chain-of-Thought (CoT) annotations. This paradigm shift, however, conceals a critical vulnerability that becomes increasingly apparent as researchers attempt to apply RL training to real-world datasets. When models are trained on data with mixed difficulty levels, the coupling between sample difficulty and the relative advantages within GRPO rollout groups produces a devastating phenomenon that undermines the very foundation of learning.

The core intellectual contribution of this work lies in its penetrating diagnosis of two interrelated pathologies that afflict standard GRPO training: Gradient Starvation and Policy Degradation. When training samples exhibit extreme difficulty relative to the current policy's capability, the reward signal collapses into a binary regime where simple samples uniformly receive rewards of 1 while difficult samples invariably yield rewards of 0. This reward collapse propagates through the advantage function, causing it to vanish entirely and thereby nullifying the policy gradient. The model, deprived of any meaningful learning signal, enters a state of gradient starvation where no productive weight updates occur. Concurrently, the KL divergence penalty imposed on these invalid samples dominates the optimization landscape, forcing the policy to retreat toward a conservative reference model and effectively erasing any reasoning capabilities that had been acquired through prior RL training. The authors present compelling evidence that these invalid samples are not rare anomalies but pervasive features of standard open-source datasets, as illustrated by their cumulative count analysis during GRPO training.

To address these fundamental challenges, the authors propose AdaCuRL, an adaptive curriculum reinforcement learning framework that reimagines the relationship between model capability and training data difficulty. Rather than treating the training dataset as a static, homogeneous collection, AdaCuRL introduces a sophisticated coarse-to-fine difficulty estimation strategy that enables precise characterization of each sample's difficulty relative to the current model. This estimation feeds into an adaptive curriculum scheduling mechanism that dynamically aligns data difficulty with the model's evolving capability, ensuring that the model consistently trains on samples that are appropriately challenging—neither trivially easy nor impossibly hard. Furthermore, the framework incorporates a data revisitation mechanism that periodically reintroduces historical data to mitigate catastrophic forgetting, a common ailment in curriculum learning where performance on earlier, easier material deteriorates as training progresses toward harder samples. Complementing these curriculum innovations, AdaCuRL deploys adaptive reference and sparse KL strategies that prevent the policy degradation caused by invalid samples, preserving the model's hard-won reasoning capabilities throughout the training trajectory.

The experimental validation of AdaCuRL spans both multimodal large language models (MLLMs) and unimodal LLMs, demonstrating consistent and significant performance improvements across diverse reasoning benchmarks. On multimodal mathematical reasoning tasks, AdaCuRL achieves improvements of 3.17% and 2.16% on Qwen2.5-VL-3B and Qwen2.5-VL-7B respectively, while on language-only mathematical reasoning, the gains reach 3.45% and 5.53% on Qwen2.5-Math-1.5B and Qwen2.5-Math-7B. These results are not marginal increments but substantial advances that underscore the critical importance of aligning training data difficulty with model capability. The self-pacing variant, Re-AdaCuRL, pushes performance even further by iteratively re-estimating sample difficulty and re-sampling the training data, enabling the model to continuously mine increasingly informative samples from the dataset. This work matters because it provides a principled, theoretically grounded solution to a fundamental training pathology that has likely constrained the reasoning potential of many RL-based approaches, opening pathways for more efficient and effective development of reasoning-capable AI systems.

理论框架

The intellectual foundations of AdaCuRL draw upon a rich tapestry of learning theory spanning curriculum learning, self-paced learning, and policy optimization in reinforcement learning. To understand how this work advances the field, we must trace the evolution of ideas that converge in this framework and examine how the authors reframe existing understanding through the lens of invalid sample mitigation.

Curriculum learning, as originally formulated by Bengio et al. (2009), draws inspiration from the human educational experience where learners progressively master increasingly complex material. The formal paradigm organizes training data according to a difficulty measure, ensuring that the model encounters examples in ascending order of complexity. In mathematical terms, given a dataset $\mathcal{D}=\{(x_i,y_i,d_i){\}}_{i=1}^N$ where $d_i$ represents the difficulty of sample $(x_i,y_i)$, the curriculum enforces the ordering $d_1\le d_2\le \cdots \le d_N$. This seemingly simple ordering principle carries profound implications for optimization dynamics: by initially training on easier samples, the model can acquire foundational patterns and representations that serve as scaffolding for learning more complex concepts. However, classical curriculum learning leaves two critical questions unanswered: how should difficulty be measured, and how should the training schedule be determined? Early approaches relied on predefined heuristics or human-crafted curricula, which inevitably fail to capture the model's subjective experience of difficulty and cannot adapt to the evolving capabilities of the learner.

Self-paced learning (SPL) addresses the difficulty estimation question by making it model-dependent. Rather than relying on external assessments, SPL selects samples according to the current loss, constructing training sets for each epoch as $\mathcal{D}=\{(x_i,y_i)\mid \ell (f_w(x_i),y_i)\le \tau \}$ where $\tau$ is an adaptive loss threshold. This formulation elegantly captures the intuition that a sample's difficulty is not an intrinsic property but a relational property between the sample and the model's current state. However, SPL was designed for supervised learning settings and does not directly translate to the reinforcement learning context where the reward signal, rather than a direct loss, drives learning. The predefined versus automatic scheduler distinction further complicates matters: predefined schedulers follow fixed rules regardless of model progress, while automatic schedulers make decisions based on model feedback, but existing automatic schedulers have not adequately addressed the unique challenges posed by GRPO's group-relative reward structure.

Group Relative Policy Optimization (GRPO) represents the proximal foundation upon which AdaCuRL builds. GRPO eliminates the need for a separate value model—a significant architectural simplification—by normalizing outcome rewards within a group of $G$ samples and applying a policy-gradient objective regularized by a KL divergence term. For a prompt $q$, the policy ${\pi }_{\theta }$ generates $G$ responses $\{o_i\}$ with scalar rewards $\{r_i\}$. Letting ${\mu }_r$ and ${\sigma }_r$ denote the group mean and standard deviation, GRPO defines the group-relative advantage as:

where $\epsilon >0$ prevents division by zero. The importance ratio is defined as ${\rho }_i=\frac{{\pi }_{\theta }(o_i|q)}{{\pi }_{\text{old}}(o_i|q)}$, and the CLIP operation clips this ratio to the range $[1-ϵ,1+ϵ]$. The complete GRPO objective then becomes:

where $\beta$ controls the KL regularization strength. This objective appears elegant and well-posed, but the AdaCuRL authors expose a critical vulnerability in its assumptions. When all rewards within a group collapse to 0 or 1, the numerator $r_i-{\mu }_r$ becomes zero for all samples, causing the entire advantage vector to vanish. In this scenario, the first term of the objective collapses to zero, leaving only the KL divergence term. The optimization then reduces to minimizing the distance from the reference policy, which actively erases any reasoning improvements that the model had achieved. The authors term this dual pathology "Gradient Starvation" and "Policy Degradation"—the former describing the absence of learning signal, the latter describing the regressive pressure toward the reference model.

The theoretical insight that distinguishes AdaCuRL is the recognition that sample difficulty and the validity of GRPO's group-relative advantages are fundamentally coupled. This coupling implies that curriculum learning is not merely a training convenience but a theoretical necessity for stable GRPO training. The coarse-to-fine difficulty estimation strategy operationalizes this insight by treating difficulty estimation as a sampling problem rather than a classification problem. The coarse stage generates five responses per problem and bins them into three groups based on the number of correct answers: $G_1=\{i\mid c_i\in \{0,1\}\}$ for hard problems, $G_2=\{i\mid c_i\in \{2,3\}\}$ for medium problems, and $G_3=\{i\mid c_i\in \{4,5\}\}$ for easy problems. Sampling according to predefined ratios from these bins achieves the desired overall difficulty distribution without requiring exhaustive inference on the entire dataset. The fine stage then performs more extensive generation ($N\gg 5$) to obtain precise difficulty scores, defined as $\text{Difficulty}(q)=1-\frac{c(q)}{N}$, which measures the proportion of failed attempts. This two-stage approach elegantly balances inference cost against estimation accuracy, recognizing that coarse estimation suffices for distributional sampling while fine estimation enables precise curriculum ordering.

The competence score mechanism introduces a novel theoretical construct for modeling the model's mastery of the current curriculum. Initialized at $c{s}^{(0)}=0$ and updated as $c{s}^{(t+1)}\leftarrow c{s}^{(t)}+(\overline{r}-0.5)\cdot max(1-c{s}^{(t)},\gamma )$, this score integrates the model's recent performance into a running estimate of capability. The term $(\overline{r}-0.5)$ captures whether the model is performing above or below chance level on the current bucket, while $max(1-c{s}^{(t)},\gamma )$ acts as a decay factor that reduces update magnitude as competence increases, mimicking the diminishing returns of additional practice on mastered material. The curriculum expansion condition $cs\ge \frac{k-1}{K}$ provides a principled threshold for introducing the next difficulty level, ensuring that the model demonstrates adequate mastery before advancing. This theoretical formulation transforms curriculum learning from a fixed schedule into an adaptive process that responds to the model's actual learning trajectory, rather than a predetermined timetable that may overtrain easy material or prematurely expose the model to overwhelming difficulty.

The sparse KL and adaptive reference mechanisms address the theoretical necessity of preventing policy degradation without abandoning the stabilizing benefits of KL regularization. The conditional KL computation excludes the KL divergence term when the advantage vector is entirely zero—precisely the scenario where it would cause harmful regression—while retaining it when valid learning signals exist. This can be expressed as:

where $\mathbb{I}[{\hat{A}}_i\ne 0]$ is the indicator function that gates the KL term. The reference model resetting strategy resets ${\pi }_{\text{ref}}$ to the current policy after each bucket update, which theoretically prevents the growing divergence between the improving policy and a static reference model. This is particularly important because as the model acquires reasoning capabilities, the distance from the initial base model increases, and the KL penalty becomes increasingly distortive. The theoretical assumption here is that the reference model should serve as a stabilization anchor near the current policy, not as a rigid tether to the initialization point.

The assumptions underlying this theoretical framework bear careful consideration. The difficulty estimation assumes that the model's success rate on multiple generations is a reliable proxy for the sample's intrinsic difficulty, which holds when the model's sampling is reasonably calibrated but may break down for highly ambiguous or noisy problems. The curriculum assumption is that difficulty is monotonically related to learning progression, which may not hold for all types of reasoning—some complex problems might require fundamentally different skills rather than simply more advanced versions of the same skills. The competence score assumes that average reward over recent samples is a faithful measure of mastery, which depends on the sample size $M$ being large enough to smooth out noise but small enough to remain responsive to recent changes. The framework also assumes that catastrophic forgetting can be adequately mitigated by revisiting historical data, though the optimal frequency and depth of such revisiting remains an open theoretical question.

技术架构

AdaCuRL presents a unified technical architecture that weaves together three interconnected components into a coherent training system: coarse-to-fine difficulty estimation, adaptive curriculum reinforcement learning, and the self-pacing Re-AdaCuRL extension. The architecture operates as a pipeline that transforms raw datasets into progressively more capable reasoning models through a carefully orchestrated interplay of data selection, dynamic scheduling, and loss engineering.

At the entry point of the system, the coarse-to-fine difficulty estimation module serves as the data curator. This module receives a large-scale training dataset—potentially hundreds of thousands of problems—and must produce a carefully ordered subset of manageable size that matches a target difficulty distribution. The coarse stage operates as a rapid triage mechanism: for each problem, the model generates five responses, and the count of correct answers $c_i\in \{0,\dots ,5\}$ determines the initial bin assignment. This generates three groups: $G_1$ for problems where the model fails on most attempts (0-1 correct), $G_2$ for problems with mixed success (2-3 correct), and $G_3$ for problems the model solves reliably (4-5 correct). From these bins, the system samples according to predefined ratios ${\rho }_k$ to construct a subset of size $S$ that contains more hard problems than easy ones, following the pedagogical principle that appropriately challenging material drives the most effective learning. The formal sampling is expressed as $S=\bigcup _{k=1}^3\text{Sample}(G_k,n_k=\lfloor {\rho }_k|G_k|\rfloor )$, which ensures that the selected samples remain evenly distributed across the source datasets while achieving the desired difficulty mix.

The fine stage then refines this coarse selection through more intensive inference. For each problem in the sampled subset $S$, the model generates $N$ responses (where $N$ is much larger than the coarse stage's 5, typically 100) to obtain a precise difficulty estimate. The difficulty score is computed as $\text{Difficulty}(q)=1-\frac{c(q)}{N}$, which ranges from 0 (the model always solves it) to 1 (the model never solves it). The system filters out problems with difficulty above 0.95 or below 0.05, eliminating trivially easy problems that offer no learning value and impossibly hard problems that would trigger the invalid sample pathology. The remaining data is sorted by ascending difficulty to form the final training dataset $\mathcal{D}$, establishing the foundation for the curriculum.

The curriculum reinforcement learning module forms the core training engine. After sorting $\mathcal{D}$ by difficulty, the system partitions it into $K$ consecutive buckets of equal size: $B_k=\{q_{(k-1)\frac{|\mathcal{D}|}{K}+1},\dots ,q_{k\frac{|\mathcal{D}|}{K}}\}$ for $k=1,\dots ,K$. The current training curriculum ${\mathcal{D}}_c$ is initialized as the first bucket $B_1$. During training, the system maintains a reward buffer $R_b$ that records the accuracy rewards of recently processed samples, and a competence score $cs$ that tracks the model's mastery of the current curriculum. The competence score update follows a carefully designed dynamics equation: $c{s}^{(t+1)}\leftarrow c{s}^{(t)}+(\overline{r}-0.5)\cdot max(1-c{s}^{(t)},\gamma )$. Here, $\overline{r}$ is the average reward over the most recent $M$ training samples, and the $max(1-c{s}^{(t)},\gamma )$ term ensures that the update rate remains bounded below by $\gamma$ (set to 0.5), preventing the learning rate from collapsing to zero even as competence approaches perfection. This mechanism mimics the pedagogical insight that learners should spend more time on harder material while still making measurable progress on mastered material.

The curriculum expansion logic determines when the model is ready for more challenging material. Once the reward buffer contains $M$ samples, the system updates the competence score and checks the expansion condition $cs\ge \frac{k-1}{K}$. When satisfied, the next bucket $B_k$ is merged into the current curriculum: ${\mathcal{D}}_c^{(t+1)}={\mathcal{D}}_c^{(t)}\cup B_{t+2}$. Importantly, upon merging, the competence score is re-initialized to $cs=\frac{k-1}{K}$ to ensure that the subsequent mastery estimate reflects only the newly added, harder material rather than being contaminated by the already-mastered easier material. This re-initialization is a subtle but crucial design choice: without it, the high competence accumulated on easy material would prematurely trigger the next expansion, potentially flooding the curriculum with overwhelming difficulty before the model has actually mastered the current level. The system continues until ${\mathcal{D}}_c=\mathcal{D}$, at which point the full curriculum has been traversed.

The reward function design addresses an important imbalance in GRPO training. The authors observe that format rewards converge rapidly while accuracy rewards, especially from harder buckets, progress slowly. This imbalance distorts the advantage function because the format reward dominates the group-relative normalization, causing the model to optimize for formatting rather than reasoning. AdaCuRL resolves this by using both format and accuracy rewards during the initial $T_f$ steps (where $T_f=64$), then switching to accuracy-only rewards thereafter. This staged reward design ensures that the model first learns to produce properly formatted outputs before focusing exclusively on reasoning correctness, preventing the format reward from masking the more meaningful accuracy signal.

The KL divergence design represents a critical engineering innovation. Standard GRPO computes the KL divergence with the base model during every loss computation, which creates two problems. First, when the advantage function is a zero vector (the gradient starvation scenario), the loss becomes entirely dominated by the KL term, forcing unnecessary alignment with the base model. Second, as the model's reasoning improves, the distance from the base model grows, and the KL penalty increasingly undermines the acquired reasoning capabilities. AdaCuRL introduces two strategies: conditional KL computation and reference model resetting. The conditional computation checks whether all rewards in a rollout group are 0 or 1 (which would produce a zero advantage vector), and excludes the KL term for that specific group. The loss becomes:

The indicator function $\mathbb{I}[{\hat{A}}_i\ne 0]$ acts as a gate that preserves the KL regularization when learning signals are present but eliminates it when they are absent. The reference model resetting strategy updates ${\pi }_{\text{ref}}$ to the current policy after each bucket update, ensuring that the KL divergence is computed relative to a recent checkpoint rather than the increasingly distant initialization. This prevents the KL penalty from growing ever more distortive as the model acquires substantial reasoning capabilities.

The self-pacing mechanism, Re-AdaCuRL, extends the architecture into an iterative training paradigm. After completing the first round of curriculum training, the model has developed stronger reasoning capabilities, which means its perception of difficulty has evolved. Problems that were previously impossibly hard may now be tractable, while problems that were previously challenging may now be trivially easy. Re-AdaCuRL re-runs the coarse-to-fine difficulty estimation using the updated policy model ${\pi }_{\theta }$ and discards samples with difficulty scores below a threshold (0.2), filtering out material that no longer provides learning value. The remaining data is re-sorted and re-partitioned into $K$ buckets based on the updated difficulty estimates, and the curriculum training process is repeated. This creates a virtuous cycle where the model's improving capabilities unlock access to previously inaccessible training material, while the curriculum ensures that this new material is introduced at the appropriate pace. The iterative nature of Re-AdaCuRL means that data utilization continues to improve across rounds: samples that were excluded in the first round due to excessive difficulty can be revisited in subsequent rounds, while the filtering mechanism prevents wasteful training on already-solved problems.

实验评估

The experimental evaluation of AdaCuRL is designed as a comprehensive investigation that systematically addresses multiple research questions: whether the proposed curriculum approach outperforms standard GRPO and SFT baselines, whether the coarse-to-fine difficulty estimation is effective, whether the KL divergence design is necessary, and whether the self-pacing mechanism provides additional benefits. The authors construct a rigorous evaluation framework spanning both multimodal and unimodal reasoning tasks, using multiple model sizes and extensive benchmark suites to ensure the robustness and generalizability of their findings.

For multimodal experiments, the training dataset is curated from twelve diverse mathematical reasoning sources including CLEVR, CLEVR-Math, Geo3K, GeoMverse, GeoQA+, IconQA, Super-CLEVR, TabMWP, UniGeo, GEOS, WeMath, SceMQA, and PolyMath, comprising approximately 100,000 problems across various types such as geometry, algebra, and counting. The evaluation employs two complementary benchmark suites: a mathematical reasoning benchmark consisting of DynaMath, MathVista MINI, Math-V, MathVerse MINI, and LogicVista; and a general multimodal reasoning benchmark comprising MMStar, MMMU, HallusionBench, AI2D, and MMVET. For unimodal language experiments, the Open-RS dataset with 7,000 samples is used for training, while evaluation employs AIME24, AMC23, Math500, Minerva, and Olympiad-bench. This dual-track evaluation design enables the assessment of whether AdaCuRL's benefits generalize across both visual and textual reasoning domains.

The model configurations span both MLLMs and LLMs. For multimodal tasks, the authors employ Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct, representing small and moderately sized vision-language models. For language tasks, they use Qwen2.5-Math-1.5B and Qwen2.5-Math-7B, enabling the examination of whether curriculum learning benefits scale with model capacity. The training hyperparameters are carefully configured: for MLLMs, the learning rate is set to $1e-6$ with a linear schedule, global batch size of 16, and maximum 1050 steps; for LLMs, the learning rate is similarly $1e-6$ but with a cosine schedule with minimum learning rate ratio of 0.1, and a maximum of 650 steps. The number of generations per prompt is 6 for both settings, and the format reward cutoff $T_f$ is set to 64 based on empirical observation that format rewards typically converge around 32 steps in baseline GRPO training. The competence score update interval $M$ is set to 512, balancing training efficiency against accurate reflection of the model's current mastery, and the number of buckets $K$ is set to 4 for MLLMs and 3 for LLMs (the latter using fewer buckets due to the smaller dataset size).

The main results, presented in Tables 1 and 2, reveal a striking pattern: neither the original GRPO nor SFT significantly enhances reasoning capabilities on MLLMs. On Qwen2.5-VL-3B, GRPO improves mathematical and general reasoning by only 0.85% and 0.19% respectively, while SFT actually leads to degraded performance, particularly on the larger 7B model. The authors attribute this SFT degradation to fine-tuning on lower-quality open-source data that may harm an already strong baseline. For language models, the original GRPO shows more noticeable gains, improving by 3.56% on Qwen2.5-Math-1.5B and 2.84% on Qwen2.5-Math-7B, which the authors attribute to the additional complexity introduced by multimodal information fusion that makes RL fine-tuning more challenging for MLLMs. These baseline results establish the critical context: without curriculum learning, RL-based post-training on these datasets provides limited or even negative returns.

Against this backdrop, AdaCuRL achieves substantial improvements across all configurations. On the multimodal mathematical reasoning average, AdaCuRL improves by 3.17% on Qwen2.5-VL-3B and 2.16% on Qwen2.5-VL-7B compared to standard GRPO. On the language model mathematical reasoning average, the gains are even more pronounced: 3.45% on Qwen2.5-Math-1.5B and 5.53% on Qwen2.5-Math-7B. These improvements are not isolated to a single benchmark but consistently appear across the majority of individual datasets. The results are summarized in the following tables:

The ablation studies provide crucial insight into the contribution of each component. Removing SparseKL, Reset Ref, or Revisiting individually degrades performance, confirming that each component serves a necessary function. Notably, completely removing the KL divergence term causes a substantial performance drop, indicating that the KL term itself is essential for preventing overfitting to simpler samples during historical revisiting, and that AdaCuRL's innovation is not the removal of KL but its conditional, adaptive application. The ablation of different training schedulers reveals that the naive curriculum learning approach, which processes buckets from easy to hard without considering model feedback, consistently underperforms AdaCuRL's adaptive scheduling across all models. This validates the theoretical claim that fixed schedules are suboptimal because they ignore the model's evolving capabilities and may overtrain easy material while under-training hard material.

The difficulty distribution analysis further illuminates the method's dynamics. Training exclusively on easy data yields high rewards but fails to develop deeper reasoning capabilities, evidenced by completion lengths that actually decrease during training. Conversely, training solely on hard data produces longer reasoning but persistently low average rewards, indicating that the model struggles to reach correct answers. The default curriculum distribution, which progressively increases difficulty, achieves the best balance: maintaining high data utilization while steadily increasing reasoning length as harder material is introduced. This empirical finding supports the theoretical intuition that curriculum learning must align difficulty with capability, not simply maximize or minimize it.

The statistical significance analysis using the Wilcoxon signed-rank test confirms that AdaCuRL's improvements are not due to random variation. All p-values across the four evaluated models are below the significance threshold of $\alpha =0.05$, with values ranging from 0.001 to 0.031, providing strong statistical evidence that AdaCuRL significantly outperforms baseline GRPO. The bucket number analysis demonstrates robustness to moderate choices (3, 4, or 5 buckets), while extreme values (1 or 10 buckets) degrade performance, suggesting that the method is not hypersensitive to hyperparameter tuning within a reasonable range.

案例研究

The paper provides several empirical analyses that serve as instructive case studies illuminating the inner workings of AdaCuRL. One particularly revealing example concerns the coarse-grained difficulty distribution shift observed after one round of AdaCuRL training. Table 3 in the paper tracks the number of samples falling into each of the three coarse groups before and after training, using a subset of source datasets for clarity. Before training, the distribution across the clever, math, and geo3k datasets shows a substantial presence in $G_1$ (the hardest group). After training, the counts in $G_1$ decrease dramatically—clever drops from 1142 to 849, math from 1324 to 1017, and geo3k from 558 to 451—while the counts in $G_3$ (the easiest group) increase significantly, with clever rising from 1560 to 1964 and math from 151 to 399. This distribution shift serves as a compelling case study of adaptive learning: the model's improved reasoning capabilities have transformed previously hard problems into relatively easy ones, as measured by the updated policy's success rate. The fact that the model can now solve problems in $G_3$ that it previously struggled with demonstrates that the curriculum has successfully scaffolded the acquisition of transferable reasoning skills.

The analysis of different difficulty estimation strategies, presented in Table 4, provides another illuminating case. When training Qwen2.5-VL-3B using only coarse-grained estimation, the average mathematical reasoning score drops to 42.81 compared to 43.81 achieved by fine-grained estimation. Similarly, using fine-grained estimation from a stronger external model (Qwen2.5-VL-7B) also yields suboptimal results (43.62). This case study reveals a subtle but crucial insight: the difficulty of a problem is not an intrinsic property but a model-dependent relationship. A stronger external model's assessment of difficulty does not necessarily provide the optimal curriculum for a weaker student model, because what is hard for the student may not be hard for the expert. The optimal curriculum must be calibrated to the learner's current capability, not to an external standard. This finding reinforces the theoretical foundation of AdaCuRL's model-dependent difficulty estimation.

The training dynamics comparison between AdaCuRL and randomly shuffled data, illustrated in Figure 4, provides a temporal case study of how curriculum scheduling affects learning trajectories. The accuracy reward curve for AdaCuRL demonstrates a steady, stair-case progression as the model masters each bucket before advancing to the next, while the shuffled data baseline shows more volatile and ultimately lower rewards. The KL loss dynamics reveal that the adaptive reference strategy in AdaCuRL maintains a lower average KL divergence compared to using a fixed reference model throughout training. This is because resetting the reference after each bucket update prevents the growing divergence between the improving policy and an increasingly stale reference point. The case study demonstrates that the KL divergence design is not merely a regularization detail but an active contributor to the model's ability to retain and build upon acquired reasoning capabilities.

The completion length analysis in Figure 5 offers a case study of how difficulty distribution shapes reasoning behavior. When training exclusively on easy data, the model's completion length decreases over time, suggesting that the model learns to solve these problems with minimal reasoning effort. When training exclusively on hard data, the completion length increases but the average reward remains low, indicating that the model is generating elaborate reasoning chains that often fail to arrive at correct answers. The default AdaCuRL curriculum, however, produces a steady increase in completion length while maintaining higher average rewards, indicating that the model is developing more sophisticated reasoning chains that are increasingly likely to be correct. This case study demonstrates that reasoning length is not inherently valuable—it is only valuable when it is calibrated to the problem's difficulty and the model's capability.

综合价值与局限

AdaCuRL represents a significant theoretical and practical contribution to the field of reasoning-oriented reinforcement learning. Theoretically, the work changes how we conceptualize the relationship between training data and model capability in the RL context. Rather than viewing data as a static resource to be consumed, AdaCuRL frames it as a dynamic landscape that must be navigated in harmony with the model's evolving capabilities. The identification of Gradient Starvation and Policy Degradation as fundamental pathologies of mixed-difficulty GRPO training provides a new diagnostic lens through which to evaluate training failures, and the proposed solutions offer principled design patterns for future RL training systems. The competence score mechanism and adaptive curriculum expansion provide concrete conceptual tools for thinking about how to measure and manage learning progress in reinforcement settings.

Practically, the impact of AdaCuRL is substantial because it enables more effective utilization of existing datasets without requiring additional expensive annotations or external models. The framework does not rely on CoT distillation data, expert models for difficulty estimation, or handcrafted curricula—all of which impose significant overhead or dependencies. Instead, AdaCuRL operates autonomously, using the model's own performance as the sole guide for difficulty assessment and curriculum scheduling. This self-contained nature makes it highly deployable in production settings where external resources may not be available or where data privacy concerns preclude the use of third-party models. The consistent improvements across both MLLMs and LLMs, spanning multiple model sizes and diverse benchmark suites, suggest that the benefits of adaptive curriculum learning are robust and generalizable.

The strengths of this work are numerous and compelling. The theoretical diagnosis of invalid sample pathologies is both original and rigorous, providing a clear causal explanation for training failures that might otherwise be attributed to model architecture or hyperparameter choices. The coarse-to-fine difficulty estimation strategy is an elegant engineering solution that balances inference cost against estimation accuracy, making the method feasible to apply to large-scale datasets. The sparse KL and adaptive reference mechanisms demonstrate sophisticated understanding of how KL divergence interacts with RL optimization, providing targeted solutions rather than crude workarounds. The comprehensive experimental validation, including statistical significance testing, ablation studies, and cross-domain evaluation, provides strong empirical support for the method's claims. The narrative of the paper is clear and well-structured, making the technical contributions accessible to readers.

However, honest assessment demands acknowledgment of limitations. The method assumes that problem difficulty is well-captured by the model's success rate on repeated generations, which may not hold for problems where the model's responses are highly variable or where the ground truth evaluation is noisy. The curriculum assumes a monotonic progression of difficulty, which may not reflect the true structure of reasoning skills—some advanced problems might require qualitatively different capabilities rather than simply more of the same capabilities. The reliance on a predefined target difficulty distribution (the 2K/3K/5K sampling ratios in the coarse stage) introduces a hyperparameter that may require tuning for different domains or datasets. The number of buckets $K$ requires empirical selection, and while the paper shows robustness to moderate values, the extreme values do degrade performance. The method does not address the fundamental question of what makes a good reasoning trajectory beyond the binary reward signal, potentially leaving value on the table compared to more fine-grained reward modeling. Additionally, the paper does not explore the computational overhead of the difficulty estimation stage relative to the training stage, which is relevant for practical deployment decisions.

Broader implications of this work connect to the growing recognition that data curation and training dynamics are as important as model architecture for achieving strong reasoning capabilities. AdaCuRL contributes to a trend in the field that emphasizes the algorithmic aspects of post-training—how we train, not just what we train on. This aligns with the broader realization that the remarkable capabilities of models like DeepSeek-R1 emerge not solely from scale but from the careful design of training procedures. AdaCuRL opens a research direction toward more sophisticated, adaptive training algorithms that respond to the model's internal state rather than following fixed schedules, potentially leading to a new generation of self-improving systems that require less human intervention in the training process.

延伸阅读与思考

The intellectual lineage of AdaCuRL can be traced through several foundational works that establish the theoretical and empirical groundwork upon which this research builds. The original curriculum learning formulation by Bengio et al. (2009) provides the conceptual foundation for organizing training data by difficulty, though their work focused on supervised learning settings and did not address the unique challenges of reinforcement learning. In the RL context, Graves et al. (2017) explored automated curriculum learning for neural networks, demonstrating that curriculum design can be itself learned or optimized. The work of Shi et al. (2025) on efficient reinforcement finetuning via adaptive curriculum learning is a particularly close predecessor, estimating problem difficulty using expert models and proposing adaptive schedulers, though their method lacks the historical data revisiting and policy degradation prevention mechanisms that distinguish AdaCuRL. The GRPO algorithm, as developed by Guo et al. (2025) in the DeepSeek-R1 work, provides the algorithmic substrate upon which AdaCuRL operates, and the identification of GRPO's limitations in mixed-difficulty settings is a direct response to the training challenges observed in applying this algorithm to diverse datasets.

In the broader landscape of reasoning-oriented LLM training, several alternative approaches exist for the same fundamental problem of enhancing reasoning capabilities. CoT finetuning approaches such as those by Dong et al. (2025) and Xu et al. (2024) fine-tune models on large-scale CoT datasets, providing explicit reasoning supervision but requiring expensive annotation or distillation. DeepSeek-R1 demonstrated that RL can spontaneously induce strong reasoning abilities, reducing the need for extensive CoT data, but this approach may require substantial computational resources and careful tuning. Hint-GRPO (Huang et al. 2025a) incorporates expert reasoning trajectories during RL to help difficult samples, but this approach introduces dependency on external expert demonstrations. Vision-R1 (Huang et al. 2025b) and R1-OneVision (Yang et al. 2025) extend the RL paradigm to multimodal settings, while MM-Eureka (Meng et al. 2025) explores rule-based large-scale reinforcement learning for visual reasoning. Compared to these alternatives, AdaCuRL occupies a unique position: it does not require external experts or CoT data, it does not rely on handcrafted rules, and it adapts to the model's evolving capabilities rather than following a fixed schedule.

The future directions opened by AdaCuRL are numerous and promising. One immediate extension would be to explore more sophisticated difficulty estimation methods that capture not just the success rate but the quality of reasoning trajectories—perhaps using process-based rewards or step-by-step evaluation rather than binary outcome rewards. Another direction would be to investigate adaptive curriculum learning in the context of online data collection, where the system could actively generate or select new problems rather than merely ordering existing ones. The self-pacing mechanism of Re-AdaCuRL could be extended to more iterations, potentially creating a continuous learning system that progressively unlocks more challenging material as capabilities improve. The application of adaptive curriculum principles to other RL algorithms beyond GRPO, such as PPO or DPO, represents another fertile research direction. Additionally, the integration of curriculum learning with test-time compute scaling, as explored by Snell et al. (2024), could yield synergistic benefits where both training and inference adapt to problem difficulty.

The deepest unsolved challenges in this area revolve around the fundamental nature of reasoning itself. We do not yet have a comprehensive theory of how reasoning capabilities compose, transfer, and scale across different domains and difficulty levels. The difficulty estimation in AdaCuRL, while effective, is ultimately a behavioral proxy—we measure difficulty by whether the model succeeds, not by any intrinsic structural analysis of the problem. A more foundational understanding of what makes problems difficult for neural networks, perhaps drawing on computational complexity theory or cognitive science, could lead to more principled curriculum designs. The phenomenon of catastrophic forgetting, which AdaCuRL addresses through data revisiting, remains a deep challenge in continual learning, and more sophisticated memory mechanisms—perhaps drawing on biological inspiration—could further improve retention of previously learned skills. The question of how to balance exploration and exploitation in curriculum selection, ensuring that the model is sufficiently challenged without being overwhelmed, connects to fundamental questions in learning theory and optimization.

Reflecting personally on this work, the most surprising and thought-provoking aspect is the revelation that invalid samples—those that are too easy or too hard—are not merely wasted training examples but actively harmful training examples. This counterintuitive finding challenges the common assumption that more data is always better, and instead suggests that the alignment between data and model is a critical, underappreciated dimension of training design. The fact that the KL divergence term, normally regarded as a benign regularizer, can become a destructive force when learning signals are absent, is another surprising insight that reveals how seemingly safe algorithmic choices can have pathological consequences under specific conditions. What I would most want to explore further is the boundary between where curriculum learning provides diminishing returns and where it becomes essential—are there regimes where model scale or data quality are sufficient that curriculum learning offers only marginal benefits, or is the invalid sample pathology fundamental to the RL training dynamics regardless of scale? Understanding this boundary would be crucial for determining when to invest in the computational overhead of curriculum design versus relying on simpler training approaches.

笔记创建时间: 2026-07-08
阅读方式: L2 深度阅读

Topics:

• "reinforce_learning"
• "llm"
• "reasoning"
• "agent_architecture"
• "evaluation"
  References:
• "amap"