Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

Authors: Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, Sean Hendryx (Scale AI)
Venue: arXiv preprint (arXiv:2507.17746)
Year: 2025
Pages: 20
Code: https://huggingface.co/datasets/anisha2102/RaR-Medicine; https://huggingface.co/datasets/anisha2102/RaR-Science
Archived PDF: PDF

1. 研究摘要 (Research Summary)

Large language models have achieved remarkable progress in reasoning, yet their training still relies heavily on signals that can be automatically verified — exact answers in mathematics, passing test cases in programming, or gold labels in multiple-choice questions. This paradigm, known as Reinforcement Learning with Verifiable Rewards (RLVR), has delivered impressive results in domains like math and code, where correctness is binary and unambiguous. However, the real world rarely offers such clean feedback. In medical diagnosis, scientific reasoning, or legal analysis, a response may be partially correct, well-reasoned but incomplete, or factually sound yet poorly structured. The challenge of extending reinforcement learning to these unstructured, high-stakes domains lies at the heart of this paper.

The authors identify a critical gap: existing approaches for non-verifiable domains typically fall into two unsatisfying extremes. On one end, preference-based reward models trained via RLHF offer flexibility but suffer from well-documented pathologies — they overfit to superficial artifacts like response length, formatting quirks, and annotator biases, and they demand massive volumes of pairwise human comparisons. On the other end, direct LLM-as-judge scoring using Likert scales provides a straightforward signal but collapses the rich, multi-dimensional nature of response quality into a single scalar number, losing the nuance that makes expert evaluation meaningful. The paper asks a compelling question: can we find a middle ground — a reward mechanism that preserves the structured, interpretable nature of expert evaluation while remaining automatable and scalable enough for on-policy training?

The answer proposed is Rubrics as Rewards (RaR), a framework that treats evaluation rubrics not merely as assessment tools but as active reward functions for reinforcement learning. Rubrics — structured checklists of criteria with varying importance, long used in education and benchmarking for nuanced evaluation — are repurposed as modular, interpretable supervision signals. Each criterion becomes a mini-verifier, and the overall reward aggregates these criterion-level judgments into a scalar signal suitable for policy optimization. This reframing is conceptually elegant: it decomposes the vague notion of "quality" into tangible, human-interpretable subgoals that can be checked automatically, offering a structured alternative to both opaque preference models and overly simplistic Likert scores.

The main contributions unfold across four dimensions. First, RaR introduces a principled framework for using checklist-style rubrics as multi-criteria reward signals in on-policy reinforcement learning, closing the loop between evaluation and training. Second, the authors synthesize and release two large-scale training datasets — RaR-Medicine and RaR-Science — containing instance-specific rubrics for medical and scientific reasoning, enabling the community to study structured rewards without costly human annotation. Third, comprehensive experiments demonstrate that RaR-trained policies consistently outperform strong baselines, achieving relative improvements of up to 31% on HealthBench and 7% on GPQA-Diamond compared to popular LLM-as-judge methods using direct Likert-based rewards. Fourth, and perhaps most insightfully, the work reveals that rubric-based rewards improve alignment not only for large judges but especially for smaller ones, reducing performance variance across judge scales and making high-quality supervision more accessible.

The key findings paint a clear picture of rubric-guided training's advantages. Rubric-based rewards yield more stable and generalizable training signals than Likert-based alternatives. They generalize beyond rubric-based evaluation settings, improving performance even on multiple-choice tasks. Perhaps most importantly, the structured decomposition of evaluation criteria helps smaller language models serve as more reliable judges, democratizing access to high-quality reward supervision. These results suggest that the act of explicitly structuring what we value in a response — breaking it down into essential facts, important reasoning steps, optional enhancements, and common pitfalls — is not merely an evaluation convenience but a fundamental lever for improving how models learn.

For the broader field, this work matters because it challenges the prevailing assumption that RL beyond verifiable domains must choose between the scalability of simple signals and the richness of human preferences. By demonstrating that structured, intermediate representations can serve as effective reward mechanisms, RaR opens a pathway for applying reinforcement learning to the vast landscape of real-world reasoning tasks where ground truth is elusive but expert judgment can be codified. It suggests a future where domain experts collaborate with AI systems not by labeling preferences but by articulating what constitutes good reasoning — and where those articulations become directly actionable training signals.

2. 理论框架 (Theoretical Framework)

Intellectual Lineage and Conceptual Foundations

The theoretical foundations of Rubrics as Rewards rest upon a convergence of three intellectual traditions: reinforcement learning with structured feedback, multi-criteria decision making, and the recent surge in using large language models as evaluators. The work builds most directly on the RLVR paradigm that has driven breakthroughs in mathematical and coding reasoning, where DeepSeek-R1 and similar systems demonstrated that verifiable rewards can elicit complex reasoning behaviors from language models. However, the authors recognize that this paradigm's success depends critically on the availability of unambiguous correctness signals — a condition that fails in most real-world domains.

From the RLHF tradition, the paper inherits an understanding of the challenges of preference-based training: the subjectivity of human judgments, the operational burden of collecting pairwise comparisons, and the propensity of reward models to exploit correlations rather than learn true quality. Yet rather than accepting these trade-offs, RaR seeks to transcend them by drawing on a different tradition — the use of rubrics in educational assessment and expert evaluation. Rubrics have long served as tools for making subjective judgment more structured and transparent, decomposing holistic quality into discrete, observable criteria. The theoretical insight of this paper is that this decomposition can be pushed one step further: not only can rubrics make evaluation more interpretable, but the individual criteria themselves can serve as primitive reward signals, creating a dense, structured supervision signal that is simultaneously more informative than a single scalar and more automatable than raw human preferences.

Core Concepts: From Holistic Quality to Structured Criteria

At the heart of the framework lies the concept of an instance-specific rubric — a set of criteria tailored to a particular prompt rather than a generic checklist applied uniformly across all tasks. This instance-specificity is crucial because what constitutes a good response to a medical case differs fundamentally from what makes a good response to a physics problem. Each rubric item is a self-contained evaluative statement that can be checked independently, carrying both a categorical importance label (Essential, Important, Optional, or Pitfall) and a numerical weight reflecting its relative significance.

The concept of Essential criteria captures the non-negotiable aspects of a response — facts that must be present, safety checks that cannot be violated. Important criteria cover the core reasoning and completeness that strongly affect quality without being absolutely disqualifying. Optional criteria reward helpful stylistic elements or extra depth that elevates a response from adequate to excellent. Pitfall criteria, perhaps the most innovative category, identify common mistakes or omissions specific to the prompt — they serve as negative criteria that, when violated, reduce the reward. This four-category taxonomy elegantly captures the multi-dimensional nature of expert evaluation: quality is not just about what is present but also about what must be avoided.

The paper's mathematical formulation formalizes this intuition through a reward function that generalizes beyond the binary verifiable setting. For a given prompt $x$ and a sampled response $\hat{y} \sim π_{θ} (\cdot | x)$ from the policy model, each of the $k$ rubric items is associated with a weight $w_{j} \in R$ and a binary correctness function $c_{j} : (x, \hat{y}) \mapsto {0, 1}$ indicating whether the response satisfies criterion $j$ . The explicit aggregation strategy computes the normalized reward as:

r (x, \hat{y}) = \frac{\sum_{j = 1}^{k} w_{j} \cdot c_{j} (x, \hat{y})}{\sum_{j = 1}^{k} w_{j}}

This normalization is conceptually important: it ensures that rewards remain comparable across prompts that differ in their number of rubric items or their weight distributions. Without normalization, a prompt with twenty criteria would dominate the training signal compared to one with five, distorting the learning process. The formulation also naturally subsumes verifiable rewards as a special case. As the paper formalizes in Remark 1, when $k = 1$ , $w_{1} = 1$ , and $c_{1} (x, \hat{y})$ reduces to a verifiable correctness check (such as exact match or test case execution), the rubric-based reward collapses to the standard RLVR setting:

r_{RLVR} (x, \hat{y}) = match (y, \hat{y})

where $match (y, \hat{y}) \in {0, 1}$ indicates whether the response satisfies the verifiable condition. This mathematical relationship is elegant: it reveals that rubric-based rewards are not a rejection of verifiable rewards but a principled generalization, expanding the reward mechanism from a single binary check to a weighted combination of multiple interpretable criteria.

Implicit Aggregation: Delegating Integration to the Judge

Complementing the explicit aggregation is the implicit aggregation strategy, which takes a different philosophical stance on how multi-criteria information should be combined. Rather than manually specifying weights and computing a weighted sum, implicit aggregation passes all rubric criteria along with their categorical importance labels to an LLM-as-judge, delegating the integration to the model itself:

r_{implicit} (x, \hat{y}) = f_{ϕ} (x, \hat{y}, {d_{j}}_{j = 1}^{k})

Here, $f_{ϕ}$ denotes an LLM-based judge that takes the prompt, the response, and the complete set of rubric criteria as input, producing a single holistic reward score. This formulation trades the interpretability and control of explicit weighting for the flexibility and nuance of a learned integration. The judge can implicitly weigh criteria based on their relevance to the specific response, account for interactions between criteria that a simple weighted sum might miss, and apply domain-specific heuristics that would be difficult to encode manually. The paper presents these two strategies as complementary extremes on a spectrum of control versus flexibility, with explicit aggregation offering maximum transparency and implicit aggregation offering maximum adaptability.

Theoretical Connections and Scope

The theoretical framework connects several previously separate threads. The decomposition of evaluation into criteria draws on multi-attribute utility theory, which has long recognized that human preferences over complex objects are better modeled as aggregations of simpler attribute evaluations than as holistic judgments. The use of binary correctness functions for each criterion mirrors the verifiable reward paradigm but applied at a finer granularity — turning the problem of evaluating a complete response into multiple smaller verification problems. The normalization of rewards across different prompts connects to concerns in reinforcement learning about reward shaping and the non-stationarity of learning signals.

The framework does make important assumptions that bound its applicability. It assumes that rubric criteria can be generated reliably for the target domain, which requires either access to expert-generated criteria or high-quality reference answers that can serve as proxies for expert supervision. It assumes that an LLM-as-judge can reliably assess each criterion in isolation, which may fail for criteria requiring deep domain expertise or subtle reasoning. And it assumes that the space of good responses can be adequately captured by a finite set of discrete criteria, which may not hold for creative or highly open-ended tasks. These assumptions are not presented as limitations to be overcome but as honest boundaries of the theoretical framework, helping practitioners understand when RaR is likely to succeed and when alternative approaches may be needed.

3. 技术架构 (Technical Architecture)

System Overview: The Rubric-to-Learning Pipeline

The technical architecture of Rubrics as Rewards can be understood as a two-stage pipeline that bridges the gap between expert evaluation principles and automated policy optimization. The first stage is Rubric Generation, where instance-specific evaluation criteria are synthesized for each prompt in the training corpus. The second stage is GRPO Training, where these rubrics are used to score policy rollouts and drive on-policy reinforcement learning. This pipeline is illustrated in Figure 1 of the paper, which shows the flow from a prompt and reference answer through rubric generation to the GRPO loop where the policy model's responses are evaluated against these criteria.

The rubric generation process is itself a sophisticated application of large language models, but crucially, it happens once per prompt before training begins rather than during the reinforcement learning loop. This separation is architecturally important because it means that the expensive process of generating high-quality rubrics is amortized across many training steps. Once generated, rubrics serve as reusable, interpretable supervision that can be applied consistently to new policy rollouts, offering a scalable and transparent alternative to the opaque reward models typically used in RLHF.

Data Flow: From Prompt to Optimized Policy

Tracing the journey of data through the system reveals the careful engineering that makes this pipeline practical. The process begins with a prompt drawn from the training corpus — for medical reasoning, this might be a clinical case description; for science, a conceptual question requiring explanation. This prompt is paired with a reference answer, which serves as a proxy for expert supervision. The rubric generation model (OpenAI's o3-mini for science, GPT-4o for medicine) processes this prompt-reference pair and produces a structured rubric of 7 to 20 items. Each item is a self-contained evaluative statement with a title, a description beginning with its category prefix ("Essential Criteria:", "Important Criteria:", etc.), and a weight. These rubrics are then stored as structured data, ready for repeated use during training.

During the GRPO training loop, the process becomes more dynamic. For each training prompt, the current policy model $π_{θ}$ generates $k = 16$ response rollouts, using a context length of 3,584 tokens and a sampling temperature of 1.0. These 16 responses represent diverse attempts to answer the prompt, capturing the policy's current distribution over possible outputs. Each response is then evaluated against the pre-generated rubric using gpt-4o-mini as the judge model. For the explicit aggregation strategy, the judge independently checks each criterion, producing a binary correctness signal, and the system computes the weighted normalized reward. For the implicit aggregation strategy, the judge receives all criteria and produces a single Likert-style rating on a 1-10 scale, which is normalized to [0, 1]. These rewards are then fed into the GRPO algorithm to compute group advantages and update the policy weights.

Component Interactions: The Judge-Policy Dynamics

The interaction between the judge model and the policy model is the central dynamic of the architecture. Unlike traditional RLHF where a fixed reward model is trained on human preferences, RaR uses a general-purpose LLM as a judge that evaluates responses against explicitly stated criteria. This creates a more interpretable and controllable feedback loop: when a policy learns to improve its score, we can inspect which criteria it is satisfying more often rather than observing an opaque increase in a reward model's output. The judge is not learning a hidden quality model but is applying explicit, human-readable criteria that can be audited and modified.

The choice of GRPO (Group Relative Policy Optimization) as the reinforcement learning algorithm is architecturally significant. GRPO eliminates the need for a separate value function by estimating advantages relative to the average reward within a group of responses generated from the same prompt. This is particularly well-suited to the rubric-based setting because the reward structure — normalized weighted sums or judge ratings — naturally produces comparable scores within a prompt. The relative advantage computation helps reduce variance in the learning signal, which is especially important when using LLM-as-judge evaluations that may exhibit some stochasticity. The training hyperparameters reflect a conservative optimization approach: a learning rate of $5 \times 10^{- 6}$ with a constant schedule and 10% linear warmup, a batch size of 96 (effective), and 300 training steps. The relatively small number of steps and low learning rate suggest that the rubric-based signal is dense enough to drive meaningful improvement without requiring aggressive optimization that might lead to reward hacking.

Key Innovations: Instance-Specificity and Structured Evaluation

The most significant architectural innovation is the insistence on instance-specific rubrics rather than generic, one-size-fits-all criteria. The paper includes a predefined baseline (RaR-Predefined) that uses a fixed set of generic rubrics for all prompts — criteria like "the response contains correct information without factual errors" or "the response is concise and to the point." This baseline performs substantially worse than instance-specific variants, demonstrating that the power of rubric-based supervision comes from criteria that capture the specific requirements and common failure modes of individual prompts. A generic rubric for a medical question cannot know that the specific case requires mentioning a particular contraindication or avoiding a common misdiagnosis; only a prompt-specific rubric can encode this nuance.

The architecture also innovates in how it handles the negative space of evaluation — what the paper calls Pitfall criteria. These are not just missing positive criteria but explicitly identified common errors that should be avoided. In the architecture, Pitfall criteria are phrased in positive form (e.g., "The response avoids mentioning [common misdiagnosis]"), so satisfying them contributes positively to the score, while failing them reduces the reward. This architectural choice turns the reward computation into a more nuanced instrument that can both encourage desired behaviors and discourage specific undesired ones, providing richer shaping of the policy's behavior than simple positive reinforcement alone.

Implementation Choices and Engineering Rationale

Several implementation choices reflect careful engineering trade-offs. The use of a fixed judge model (gpt-4o-mini) across all experiments, rather than allowing the judge to improve alongside the policy, ensures that the reward signal remains stable during training. This is a conservative choice that prevents the non-stationarity problems that can arise when the reward model and policy co-evolve. The 3584-token context length balances the need to accommodate long-form reasoning responses against the computational cost of evaluation. The temperature of 1.0 during rollout generation encourages exploration, producing diverse responses that give the GRPO algorithm a rich group for advantage estimation. The training runs on a single compute node with 8 NVIDIA H100 GPUs, suggesting that the method is computationally accessible by modern standards — the rubric generation is the most expensive step, but it is performed once and reused.

4. 实验评估 (Experimental Evaluation)

Experimental Design and Strategy

The experimental design of the paper is built around a central question: can rubric-based rewards serve as effective training signals for on-policy reinforcement learning in domains without verifiable ground truth? To answer this, the authors construct a comprehensive evaluation that tests RaR across multiple dimensions — different domains (medicine and science), different evaluation formats (rubric-based free-form and multiple-choice), different reward aggregation strategies (explicit and implicit), and different judge model scales. This multi-faceted design is crucial because the paper's central claim is not merely that rubrics improve performance on a specific benchmark, but that they provide a more generalizable and aligned training signal that works across settings.

The experiments systematically compare RaR against both strong baselines and off-the-shelf models. The baselines include Direct-Likert, where an LLM judge scores responses on a 1-10 scale without any rubric guidance; Reference-Likert, where the judge compares the response to a reference answer and scores accordingly; and the base Qwen2.5-7B model plus its instruction-tuned variant. By holding the base model constant across all training methods, the experiments isolate the effect of the reward signal itself, creating a fair comparison of how different supervision strategies shape the same policy's behavior.

Datasets and Evaluation Metrics

The paper's evaluation rests on two carefully chosen domain benchmarks. For medicine, HealthBench provides a rigorous test of clinical reasoning: 5,000 prompts with expert-authored rubrics evaluating responses across multiple axes including clinical accuracy, safety, completeness, and tone. This benchmark is particularly valuable because it captures the nuanced, multi-dimensional nature of medical reasoning where a response can be partially correct yet dangerously incomplete. The authors further curate a subset of 1,000 prompts (HealthBench-1k) for ablation studies, reserving the rest for training. For science, GPQA-Diamond offers a challenging multiple-choice benchmark of graduate-level physics, chemistry, and biology questions, testing whether rubric-induced reasoning skills generalize to a verifiable evaluation format.

Method	HealthBench Overall	GPQA-Diamond Mean Accuracy
Qwen2.5-7B	7.7%	31.7%
Qwen2.5-7B-Instruct	12.5%	31.7%
Direct-Likert	22.7%	34.8%
Reference-Likert	25.5%	35.0%
RaR-Predefined	28.9%	36.5%
RaR-Explicit	31.2%	36.9%
RaR-Implicit	29.7%	37.6%

Table: Performance comparison across evaluation settings. Values for HealthBench are overall scores (percentages); values for GPQA-Diamond are mean accuracy over 10 independent runs with 95% confidence intervals (error bars shown in Figure 2).

The results presented in the table and Figure 2 reveal a clear hierarchy of performance. RaR-Implicit consistently achieves the strongest results, outperforming Direct-Likert with relative improvements of up to 31% on HealthBench and 7% on GPQA-Diamond. RaR-Explicit also surpasses the Likert baselines, though its fixed weighting proves slightly less effective than the implicit strategy's adaptive integration. Both instance-specific rubric variants dramatically outperform the RaR-Predefined baseline, which applies generic rubrics uniformly across all prompts, underscoring that the specificity of criteria to individual tasks is not a cosmetic feature but a fundamental requirement for effective training. The fact that RaR-trained models improve on GPQA-Diamond — a multiple-choice benchmark with verifiable answers, not rubric-based evaluation — is particularly significant, demonstrating that rubric-guided reasoning skills transfer beyond the evaluation format they were trained on.

Baseline Comparisons and Mechanistic Insights

The comparison between RaR-Implicit and Reference-Likert is especially illuminating. Both methods use reference answers to guide the reward signal: Reference-Likert asks the judge to directly compare the response to a reference, while RaR first converts the reference into explicit rubric criteria and then evaluates against those criteria. The fact that RaR-Implicit still outperforms Reference-Likert suggests that the act of converting open-ended answers into explicit criteria yields more effective supervision than direct comparison. One plausible explanation is that rubrics make the evaluation process more transparent and structured, reducing the noise and inconsistency that can arise from asking a judge to make holistic comparisons. The rubric effectively provides the judge with a checklist of what to look for, making the evaluation more systematic and reliable.

Ablation Studies and Robustness Analysis

The ablation studies, summarized in Table 2, probe the contributions of different rubric design elements. Training with only Essential criteria yields a score of 34.9% on HealthBench-1k, while using all criteria (including Important, Optional, and Pitfall) achieves 37.2%. Removing categorical labels (treating all criteria as having equal weight) produces 38.8%, while omitting Pitfall criteria yields 37.2%. These results suggest that the breadth of criteria coverage matters more than the precise weighting scheme or the inclusion of negative criteria — at least for synthetically generated rubrics. The authors note that generating effective Pitfall criteria is inherently difficult because it requires anticipating the model's most common failure modes, a task that often demands human intuition and domain expertise. As a result, synthetic Pitfall criteria may lack the specificity needed to meaningfully penalize undesirable responses.

The judge scale analysis provides another layer of insight. The paper evaluates whether rubric-guided evaluation improves judge effectiveness compared to rubric-free Likert scoring when used for GRPO training. Results across different judge models (from Qwen-7B-Instruct to GPT-4o-mini) show that rubric-based evaluation improves accuracy for every judge size, with the largest gains for smaller models. This finding is practically important because it means that rubric-based rewards can make smaller, more efficient judges viable for training, reducing the computational cost and operational complexity of the RL pipeline.

Statistical Significance and Generalization

The evaluation on GPQA-Diamond is conducted across 10 independent runs with greedy decoding, and the reported accuracy includes 95% confidence intervals to account for run-to-run variance. This rigorous statistical reporting, combined with the cross-domain evaluation (medicine and science), supports the paper's claim that rubric-based training yields a stable and generalizable signal. The consistency of gains across both rubric-based and multiple-choice settings suggests that the improvement is not an artifact of evaluation-method alignment but reflects genuine enhancement of the policy's reasoning capabilities.

5. 案例研究 (Case Studies)

Medical Case: Sodium Bicarbonate Administration

To understand how rubric-based rewards work in practice, consider the medical example from the RaR-Medicine dataset presented in the appendix. The prompt asks: "A 50-year-old male patient weighs 65 kg with a pH of 7.05, PCO2 of 15 mmHg, HCO3 of 5 mEq/L, and a base deficit of -40 mEq/L. How much sodium bicarbonate should be administered in the first 4 hours to correct his metabolic acidosis?" The reference answer explains the calculation, the standard formula (Base Deficit × Body Weight × 0.3), the full computed value of 780 mEq, and the critical clinical principle that only partial correction (about 150 mEq over 4 hours) should be administered to avoid complications from overcorrection.

From this reference answer, the rubric generation system produces seven criteria with carefully assigned weights. The Bicarbonate Calculation criterion (weight 5, Essential) requires that the response correctly identify and apply the formula. The Safe Dosing Recommendation criterion (weight 5, Essential) demands a clear recommendation of about 150 mEq over the first 4 hours. The Partial Correction Justification criterion (weight 4, Important) asks for an explanation of why only partial correction is administered. The Step-by-Step Calculation criterion (weight 3, Important) requires detailing the calculation steps. The Patient Data Accuracy criterion (weight 3, Important) checks that the response incorporates the patient's weight and critical lab values. The Base Deficit Interpretation criterion (weight 2, Optional) allows mentioning that the deficit indicates severe metabolic acidosis. Finally, the Avoid Overcorrection Risk criterion (weight -1, Pitfall) checks that the response does not neglect the risks of rapid overcorrection.

This example illuminates several key aspects of the RaR framework. First, the rubric decomposes a complex medical reasoning task into distinct, independently assessable components. A policy could get the calculation right but fail to recommend safe dosing, or could recommend safe dosing but without explaining the reasoning. Each of these partial successes is captured by the rubric, providing a dense reward signal that guides the policy toward the complete, expert-level response. Second, the weighting reflects genuine medical priorities: getting the calculation right and recommending safe dosing are both Essential (weight 5), while explaining the reasoning behind partial correction is Important but not absolutely critical (weight 4). This weighting mirrors how a clinician might evaluate a resident's response — some errors are unacceptable, others are significant but not catastrophic. Third, the Pitfall criterion captures domain-specific safety knowledge that a generic evaluation would miss. The risk of overcorrection is a specific hazard of this particular clinical scenario, and the rubric explicitly checks for awareness of it.

Science Case: Solubility and "Like Dissolves Like"

The science example from the RaR-Science dataset asks about the solubility of boric acid in ethanol versus benzene, requiring application of the "like dissolves like" principle and discussion of Ksp values and immulcifiers. The generated rubric includes Essential criteria for correctly identifying that boric acid is more soluble in ethanol and for explaining the polarity principle. Important criteria cover the role of Ksp values (with the subtlety that boric acid is covalent, not ionic), how immulcifiers modify solubility, and analysis of inherent chemical properties. A Pitfall criterion (weight -1) checks that the response does not incorrectly assume Ksp values for ionic compounds directly determine the solubility of a covalent acid like boric acid.

This example reveals how rubric generation handles the nuances of scientific reasoning. The Ksp criterion is particularly subtle: it requires the response to acknowledge the typical relevance of Ksp values while recognizing that boric acid's covalent nature means the standard ionic solubility product framework does not directly apply. The Pitfall criterion captures a specific conceptual error that students commonly make — conflating ionic and covalent solubility principles. These criteria are not generic statements about "good scientific reasoning" but are precisely tailored to the conceptual challenges of this specific question. When used as rewards, they provide the policy with targeted guidance: satisfy the Essential criteria for basic credit, demonstrate deeper understanding through the Important criteria, and avoid the specific Pitfall to prevent losing points. This granular, task-specific feedback is precisely what Likert-based scoring cannot provide.

6. 综合价值与局限 (Synthesis — Value and Limitations)

Theoretical Significance

Rubrics as Rewards makes a meaningful theoretical contribution by reframing the problem of reward design for non-verifiable domains. The prevailing discourse has often framed this as a choice between scalable but simple signals (Likert scales, verifiable checks) and rich but expensive signals (human preferences). RaR introduces a third path: structured, interpretable, and automatable signals that preserve the multi-dimensional nature of expert judgment without requiring massive human annotation. The formalization showing that rubric-based rewards subsume verifiable rewards as a special case (Remark 1) is particularly elegant, positioning the framework not as an alternative to RLVR but as a natural generalization that extends its applicability to the vast majority of real-world tasks where binary correctness is unavailable.

The work also provides conceptual tools for thinking about evaluation itself. The four-category taxonomy — Essential, Important, Optional, and Pitfall — offers a vocabulary for discussing response quality that is more nuanced than "good/bad" or "better/worse." This taxonomy can be valuable beyond reinforcement learning, informing how we design benchmarks, how we collect human feedback, and how we think about model alignment more broadly. The insight that explicitly structuring evaluation criteria improves not only the evaluation itself but also the training signal derived from it suggests a deeper principle: the act of articulating what we value in a structured form may be as important as the values themselves.

Practical Impact and Strengths

The practical strengths of this work are substantial. The release of RaR-Medicine and RaR-Science datasets provides the community with valuable resources for studying structured rewards in expert domains. The demonstration that rubric-based training improves performance across both rubric-based and multiple-choice evaluations suggests that the skills induced are genuine reasoning improvements rather than evaluation-specific hacking. The finding that rubric guidance especially benefits smaller judges has immediate practical implications: it means that high-quality RL training can be achieved with more efficient, accessible models rather than requiring the largest proprietary systems.

The paper's methodological rigor is also a strength. The controlled experiments hold the base model constant across methods, isolating the effect of the reward signal. The use of independent runs with confidence intervals provides statistical robustness. The ablation studies probe the contributions of specific design choices rather than presenting the final method as a monolithic black box. The inclusion of both a predefined rubric baseline and a synthetic-rubric-without-reference baseline helps distinguish the contributions of instance-specificity and expert grounding from the general idea of using rubrics.

Honest Limitations

Several limitations deserve honest acknowledgment. First, the work focuses on medicine and science domains, which are amenable to structured evaluation but may not generalize to more creative or open-ended tasks like dialogue, storytelling, or tool use. The assumption that good responses can be captured by discrete criteria may fail in domains where quality is inherently holistic and subjective. Second, the paper evaluates only two reward aggregation strategies, which represent extremes on the control-flexibility spectrum. The authors acknowledge that more sophisticated approaches — such as learning continuous weights for each criterion or dynamically adjusting weights over training — could improve performance further. Third, the reliance on off-the-shelf LLMs as judges, while practical and reproducible, means that the ultimate quality of the reward signal is bounded by the judge's capabilities. In specialized domains, even state-of-the-art LLMs may lack the expertise to reliably evaluate certain criteria, and the paper does not fully address how to validate or calibrate judge accuracy against true expert judgment.

A deeper limitation concerns the scalability of rubric generation. While the paper demonstrates that synthetic rubrics generated from reference answers can be effective, the quality of these rubrics depends on the quality of the reference answers and the capabilities of the generation model. The ablation showing that synthetic rubrics without reference guidance perform notably worse (Table 1) highlights that expert grounding is not easily replaced. For domains without high-quality reference corpora, the rubric generation step may become a bottleneck, potentially requiring costly human annotation to achieve the quality levels demonstrated in the paper. The question of whether rubric generation can be further automated or whether it will remain a domain-expert-dependent process is left as an important open challenge.

Broader Implications

Looking at the broader landscape, this work connects to a growing trend toward structured supervision in AI alignment. Recent concurrent work exploring checklists and principled rubric criteria for preference tuning and LLM safety reflects a shared recognition that opaque, holistic signals are insufficient for reliable training. RaR contributes to this trend by demonstrating that structured criteria can be used directly as reward signals, not just for evaluation or preference conditioning. The paper also speaks to the broader challenge of aligning language models with human expertise in domains where expertise is scarce and expensive. By turning expert evaluation principles into automatable training signals, RaR offers a pathway for scaling expert-level supervision beyond what human annotation budgets could support directly.

7. 延伸阅读与思考 (Further Reading and Reflection)

Prior Work and Intellectual Foundations

RaR builds upon several important lines of prior work. The RLVR tradition, exemplified by DeepSeek-R1 (Guo et al., 2025) and the General-Reasoner framework (Ma et al., 2025), established that verifiable rewards can elicit complex reasoning in mathematics and code. The paper's formalization of rubric-based rewards as generalizing RLVR directly acknowledges this debt. From the RLHF side, the work inherits concerns about reward hacking and preference model limitations documented by Ouyang et al. (2022) and others. The paper's critique of preference-based methods echoes findings that reward models tend to overfit to superficial artifacts (Singhal et al., 2023; Chen et al., 2024).

In evaluation methodology, the paper draws on the growing recognition that instance-specific rubrics improve evaluation reliability. Pathak et al. (2025) showed that rubric-prompted LLM graders are more accurate than question-agnostic checklists, while HealthBench (Arora et al., 2025) scaled this approach to medicine with clinician-authored criteria. The paper extends these evaluation insights into the training domain, demonstrating that rubrics are not just better evaluators but better teachers. Recent concurrent work on checklists for preference tuning (Viswanathan et al., 2025) and constitutionally decomposed QA for alignment (Dineen et al., 2025) shares the philosophical commitment to structured supervision, though these approaches primarily use rubrics for conditioning preference data rather than as direct reward signals.

Alternative methods for non-verifiable reward design offer different philosophical approaches. Process supervision (Lightman et al., 2023) provides denser guidance through step-level labels, but requires costly human annotation of reasoning chains. Generative reward models like ThinkPRM (Khalifa et al., 2025) and J1 (Whitehouse et al., 2025) learn to evaluate outputs, maintaining flexibility but sacrificing the interpretability and control of explicit criteria. Med-RA-VR (Zhang et al., 2025) applies verifiable-reward methods to medical multiple-choice questions, demonstrating that some healthcare tasks can be addressed with simpler signals. Cross-domain RLVR efforts (Su et al., 2025) expand the applicability of verifiable rewards to physics, finance, and policy, but still rely on domains where some form of automatic verification is possible. RaR occupies a distinct position: it requires more upfront structure than Likert-based methods but provides richer, more interpretable signals; it offers less dense supervision than process-level labels but avoids their annotation cost; it trades some flexibility for generative reward models for greater transparency and control.

Future Directions and Open Problems

The paper's limitations section points to several promising research directions. The exploration of learned or dynamic weighting strategies — where the model learns to adjust criterion weights based on training progress or response characteristics — could combine the interpretability of explicit aggregation with the adaptability of implicit aggregation. Extending rubric-based methods to dialogue, tool use, and agentic tasks remains an important direction for validating the framework's generality. Developing specialized evaluators or generative reward models with stronger reasoning capabilities could improve the reliability of criterion evaluation in highly specialized domains. Perhaps most fundamentally, the question of how to generate high-quality rubrics at scale with minimal expert supervision remains open. The paper shows that synthetic generation from reference answers works, but also that reference-free synthesis falls short. Advances in automated rubric generation could dramatically expand the applicability of RaR to domains where expert reference corpora are unavailable.

Personal Reflection

What makes this work most thought-provoking is its reframing of the alignment problem. Rather than asking "how do we get models to produce outputs that humans prefer?" — a question that often leads to opaque preference modeling and reward hacking — RaR asks "how do we articulate what makes a good response in a form that can be directly taught?" This pedagogical perspective is refreshing. It treats alignment not as a statistical learning problem over human judgments but as a knowledge transfer problem where expert understanding is structured into explicit criteria and communicated to the model. The most surprising result is that this structured articulation helps not just the model being trained but also the judge evaluating it: smaller models become more reliable judges when given explicit criteria. This suggests that the difficulty of evaluation may not be solely a capacity problem but also a specification problem — that we often ask too much of judges by demanding holistic quality assessment without providing the structure needed to perform it well. The implication is that making our values explicit and structured may be a key not only to better training but also to more reliable evaluation, a dual benefit that makes the rubric-based approach particularly compelling.

Topics: