Enabling Scalable Oversight via Self-Evolving Critic

Let $\mathcal{P}$ denote a set of mathematical problems, where each problem $p\in\mathcal{P}$ is paired with a ground truth answer $a_{p}$. For each problem $p$, we collect a set of solutions $\mathcal{S}_{p}=\{s_{1},s_{2},...,s_{n}\}$ from different models, where each solution $s_{i}$ consists of:

• •

  A sequence of reasoning steps $\mathbf{r}_{i}=[r_{i}^{1},r_{i}^{2},...,r_{i}^{k_{i}}]$, where $k_{i}$ is the number of steps

• •

  A final answer $a_{s_{i}}$

A critique $c$ for a solution $s$ is defined as a tuple $c=(\mathbf{e},l,t)$, where:

• •

  $\mathbf{e}=[e_{1},e_{2},...,e_{k}]$ is a sequence of step-wise critiques, where each $e_{i}$ corresponds to the analysis of step $r^{i}$

• •

  $l=(y,j)$ is the conclusion, where $y\in\{0,1\}$ indicates solution correctness and $j\in\{-1\}\cup\mathbb{N}$ denotes the first error step ($j=-1$ means no error)

• •

  $t$ is the correction, consisting of a sequence of corrected steps and a final answer $a_{t}$

Our objective is to learn a critique function $f_{\theta}:\mathcal{P}\times\mathcal{S}\rightarrow\mathcal{C}$ that maps a problem $p$ and a solution $s$ to an effective critique $c$, where $\theta$ represents the parameters of a language model.

To achieve this objective, we propose SCRIT (Self-evolving CRITic), a framework that systematically leverages the shared mathematical understanding across different solutions to enable truly self-evolving critique abilities. As illustrated in Figure 3, SCRIT operates through a complete self-evolving cycle: it takes a problem and solutions as input, generates critiques through analyzing reference solutions, validates their quality, and uses the validated critiques for self-training. This forms a complete self-evolving cycle without any external supervision.

Dataset The first step in our framework is to collect a diverse set of solutions. We build our collection process on the NuminaMath dataset (LI et al., 2024), a large-scale mathematical problem dataset covering various topics from elementary mathematics to competition-level problems. To ensure data quality, we develop a robust pipeline to compute reliable ground truth answers (detailed in Appendix A), resulting in 452K validated problem-answer pairs.

Solution Generation Models To enhance the diversity of generated data, we gather solutions from seven models: deepseek-math-7b-rl (Shao et al., 2024), mathstral-7B-v0.1 (Mistral-AI, 2024a), Mistral-Large-Instruct-2411 (Mistral-AI, 2024b), DeepSeek-V2-Chat-0628 (DeepSeek-AI, 2024), Qwen2.5-Math-7B-Instruct (Qwen-Team, 2024), Qwen2.5-Math-1.5B-Instruct (Qwen-Team, 2024), and Qwen2-Math-1.5B-Instruct (Qwen-Team, 2024). It is important to note that the outputs from these models serve as inputs for the critic model, with no external supervision involved in the critic’s learning process.

Data Filtering For each problem $p\in\mathcal{P}$, we classify its collected solutions into correct solutions $\mathcal{S}_{p}^{+}$ and incorrect solutions $\mathcal{S}_{p}^{-}$ based on answer correctness. A crucial filtering criterion in our framework is that each problem must have at least one correct solution and one incorrect solution to enable later contrastive critic. Formally, we only retain problems that satisfy:

A key challenge in enabling effective critique generation is how to ensure the model can identify and correct errors in complex mathematical reasoning, particularly when the problem difficulty approaches or exceeds the model’s current capabilities. Our preliminary experiments reveal that the model often exhibits “rubber-stamping behavior” - blindly approving incorrect steps without genuine understanding of the mathematical concepts involved, as illustrated in Figures 2 and 8. This also aligns with findings in (Huang et al., 2023).

We initially explored two straightforward approaches: (1) Direct Critic (Zheng et al., 2024a), where a language model, such as Qwen2.5-72B-Instruct, directly critiques a solution; and (2) Bug-Injection Critic (McAleese et al., 2024), a two-stage approach of first injecting errors into a correct solution and then ask the LLM to critic and correct it. However, both approaches showed limited effectiveness (detailed in Section 5.2).

To address these issues, we develop a new technique called Contrastive Critic. Our key insight stems from a fundamental property of mathematical reasoning: while problems may have multiple valid solutions, they inherently share the same underlying mathematical concepts and key solving strategies. By explicitly providing a correct reference solution during critique generation, we enable the model to first understand these core mathematical concepts and solving strategies, then leverage this understanding to perform step-by-step critique of the target solution. This approach addresses the rubber-stamping issue by grounding the critique process in concrete mathematical understanding derived from correct references.

For each problem $p\in\mathcal{P}_{valid}$, we generate critiques through two types of solution pairings:

Correct-Incorrect Pairs. For each $s^{-}\in\mathcal{S}_{p}^{-}$, randomly select $s_{ref}\in\mathcal{S}_{p}^{+}$ and generate $c=f_{\theta}(p,s^{-}|s_{ref})$.

Correct-Correct Pairs. For each $s^{+}\in\mathcal{S}_{p}^{+}$, randomly select $s_{ref}\in\mathcal{S}_{p}^{+}\setminus\{s^{+}\}$ and generate $c=f_{\theta}(p,s^{+}|s_{ref})$.

Both pairing strategies promote diversity in the generated critiques, a factor we empirically validate for effectiveness in subsequent experiments. The self-critic function $f_{\theta}$ (prompt template in Appendix B) decomposes critique generation into four sequential stages:

Stage 1: Reference Analysis. Generate reference analysis $r=f_{\theta}^{r}(p,s_{ref})$ that captures key mathematical concepts, critical solution steps, and potential pitfalls.

Stage 2: Step-wise Critique. For each step $s^{i}$, generate critique $\mathbf{e}=[f_{\theta}^{e}(p,s^{i},r)]_{i=1}^{k}$ by verifying mathematical and logical validity using $r$, identifying error type and suggesting corrections if found, and stopping analysis upon first error detection.

Stage 3: Conclusion. Generate conclusion $l=f_{\theta}^{l}(p,s,\mathbf{e})$ where $l=(y,j)$ indicates solution correctness ($y\in\{0,1\}$) and first error step ($j\in\{-1\}\cup\mathbb{N}$).

Stage 4: Correction. Generate correction $t=f_{\theta}^{t}(p,s,\mathbf{e})$ by following original approach up to error step (if any), then complete with proper correction.

With self-generated critique data, we apply post-validation techniques to further enhance the quality of generated outputs. This process specifically filters out low-quality cases where the model blindly approves all intermediate steps, only to suddenly reject the final answer upon detecting a discrepancy (see Appendix D).

To address these challenges, we employ direct validation on the correction part of the critique. Formally, we have that:

where $t$ is the correction part of critique $c$, and $g_{\theta}^{l}$ (prompt template in Appendix B) denotes direct critic’s conclusion generation function. A value of $v_{\theta}(c)=1$ means that model confirms the critique $c$ as effective, while $v_{\theta}(c)=0$ indicates it is ineffective. This validation mechanism ensures that only critiques whose corrections can be independently verified as correct are used for self-training.

Let $\mathcal{V}$ denote the set of validated solution-critique pairs across all problems:

For each validated triplet $(p,s,c)\in\mathcal{V}$, we construct training pairs with input $g_{\theta}(p,s)$ and target $(e,l,t)$ from $c$. Note that we exclude the reference analysis $r$ from the target as it is specific to contrastive critic generation.

We fine-tune the base model Qwen2.5-72B-Instruct to minimize the following loss function:

Note that $g_{\theta}(p,s)$ is gradient-stopped during the optimization process. This training process enables genuine self-evolution of critique abilities, as the model learns from its own generated and validated critiques without any external supervision.

We present detailed statistics of data flow through each component of our framework.

Solution Collection We start with 452K problem-answer pairs from our own NuminaMath dataset (see Appendix A). For solution generation, we employ 7 models of varying capabilities as described in Section 3.2. Each model generates one solution per problem, with solutions classified as correct or incorrect based on their final answers using Qwen2.5-72B-Instruct (detailed in Appendix H). Then we apply two filtering criteria: (a) Each problem must have at least one correct and one incorrect solution to enable contrastive learning; (b) Solutions from each model are capped at 50K for both correct and incorrect categories. After filtering, we obtain 665K problem-solution pairs, evenly split between good solutions (332K) and bad solutions (332K).

Self-Critic & Self-Validation To analyze the self-critic and self-validation step, we track the data flow from the initial 665K problem-solution pairs through these steps. Out of these pairs, 342K (51.4%) successfully pass the self-critic and self-validation step, yielding high-quality problem-solution-critique triplets. Figure 4 presents a detailed analysis of this filtering process across different dimensions, revealing interesting patterns in validation rates.

• •

  Domain Complexity: Validation rates decrease systematically from elementary domains (GSM8K: 91.8%, ORCA Math: 77.6%) to competition-level problems (Olympiads: 27.1%)

• •

  Problem Difficulty: The validation rate shows a clear negative correlation with the number of unique answers, dropping from 91.7% for single-answer problems to 15.5% for problems with seven distinct answers

• •

  Solution Model Impact: Solution generation models show relatively consistent validation rates (48.9% to 57.4%), suggesting that our self-validation process is more sensitive to problem difficulty than to the source model

Analysis of error positions in critiqued solutions (see Figure 19) reveals that a majority of errors occur in earlier steps, aligning well with human-labeled error distributions in ProcessBench (Zheng et al., 2024a). This correlation suggests that our self-critic framework successfully captures human-like error identification patterns.

Self-Training For the self-training, we maintain a balanced 1:1 ratio between correct and incorrect solutions, resulting in 170K training examples. These balanced training data are used to fine-tune Qwen2.5-72B-Instruct following Section 3.5 (complete training details in Appendix I).

We present two complementary evaluation protocols to assess different aspects of critique capabilities:

Critic and Correct The first protocol evaluates a model’s ability to critic and correct a given solution, following the assumption (Zheng et al., 2024b) that truly effective critiques should be able to guide the correction of errors and lead to correct answers. We conduct experiments on RealCritic, an internal benchmark we developed and plan to release publicly, which systematically spans 8 datasets (GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), ARC-C (Clark et al., 2018), College Math (Tang et al., 2024), GPQA (Rein et al., 2023), Minerva Math (Lewkowycz et al., 2022), MMLU-STEM (Hendrycks et al., 2020), OlympiadBench (He et al., 2024)) across 3 scenarios: critic on deliberately incorrect solutions, balanced solutions, and the base model’s self-generated solutions (i.e., Qwen2.5-72B-Instruct’s own solutions).

Critic and Correct with Error Identification The second protocol adds a stricter requirement: models must not only provide accurate correction but also tell the first step where an error occurs. We evaluate on PRM800K (Lightman et al., 2023)¹¹1https://github.com/openai/prm800k/blob/main/prm800k/data/phase2_test.jsonl and ProcessBench (Zheng et al., 2024a), two benchmarks with human-labeled error steps on solutions from advanced models (GPT-4, LLaMA, Qwen2.5 series). ProcessBench provides an evaluation suite across 4 datasets: GSM8K, MATH, OlympiadBench, and Omni-Math (Gao et al., 2024). Following ProcessBench’s methodology, we use the F1 score of accuracies on incorrect and correct samples as our metric, with two adaptations to ensure critique effectiveness (See Appendix F).

Baselines Since our goal is to improve Qwen2.5-72B-Instruct’s critique ability through self-evolution, we use the original Qwen2.5-72B-Instruct as our primary baseline. Additionally, we compare against o1-mini (OpenAI, 2024), currently one of the most capable models in terms of critique ability (Zheng et al., 2024a), to benchmark our approach against the state-of-the-art.

Critic and Correct Table 1 presents results across three increasingly challenging scenarios. In critiquing deliberately incorrect solutions, SCRIT achieves substantial improvements over the base Qwen2.5-72B-Instruct model, raising the average performance from 39.7% to 50.0%. For balanced solutions, SCRIT maintains its advantage with an average improvement of 4.4%, despite the increased difficulty of distinguishing correct from incorrect solutions. Most impressively, when critiquing Qwen2.5-72B-Instruct’s own solutions, SCRIT still manages to improve upon the base model (62.9% vs 61.7%), demonstrating its ability to identify and correct errors in solutions generated by its own base model. Across all scenarios, SCRIT’s performance approaches that of o1-mini.

Critic and Correct with Error Identification As shown in Table 2, SCRIT also demonstrates strong capabilities in error identification, achieving consistent improvements across all datasets in both PRM800K and ProcessBench. The average F1 score improves from 37.8% to 45.0%, with particularly strong gains on mathematical reasoning tasks (GSM8K: +11.3%, MATH: +9.1%). While there remains a gap with o1-mini, SCRIT’s improvements are notable given its self-evolving nature without reliance on external supervision.

We investigate how SCRIT’s performance scales with both training data size and model size (see Figures 5 and 6).

Data Size Scaling For data scaling experiments, we train SCRIT with different amounts of training examples, ranging from 10K to 170K. Both CC-Acc and EI-F1 show consistent improvements with increased training data. The CC-Acc and EI-F1 improves from 53.0% to 58.3%, with the steepest gains in the early stage (0-20K examples) and continued but more gradual improvements afterwards. Similarly, EI-F1 increases from 37.8% to 45.1%, demonstrating that SCRIT can effectively leverage more training data to evolve its critique capabilities.

Model Size Scaling We evaluate SCRIT across three model sizes of Qwen2.5: 1.5B, 7B, and 72B. Both metrics show strong positive correlation with model scale. The CC-Acc increases substantially from 41.7% (1.5B) to 51.2% (7B) and further to 58.3% (72B). The improvement is more pronounced for EI-F1, where metric rises from 12.5% to 29.9% and then to 45.1%, suggesting that larger models are particularly better at error identification. While we acknowledge that fine-tuning smaller models with data generated by Qwen2.5-72B-Instruct bears similarity to distillation from stronger AI supervision, this experiment primarily serves to investigate whether the SCRIT data benefits model size scaling.

To identify the most effective critic mechanism for our self-evolving framework, we conduct strictly controlled experiments comparing three different critic approaches described in Section 3.3 using identical sets of problems and solutions.

Our experiments in Figure 5 reveal several key findings. First, Contrastive Critic shows strong performance from the early stages across both metrics: with just 10K training examples, it achieves 56.8% CC-Acc and 40.2% EI-F1, outperforming both Direct Critic and Bug-Injection Critic. More importantly, as training data increases to 170K examples, Contrastive Critic continues to show positive scaling behavior, reaching 58.3% CC-Acc and 45.1% EI-F1. In contrast, Direct Critic quickly plateaus at around 55.1% CC-Acc and 38.7% EI-F1, while Bug-Injection Critic exhibits performance degradation in CC-Acc (dropping to 49.0%) and unstable performance in EI-F1).

Through case studies (detailed in Appendices C and E), we identify the key mechanisms behind these performance differences. Direct Critic often falls into superficial critiquing, tending to blindly agree with solutions without deep understanding. Contrastive Critic avoids this pitfall by first analyzing reference solutions, enabling the model to develop a deeper understanding of the underlying mathematical concepts and solution strategies before attempting critique. While Bug-Injection Critic has the theoretical advantage of known error descriptions, our analysis reveals that model-injected bugs tend to be simplistic and repetitive, predominantly focusing on basic arithmetic errors and variable confusions, limiting its effectiveness in real-world scenarios where errors are more diverse and subtle.

These comprehensive results validate our choice of Contrastive Critic for the SCRIT pipeline, as it not only demonstrates superior initial performance but also shows stronger potential for continued improvement with increased training data.

To assess the necessity of self-validation in SCRIT, we conduct controlled experiments by removing the self-validation component while keeping all other settings identical. The results in Table 3 show clear performance degradation across both evaluation metrics: the CC-Acc drops by 0.8%, and more significantly, the EI-F1 decreases by 3.0%. Case analysis (see Appendix D) shows that the self-critic may still generate low-quality critiques, often blindly approving all intermediate steps only to suddenly claim ”the final step is incorrect” when encountering answer discrepancies. By incorporating self-validation, we are able to further enhance the quality of data for self-training.

To investigate the importance of problem domain diversity, we conduct controlled experiments by restricting the training data to only GSM8K and MATH domains, while keeping other settings unchanged. This represents a significant reduction in domain coverage compared to our full setting which spans 9 sources ranging from elementary to competition-level mathematics.

The results in Table 3 demonstrate the value of domain diversity: when training with limited domains, the CC-Acc drops by 1.4% and the EI-F1 decreases by 1.4%. It suggests that exposure to diverse problem-solving patterns and error types is crucial for developing robust critique abilities.

To understand the impact of problem difficulty, we conduct experiments by selecting training examples based on the number of unique answers generated across solution models - a proxy for problem complexity. We compare two settings: training with problems that have more unique answers (indicating higher complexity) versus those with fewer unique answers (indicating lower complexity).

Interestingly, training with less complex problems leads to better performance in EI-F1 in Table 3. This result suggests that SCRIT can generate more effective critiques on simpler problems, possibly because the mathematical concepts and solution strategies in these problems are more structured and well-defined, enabling the model to develop more precise and reliable critique patterns.

This finding leaves space for future work: how to optimally select training examples based on difficulty levels in a self-evolving framework. While our current approach uses all available data, a more sophisticated curriculum that gradually increases problem complexity might lead to more effective self-evolution.

To study whether critiquing solutions from different models affects SCRIT’s performance, we conduct controlled experiments by restricting the solutions being critiqued to those from a single model while keeping other settings identical. Our results in Table 3 show that the source model of solutions has limited impact on SCRIT’s final performance.

Since solution generation models only provide the solutions for constructing contrastive critique pairs and do not directly participate in improving critique effectiveness, their individual capabilities have less influence on the final performance. What matters more is how to construct diverse and informative contrastive pairs that help the model learn effective critique strategies, regardless of the solution models.

Finally, we investigate the impact of good-to-bad solution ratio in the training data. Training with a higher proportion of bad solutions (0.25:0.75) shows significantly better performance than using more good solutions (0.75:0.25). As shown in Table 3, using more good solutions results in performance degradation across both evaluation metrics. This suggests that exposure to more bad solutions helps SCRIT develop stronger error identification capabilities, likely because it provides more diverse examples of mathematical mistakes and their corresponding corrections. More importantly, analyzing incorrect solutions forces the model to actively engage in error detection and correction, rather than simply validating correct steps. This finding tells us that while maintaining some balance is important, slightly favoring incorrect solutions may be a better choice for training effective critique models.