No GPT-4, No Problem: Meet the ‘Tiny Self Improving’ AI That Conquers Math Olympiads All by Itself

In the world of AI, large language models (LLMs) have often dominated headlines for their striking performance on complex tasks. Yet a new approach suggests that smaller models—even those with just a few billion parameters—can equal or outperform these giants in specialized domains like math reasoning. Crucially, they can do this without any help from a more powerful “teacher” model.

This surprising result comes from the paper from Microsoft research “rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking.” Below, we’ll explore how the process works, why it allows smaller models to self-improve, and what it might mean for the future of AI development.

Why Small Models Traditionally Rely on “Teacher” Models

Before diving into “rStar-Math,” it helps to understand why smaller models have historically required larger ones to guide them:

1. Knowledge Distillation: In this standard procedure, a large “teacher” model with superior reasoning capabilities generates training data (or “signals”) to steer a “student” model. Over time, the smaller student picks up the teacher’s knowledge, even if it can’t match the teacher’s original size or raw power.
2. Human-Labeled Data: Alternatively, smaller models sometimes rely on huge, carefully curated datasets—often annotated by humans or bigger AI systems—to learn difficult tasks.

Both of these approaches require an external source of expertise or vast resources. So, if smaller models want to become proficient at advanced topics on their own, a different strategy is needed.

The Self-Evolution Breakthrough

“rStar-Math” demonstrates how small language models (SLMs) can bootstrap themselves to high performance—with no high-powered teacher model in the mix. Here’s a high-level look at how it works:

1. Monte Carlo Tree Search (MCTS) for Exploration The model doesn’t just generate a single solution path to a math problem. Instead, it explores many possible step-by-step solution paths through a tree search process.
2. Preference-Based Reward, Not Final-Answer-Only Rather than waiting until the final answer to decide if the entire solution path is correct, the method uses a Process Preference Model (PPM) to judge each intermediate step. Good steps get positive signals, bad steps get negative ones.
3. Iterative Refinement Over multiple rounds—called “self-evolution”—the small model iterates this search-and-score cycle, using what it learned in the previous round to generate better solutions. This repetition gradually filters out wrong or weak problem-solving paths while reinforcing promising ones.

Why It Works Without a Bigger Model

The secret sauce is in how the method generates and evaluates training data:

• Code-Augmented Chain of Thought: The small model includes Python code in its step-by-step reasoning. If the code fails to run properly or yields inconsistent results, that step is marked as likely flawed. This prevents the model from absorbing incorrect intermediate logic.
• Self-Generated Data, Verified Internally: The small model doesn’t rely on a bigger teacher to produce “gold-standard” solutions. Instead, it creates multiple candidate solutions for each math problem, evaluates them using built-in checks (like running Python code and applying a preference model), and trains on the best ones it found itself.
• Multi-Round Self-Evolution: At each iteration, the small model improves enough to tackle a broader set of problems or handle them more accurately. By the time you reach the final round, the cumulative effect of these improvements has brought the small model’s math skill to levels traditionally associated with much larger systems.

Put simply, the model’s own internal feedback loops act as a stand-in for what a giant teacher model would usually provide. It refines its steps over repeated search attempts, using algorithmic checks (like code execution and preference-based scoring) to separate sound logic from errors.

Inside the Four-Round Process

The paper describes this gradual refinement as a four-round self-evolution:

Round 1: Bootstrap

• A baseline policy model (which could be a reasonably capable model, but not necessarily a giant) performs Monte Carlo Tree Search (MCTS) on a large set of math questions.
• The system keeps only high-value reasoning paths (where code executes and the final answer matches the problem’s solution).
• The small model is then fine-tuned on these “self-verified” solutions—rather than on teacher-generated ones.

Round 2: Building a Stronger Reward Model

• With more accurate Q-values (scores reflecting how good each step was) from the first round, the team trains a preference model that is better at distinguishing correct from incorrect steps.
• They generate another batch of solutions—this time with more thorough MCTS rollouts—leading to more reliable data.

Round 3: Advanced MCTS with Preference Guidance

• Now guided by an improved reward model, the small LLM can more efficiently find and retain correct solution paths, even for tougher problems.
• This upgraded data set goes back into fine-tuning the small model again, raising its baseline abilities.

Round 4: Tackling the Hardest Problems

• The system applies extra computational effort—like 64 or 128 MCTS rollouts—on particularly tough math challenges (e.g., Olympiad-level).
• With enough expansions and consistent feedback from the preference model, the small model breaks into truly state-of-the-art math performance territory.
• By the end of Round 4, the small model has effectively taught itself from a large but messy pool of questions, generating high-quality solution paths internally and discarding faulty ones.

Monte Carlo Tree Search + Preferences = True Autonomy

1. Searching for Solutions Like a Human

Human mathematicians don’t just leap to an answer in one go; they follow multiple potential lines of reasoning, pruning dead ends and refining promising ideas. MCTS replicates this approach, branching out along many partial solutions and homing in on the best path.

2. Reward Without an “All-Knowing” Teacher

The Process Preference Model (PPM) ranks each individual step as either good or bad by comparing it to other potential steps. As a result, the small model doesn’t need a massive LLM (like GPT-4) to generate or verify solutions; its own code checks and preference structure provide the necessary feedback.

3. No Manual Step-by-Step Labeling

A final challenge that used to require big models or large-scale human efforts was labeling each partial step in a math solution with a correct/incorrect judgment. “rStar-Math” avoids that by letting MCTS tag the steps autonomously via code execution results, final-answer checks, and the preference model’s scores.

Results That Rival Much Bigger Models

When tested on benchmarks like MATH (a recognized set of math challenges), the final “rStar-Math” system scores as high as—and sometimes beats—larger LLMs that have been meticulously fine-tuned or guided by top-tier teachers. For instance:

• MATH Benchmark: Accuracy can climb to around 90%, surpassing certain large LLMs that used more traditional methods.
• AIME & Olympiad Problems: With advanced reasoning steps, the small model solves a considerable portion of Olympiad-level questions—again, a feat often reserved for bigger LLMs.

This performance starkly underlines the power of the small model’s internal, iterative improvement, proving that a well-structured feedback loop can substitute for external expert “instruction.”

Why It Matters

1. Lower Resource Footprint Training or running multi-hundred-billion-parameter models is expensive and environmentally taxing. A smaller model that can independently improve opens the door to more sustainable, cost-effective AI.
2. Broader Access to AI Smaller models that match big-model performance can be adopted by more organizations, labs, and even individuals, helping to democratize state-of-the-art capabilities in math and beyond.
3. Beyond Math Although this paper focuses on mathematical reasoning, the framework could be extended to code generation, scientific problem-solving, or any domain with a reliable way to verify correctness. The principle—search multiple paths, filter by a robust preference system, and iteratively refine—is broadly applicable.
4. A Path Toward Autonomous Learning The discovery that a model can bootstrap its own improvements is a step toward AI systems that teach themselves new skills without constant human curation or massive teacher-model “downloads.” This raises thrilling (and sometimes concerning) possibilities for how AI might evolve in the near future.

Conclusion

The central insight of “rStar-Math” is that smaller language models can become world-class problem solvers simply by iterating on their own search-based reasoning. With Monte Carlo Tree Search and a well-crafted Process Preference Model, the system generates, checks, and refines its solutions—no giant teacher model required.

This self-evolved deep thinking matters not just for math, but for the entire paradigm of AI. By proving that effective feedback loops can replace the need for behemoth teacher models, the authors open the door to a new, more autonomous era of AI development. The next time you see a smaller model outperforming many of its bigger siblings, it may owe its success to precisely this kind of self-improvement approach.