LLM Math?

At the beginning of the year, I predicted that reasoning would become a hot topic, but I didn’t expect it to blow up so quickly.

Just like my 2023 prediction about MoE, by the end of 2024, reasoning is almost certainly going to be the AI buzzword of the year. I might as well be Diavolo’s King Crimson.

But to be honest, there hasn’t been much worth writing about lately, and I’ve received some complaints from my friends.

So today, I decided to cover two papers at once.

Not just to fill up space, but because these two papers are actually related!

First Paper

What does this paper discuss?

I’ll start with a conclusion: LLMs don’t actually solve math problems; they just rely on brute-force exposure to massive amounts of problems.

Let me explain how this paper validates that claim.

It modifies and intervenes in original problems — not just randomly, but in a way that makes the new problems appear similar while requiring completely different solutions.

These interventions are divided into simple and hard categories.

• Simple interventions: Minor modifications, like changing a denominator from x+1x+1x+1 to x+2x+2x+2.
• Hard interventions: Drastic changes, like modifying the denominator from x+1x+1x+1 to just xxx.

You might think changing x+1x+1x+1 to x+2x+2x+2 or just xxx isn’t a big deal.

But it’s actually a fundamental shift.

Take the first problem as an example. Changing x+1x+1x+1 to x+2x+2x+2 still allows the problem to be solved using factoring.

But if you change it to xxx, can you still factor it?

No — you have to use the Cauchy-Schwarz Inequality instead.

For an LLM that has been pretrained on large datasets:

1. If it has learned CSI (Cauchy-Schwarz Inequality) well or has encountered similar problems frequently:

• It acquires the knowledge.
• It internalizes the reasoning and applies it.
• The model can solve the problem correctly.

1. But if it hasn’t seen such problems often (since datasets like MATH500 might be dominated by factoring problems), will it just give up?

Not at all! Instead, it stubbornly sticks to factoring and tries to force a Chain-of-Thought (CoT) reasoning process that resembles the original approach.

And of course, it gets the problem wrong.

Summary

• LLMs can definitely learn problem-solving techniques from their training sets — this is what Chain-of-Thought (CoT) provides.
• But CoT also has fixed patterns, or reasoning frameworks, and the model blindly applies them even when they’re not appropriate.
• When a similar-looking problem is modified, the model doesn’t check whether its usual CoT approach is still valid — it just applies it anyway.

Let’s look at another example:

Left: Original Problem

• Question: Find all integer values of nnn.
• Solution:
• If nnn is even, simplification leads to 0=00 = 00=0, which has no solution.
• If nnn is odd, setting n=2m+1n = 2m+1n=2m+1 and solving gives the unique solution n=5n = 5n=5.

Right: Hard Perturbation (MATH-P-Hard)

• Modified Problem: The problem is changed and now asks for the smallest integer value of nnn instead.
• LLM’s Reasoning:
• The correct answer is n=10n = 10n=10, but the model outputs both n=10n = 10n=10 and n=13n = 13n=13.
• Instead of identifying just the smallest nnn, it memorizes and outputs all possible solutions.

So, sometimes I disagree with the idea that “compression equals intelligence” — of course, I’m not pointing fingers at anyone here.

Conclusion: LLMs solve math problems using probabilities, not true understanding.

Second Paper

Now, someone might ask: “If LLMs rely on probability, why do models like OpenAI’s O-series and R1 show significant improvements in math reasoning?”

The answer: They’ve learned patterns.

Or rather, they’ve learned different CoT patterns.

With better training methods, models can acquire more diverse reasoning frameworks.

Why does this work?

Let’s look at the next paper.

What This Paper Does

The core idea is simple:

It clusters different CoT reasoning approaches into around 500 distinct “patterns”.

Then, these CoT patterns are explicitly trained into the model.

Now, the model knows which CoT pattern to apply for each type of problem.

That’s all there is to it.

But this is actually a genius idea.

What does this mean?

It turns continuous action spaces into discrete ones.

Once you make reasoning a discrete, finite space, training becomes much easier.

As for whether the paper uses MCTS, BON, or its own training method — it doesn’t really matter.

• Any method that combines MCTS + a reward model can probably achieve similar results.
• The key point is internalizing CoT patterns into the model.

Inference Process in the New Model

1. First, read the problem.
2. Identify which CoT pattern applies.
3. Use that pattern to solve the problem.

That’s it.

Does This Work?

The paper includes ablation studies, showing improvement across all model sizes.

Although the tests only covered math, because they didn’t train the model on other domains,

I believe that embedding these reasoning patterns should generalize to other scenarios as well.

Math problems already have a lot of structure, and 500 reasoning patterns is actually quite a lot.

Where Did These 500 CoT Patterns Come From?

• Whether they were extracted from Gemini’s data
• Or generated synthetically

It doesn’t matter.

What matters is that they exist and they work.

Final Thoughts

These are two of the most interesting papers I’ve read recently.

They provide strong insights into future training methodologies.

And personally, I only value papers that convince me through fundamental principles.

Another perspective?

These papers actually made me feel more optimistic.

I’ve always worried that LLMs might eventually replace humans.

But I never had solid evidence to confirm or disprove that fear.

These two papers gave me some confidence.

Can LLMs replace humans?

At least not in their current form.

Can 500 CoT pattern templates really solve all reasoning problems?

What a joke.

Only when LLMs can autonomously generate reasoning and planning pathways will they be ready to compete with humans.

Until then — I’ve got a few more years to play around.