DeepMind’s GenRM Boosts LLM Accuracy by Enabling Self-Verification

Introduction:

Large language models (LLMs) often struggle with factual and logical consistency, especially in complex reasoning tasks. To address this, researchers use verifiers or reward models to evaluate and choose the best output from several responses generated by LLMs. In a new approach, DeepMind introduces GenRM, a generative reward model designed to improve the accuracy of LLM outputs by making the models verify their own solutions.

Challenges with Traditional Verifiers:

Typical methods for improving LLM accuracy involve generating multiple answers and using a verifier or reward model to select the correct one. However, traditional reward models, which assign numerical scores, fail to fully utilize LLMs' generative abilities. Moreover, LLM-as-a-Judge, another common approach, lacks the specialized training and capabilities that verifiers possess.

The GenRM Approach:

GenRM takes a novel approach by training verifiers through next-token prediction. This method allows the model to tap into LLMs' natural strengths in generating and processing text. By uniting generation and verification, GenRM enables LLMs to generate intermediate reasoning steps through chain-of-thought (CoT) prompting before making a verification decision. This method helps identify subtle reasoning errors often overlooked by direct verification approaches.

For example, when verifying a solution, GenRM generates a CoT rationale—either human-created or generated by another LLM—then evaluates the answer using tokens like “Yes” or “No.” By leveraging multiple CoT chains, GenRM enhances its verification accuracy, particularly in complex reasoning tasks.

GenRM vs. Traditional Models:

In tests across various reasoning tasks, including word sorting and math problems, GenRM consistently outperformed other verification methods. When evaluated on the GSM8K math benchmark, GenRM surpassed even models like GPT-4 and Gemini 1.5 Pro. With chain-of-thought prompting and majority voting, GenRM achieved higher accuracy than traditional reward models and the LLM-as-a-Judge method.

Flexibility and Scalability:

One of GenRM’s key advantages is its scalability. The model continues to improve as it processes more responses and grows in size, allowing developers to balance computational costs with accuracy. Furthermore, GenRM can be fine-tuned using human or synthetic critiques, providing flexibility in training while maintaining high-quality verification.

Future Directions for GenRM:

The future of GenRM could include expanding its use in open-ended tasks, integrating it into reinforcement learning systems, and enhancing LLM capabilities with methods like retrieval-augmented generation and code execution. The approach shows promise for industries that need reliable and scalable verification systems for LLMs, such as autonomous systems or complex data processing applications.

Conclusion:

DeepMind’s GenRM offers a significant advancement in improving the accuracy of LLM-generated responses by merging solution generation and verification. Through chain-of-thought reasoning, next-token prediction, and a unified model, GenRM outperforms traditional methods, paving the way for more reliable, scalable LLM applications across various domains.