OpenAI's Project "Strawberry": Enhancing AI Reasoning Capabilities

Reporters from Reuters have unveiled that OpenAI has been working on a secretive project codenamed "Strawberry." This initiative aims to enhance the reasoning capabilities of AI models, focusing on complex tasks that require planning and multi-step problem-solving. This initiative is part of OpenAI's broader efforts to improve the transparency and reliability of AI systems, which includes their ongoing work on superalignment. Superalignment focuses on ensuring that highly advanced AI systems align with human values and intentions, even as they become more capable than current models.

Key Features of Project Strawberry

• Specialized "Post-Training" Phase: The project involves a unique phase to fine-tune models, potentially using techniques similar to Stanford's "Self-Taught Reasoner" method.
• Autonomous Research and Engineering: OpenAI aims for Strawberry to enable AI to conduct research and perform engineering tasks autonomously.

Innovative Training Methods

One of the most surprising aspects of the article is the innovative approach where OpenAI used GPT-2, a significantly older and less powerful model, to train GPT-4. This technique aimed to explore whether a less capable model could effectively supervise and guide the training of a more advanced one.

Key Findings

• Performance Improvement: The results showed that GPT-4, when trained with GPT-2’s responses, performed 20% to 70% better on language tasks than GPT-2 itself.
• Unexpected Outcomes: Although this method did not fully match the performance of GPT-4 trained on correct answers, the fact that it performed as well as it did was a promising and unexpected outcome.