Reinforcement learning and Mixture of Experts in Deepseek R1 a disruptor?

Big tech took a hit on 1/27/2025 on wall street due to news of 'Deepseek' R1 LLM. What makes Deepseek different in the AI world?

Reinforcement Learning (RL) has emerged as a powerful technique in the development of advanced language models, as demonstrated by DeepSeek's recent breakthrough with their R1 model. This article will focus on the recently released DeepSeek-R1 and its predecessor, DeepSeek-R1-Zero. This model also uses the concept of Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token at any given time. This reminds me of a Mercedes 8-cylinder engine that turns of four cylinders when on highway for efficiency.

The Rise of Reinforcement Learning in Language Models

DeepSeek, a small unit in a Chinese quant firm, working on a "side project" has made significant strides in the field of artificial intelligence with the introduction of their R1 model, which utilizes large-scale reinforcement learning to enhance reasoning capabilities in language models. This approach represents a departure from traditional supervised fine-tuning methods and has yielded remarkable results.

DeepSeek-R1-Zero: RL Approach + Mixture of Experts approach

DeepSeek-R1-Zero, the precursor to R1, was trained using pure reinforcement learning without any supervised fine-tuning1. This model demonstrated exceptional reasoning abilities, showcasing the potential of RL in developing advanced AI systems. The training process involved using the GRPO (Group Relative Policy Optimization) framework to improve the model's performance in reasoning tasks.The results were impressive, with DeepSeek-R1-Zero achieving a pass@1 score of 71.0% on the AIME 2024 benchmark, which further improved to 86.7% with majority voting. This performance matched that of OpenAI's o1-0912 model, highlighting the effectiveness of the pure RL approach.

Mixture of Experts (MoE) is a machine learning approach that divides an artificial intelligence (AI) model into separate sub-networks or "experts," each specializing in a subset of the input data, to jointly perform a task1. This technique combines the strengths of multiple specialized models to make more accurate and robust predictions. [5]

1. Expert Networks: These are individual machine learning models trained on different subsets of data, allowing them to become proficient in handling specific types of inputs
2. Gating Network: This component acts as a "traffic director," determining which expert(s) are most suitable for a given input.
3. Combination Function: The outputs of the selected experts are combined using methods like averaging or weighted averaging to produce the final output

"Unlike conventional dense models, mixture of experts uses conditional computation to enforce sparsity: rather than using the entire network for every input, MoE models learn a computationally cheap mapping function that determines which portions of the network—in other words, which experts—are most effective to process a given input, like an individual token used to represent a word or word fragment in NLP tasks. This allows the capacity of the model?to be increased (by expanding the total number of parameters) without a corresponding increase in the computational burden required to train and run it (because not all of those parameters will necessarily be used at any given time)." [5]

MoE architectures offer several benefits:

1. Improved Efficiency: They enable large-scale models to reduce computation costs during pre-training and achieve faster performance during inference time
2. Scalability: MoE allows for dramatically scaling up model or dataset size with the same compute budget as a dense model
3. Flexibility: New experts can be added or removed as needed
4. Better Performance: MoE can be particularly effective in tasks where the input space is large and complex

Applications of MoE include image recognition, natural language processing, and recommendation systems4. Recent developments have seen MoE being applied to Large Language Models (LLMs) to improve their capabilities while managing computational resources effectively

Key Features of DeepSeek-R1

Building upon the success of R1-Zero, DeepSeek-R1 incorporates several innovative features:

1. Hybrid Learning System: DeepSeek-R1 combines model-based and model-free reinforcement learning, allowing for faster adaptation in dynamic environments and greater efficiency in computationally intensive tasks
2. Multi-Agent Support: The model includes robust multi-agent learning capabilities, enabling coordination among agents in complex scenarios such as logistics, gaming, and autonomous vehicles
3. Explainability Features: Addressing a significant gap in RL models, DeepSeek-R1 provides built-in tools for explainable AI (XAI), allowing users to understand and visualize the model's decision-making process
4. Pre-Trained Modules: An extensive library of pre-trained modules significantly reduces deployment time across various industries
5. Customizability: The model supports a wide range of frameworks, including TensorFlow and PyTorch, with APIs for seamless integration into existing workflows

The "Aha Moment": Emergent Reasoning Behaviors

One of the most fascinating aspects of DeepSeek-R1's development is the emergence of sophisticated reasoning behaviors that were not explicitly programmed. Through the reinforcement learning process, the model developed the ability to self-correct, reevaluate flawed logic, and validate its own solutions within its chain of thought. notable example of this emergent behavior is what researchers termed the "Aha moment." During problem-solving, the model demonstrated the ability to pause, reconsider its approach, and explicitly flag its realization of a better solution3. This behavior showcases the potential of RL to foster autonomous and adaptive reasoning in AI systems.

Limitations and Challenges

Despite its impressive capabilities, DeepSeek-R1 and its RL-based approach face several limitations and challenges:

1. Readability and Language Mixing: DeepSeek-R1-Zero encountered issues with poor readability and language mixing, occasionally producing responses that combined Chinese and English6.
2. Reward Function Design: The effectiveness of RL heavily depends on well-designed reward functions, which can be complex to create for abstract reasoning tasks
3. Real-World Application: Applying RL models to real-world scenarios remains difficult due to factors such as reward sparsity, delay, and ambiguity
4. Context Limitations: There are concerns about the model's ability to handle long contexts, potentially leading to abrupt cutoffs in responses

DeepSeek's cost-benefit narrative:

1. Reported Low Development and Training Cost: DeepSeek claims its DeepSeek-V3 model cost less than $6 million US to develop, a fraction of the cost typically associated with training advanced AI models
2. Competitive Performance: Despite its lower cost, DeepSeek-R1 has demonstrated impressive capabilities, overtaking ChatGPT O1 as the top-rated free application on Apple's App Store in the US
3. Market Impact: The emergence of DeepSeek has caused a selloff in AI-related stocks, particularly affecting companies like Nvidia, as investors reassess the value of high-cost AI investments
4. Efficiency Claims: DeepSeek reports using cheaper chips and less data than competitors, challenging the assumption that AI development requires ever-increasing resources
5. Open-Weight Model: DeepSeek-R1 was released as "open-weight," allowing developers to examine and build upon its inner workings, potentially accelerating innovation in the field
6. Cost Comparison: The DeepSeek-R1 is reported to be 20 to 50 times cheaper to use than OpenAI's o1 model, depending on the task

Conclusion:

Seems like a seismic shift and yet other participants in this space may have othr techniques cooking. This is not the first of the surprise innovation AI will see.

References:

1. [2412.19437] DeepSeek-V3 Technical Report
2. https://semiengineering.com/deepseek-improving-language-model-reasoning-capabilities-using-pure-reinforcement-learning/
3. https://bgr.com/tech/developers-caught-deepseek-r1-having-an-aha-moment-on-its-own-during-training/
4. https://www.prompthub.us/blog/deepseek-r-1-model-overview-and-how-it-ranks-against-openais-o1
5. https://www.ibm.com/think/topics/mixture-of-experts
6. https://huggingface.co/blog/moe
7. https://tedai-sanfrancisco.ted.com/glossary/mixture-of-experts/
8. https://toloka.ai/blog/mixture-of-experts-approach-for-llms/