A Simple Guide to DeepSeek R1: Architecture, Training, Local Deployment, and Hardware Requirements

DeepSeek’s Innovative Approach to Enhancing LLM Reasoning

DeepSeek has introduced a groundbreaking method to enhance the reasoning capabilities of large language models (LLMs) using reinforcement learning (RL), which is highlighted in their recent research on DeepSeek-R1. This paper marks a significant leap forward in improving the problem-solving abilities of LLMs by relying on reinforcement learning, without the heavy dependence on supervised fine-tuning.

Technical Overview of DeepSeek-R1

Model Architecture:

DeepSeek-R1 is not just one model but a family of models that includes DeepSeek-R1-Zero and DeepSeek-R1. Below is a breakdown of the key differences between these two:

The Key Difference:

• DeepSeek-R1-Zero: This was the initial version where the team explored pure reinforcement learning, with no reliance on supervised fine-tuning. They applied RL directly to the base model, allowing it to develop reasoning capabilities through trial and error. While this method showed promising results, with a 71% accuracy on AIME 2024, it did face challenges related to language consistency and readability. The model contains 671 billion parameters and uses a mixture-of-experts (MoE) architecture, where each token activates around 37 billion parameters. This setup allowed the model to exhibit emergent reasoning behaviors such as self-verification and long chain-of-thought reasoning.
• DeepSeek-R1: This version builds on DeepSeek-R1-Zero but introduces a more sophisticated multi-stage training process. While it still uses RL, it begins with a supervised fine-tuning phase using a curated set of examples. This combination allows DeepSeek-R1 to achieve better readability, coherence, and overall performance while retaining the same 671 billion parameters.

Training Process

Training Methodology:

• Reinforcement Learning: DeepSeek-R1 leverages RL in a unique way, focusing on group relative policy optimization (GRPO) to maximize rewards based on reasoning accuracy and format. This avoids the need for extensive labeled datasets, setting it apart from traditional methods that rely on supervised learning.
• Distillation Techniques: To make powerful models more accessible, DeepSeek has released distilled versions of DeepSeek-R1, ranging from 1.5 billion to 70 billion parameters. These smaller models are fine-tuned with synthetic reasoning data from the full DeepSeek-R1, ensuring they deliver strong performance while being more computationally efficient.

The training process for each model follows these patterns:

• DeepSeek-R1-Zero follows a simple reinforcement learning approach:
• DeepSeek-R1 involves a more detailed, four-step process:

Performance Metrics

DeepSeek-R1 has been evaluated across several reasoning benchmarks and has shown impressive results:

• AIME 2024: Achieved a 79.8% pass rate, outpacing OpenAI’s o1–1217 by a small margin.
• MATH-500: Scored 97.3%, slightly better than o1–1217’s 96.4%.
• SWE-bench Verified: Outperformed other models in programming tasks, highlighting its coding proficiency.

Additionally, the API for DeepSeek-R1 is cost-efficient, priced at $0.14 per million input tokens for cache hits, offering a more affordable option compared to models like OpenAI’s o1.

Limitations and Future Work

Despite its advancements, there are areas where DeepSeek-R1 can improve:

• The model sometimes struggles with tasks requiring specific output formats.
• Performance on software engineering tasks could be enhanced.
• Multilingual contexts show issues with language mixing.
• Few-shot prompting often leads to degraded performance.

Future improvements will address these issues, with a focus on enhancing multi-turn interactions, function calling, and complex role-playing capabilities.

Deployment and Accessibility

Open Source and Licensing: DeepSeek-R1, along with its variants, is released under the MIT License, which encourages open-source collaboration and commercial use, including model distillation. This ensures wider access and fosters innovation in AI model development.

Model Formats: The models, including their distilled versions, are available in formats such as GGML, GGUF, GPTQ, and HF, providing flexibility for deployment across different platforms.

Web Access via DeepSeek Chat Platform:

Simply visit the DeepSeek Chat platform, register or log in, and choose the "Deep Think" mode for interactive reasoning.

API Access:

For programmatic access, users can connect through the DeepSeek API, which integrates seamlessly with OpenAI’s format. The setup involves obtaining an API key, configuring the environment, and making API calls to get responses.

Running Locally:

• Hardware Requirements: Running the full models locally requires substantial resources, including a GPU with ample VRAM (e.g., Nvidia RTX 3090 or higher). For CPU use, at least 48GB of RAM and 250GB of disk space are recommended, though performance may suffer without GPU acceleration.
• Distilled Models: For those with more modest hardware, distilled models (ranging from 1.5B to 70B parameters) are available and can be deployed on systems with less powerful specifications.

Local Running Tools: Tools like Ollama and vLLM can be used to serve the models locally. For example, running DeepSeek-R1 on a local machine using Ollama involves commands like ollama run deepseek-r1:1.5b for smaller models or up to ollama run deepseek-r1:70b for the most powerful version.

Software Tools for Local Running:

1. Ollama:

You can use Ollama to serve the models locally: (Ollama Is a tool for running open-source AI models locally on your machine. Grab it here: https://ollama.com/download)

Next, you’ll need to pull and run the DeepSeek R1 model locally.

Ollama offers different model sizes — basically, bigger models equal to smarter AI, but need better GPU. Here’s the lineup:

1.5B version (smallest):
ollama run deepseek-r1:1.5b

8B version:
ollama run deepseek-r1:8b

14B version:
ollama run deepseek-r1:14b

32B version:
ollama run deepseek-r1:32b

70B version (biggest/smartest):
ollama run deepseek-r1:70b        

To begin experimenting with DeepSeek-R1, it is advisable to start with a smaller model to familiarize yourself with the setup and ensure compatibility with your hardware. You can initiate this process by opening your terminal and executing the following command:

ollama run deepseek-r1:8b        

Sending Requests to locally downloaded DeepSeek-R1 via Ollama:

Ollama provides an API endpoint to interact with DeepSeek-R1 programmatically. Ensure that the Ollama server is running locally before making API requests. You can start the server by running:

ollama serve        

Once the server is active, you can send a request using curl as follows:

curl -X POST https://localhost:11434/api/generate -d '{
  "model": "deepseek-r1",
  "prompt": "Your question or prompt here"
}'        

Replace "Your question or prompt here" with the actual input you wish to provide to the model. This command sends a POST request to the local Ollama server, which processes the prompt using the specified DeepSeek-R1 model and returns the generated response.

Other methods to run/Access the models locally are:

vLLM/SGLang: Used for serving the models locally. Commands like vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B — tensor-parallel-size 2 — max-model-len 32768 — enforce-eager can be used for the distilled versions.

llama.cpp: You can also use llama.cpp to run the models locally.

Conclusion

The progression from DeepSeek-R1-Zero to DeepSeek-R1 demonstrates a significant evolution in LLM reasoning. While DeepSeek-R1-Zero proved that reinforcement learning could successfully enhance reasoning, DeepSeek-R1 shows that combining reinforcement learning with supervised learning results in a more robust and efficient model. This work sets the stage for even more advanced models in the future, with improvements aimed at overcoming current limitations and expanding capabilities.

Collaborations ??:

Interested in exploring AI and machine learning projects together? Let’s talk! We are open to collaborations and excited to work with other professionals in the field.