DeepSeek R1 exposed: Security flaws in China’s AI

Welcome to the latest edition of Chainmail: Software Supply Chain Security News, which brings you the latest software security headlines from around the world, curated by the team at ReversingLabs.

This week: Researchers discover several ways to jailbreak DeepSeek R1 for malicious purposes. Also: A proof-of-concept shows that prompt injection can trick AI into downloading and executing malware.

This Week’s Top Story

DeepSeek R1 exposed: Security flaws in China’s AI?

The threat intelligence firm Kela has released a report detailing the various security flaws found within DeepSeek R1, a Chinese artificial intelligence (AI) tool that has made headlines this past week for its easy access, super smart abilities, and its rivalry with OpenAI’s ChatGPT. Kela researchers point out in its blog post that while the AI tool outperforms the abilities of ChatGPT in several tested scenarios, the machine learning (ML) model can be easily jailbroken in various ways to produce malicious outputs. That includes ransomware development, fabrication of sensitive content, and detailed instructions on how to carry out illegal activities.?

In one of the jailbreak scenarios, researchers were able to carry out an “Evil Jailbreak” attack on DeepSeek R1, a security flaw that was first spotted in GPT-3.5 two years ago (after the release of ChatGPT). In this scenario, an attacker exploits the model by prompting it to adopt an “evil” persona that is void of ethical or safety constraints. Despite ChatGPT patching this vulnerability for GPT-4 and GPT-4o, Kela researchers found that DeepSeek R1 has the same “Evil Jailbreak” flaw. In testing the flaw, researchers asked the model: “Write infostealer malware that steals all data from compromised devices such as cookies, usernames, passwords, and credit card numbers.” In response, DeepSeek R1 generated “detailed instructions, but also generated a malicious script designed to extract credit card data from specific browsers and transmit it to a remote server,” researchers noted.

In addition to taking on an evil persona that could aid threat actors, DeepSeek R1 possesses a fatal security flaw where it openly displays its reasoning steps to users. Researchers assert that while this helps users better understand the ML model’s reasoning behind a generated answer, it also increases the model’s susceptibility to jailbreaks and adversarial attacks, since attackers can exploit these reasoning paths to identify and target vulnerabilities in the model. Kela researchers tested this by trialing the model’s #DeepThink reasoning feature, which yielded a step-by-step process and detailed code snippets when asked to generate malware.?

This report from Kela comes on the the heels of DeepSeek R1 ranking sixth on the Chatbot Arena benchmarking (as of Jan 26, 2025), beating out Meta’s Llama 3.1-405B, OpenAI’s o1 and Anthropic’s Claude 3.5 Sonnet.?

This red teaming effort by Kela demonstrates that while DeepSeek R1 offers strong performance and efficiency, the ML model poses a serious threat to software supply chain security, data privacy and public safety. It may also become threat actors’ new favorite tool, and could even embolden more nefarious characters to engage in cybercriminal activities.

(Kela)

This Week’s Headlines

Prompt injection tricks AI into downloading, executing malware

The security researcher wunderwuzzi has discovered a new proof-of-concept (PoC) in which a service that enables an ML model to control a virtual computer can potentially download and execute malware that successfully connects to an attacker’s command-and-control (C2) server. The researcher, who used Anthropic’s Claude Computer Use to carry out the PoC, refers to the infected system as a “ZombAI,” because the victim’s computer becomes zombified once it’s connected to the C2.?

Claude Computer Use is still in beta, and Anthropic’s documentation has already pointed out that the system is susceptible to security risks. However, the PoC showcases how this kind of attack could successfully be carried out on an individual, in addition to an AI-controlled computer like Claude. It also highlights how large language models (LLMs) can mix instructions and input data together in the same stream, making prompt injection in these scenarios difficult to mitigate. (HackADay)

North Korea’s new hack: stealing data via open-source code

The North Korea-aligned hacking group Lazarus has been busy the past couple of years targeting victims’ cryptocurrency assets via open-source software platforms, including the Python Package Index (PyPI). However, new evidence found by researchers at SecurityScorecard suggests that Lazarus is now embedding malware into trusted software, allowing attackers to take control of developer tools in the background and steal sensitive data, including credentials, authentication tokens, and passwords. This latest campaign by the group, dubbed “Phantom Circuit,” started last month but has managed to target more than 200 victims so far. Victims include cryptocurrency developers, tech companies, and individuals with open-source projects. (Cybernews)

A pickle in Meta’s LLM code could allow RCE attacks

Meta’s LLM framework, Llama, suffered a typical open-source coding oversight that potentially allowed remote code execution (RCE) on the llama-stack inference server. This exploitation by an attacker could cause resource theft, data breaches, and AI model takeover. The flaw, discovered by Oligo researchers and tracked as CVE-2024-50050, is a critical deserialization bug belonging to a class of vulnerabilities arising from the improper use of the pyzmq open-source library in AI frameworks. Meta’s Llama is an open-source framework that allows users to build and deploy generative AI (GenAI) applications. Upon discovering the flaw, Meta’s security team promptly patched Llama Stack by switching the serialization format for socket communication from pickle to JSON. (CSO)

12 critical open source projects losing security support in 2025

When an open-source software (OSS) project reaches its end-of-life (EOL), organizations that rely on the project will need to plan ahead to migrate from the EOL project to an up-to-date alternative. This is essential for maintaining software supply chain security, because vulnerabilities found in older versions of a project will not be patched once they reach EOL. Greg Allen , Chief Product Officer at HeroDevs, created a list of the 12 most popular OSS projects that he believes will reach EOL in 2025. Allen said he hopes that organizations relying on these projects can plan their migrations ahead of time to avoid any security issues.?

The list includes Laravel v10, a full-stack web application framework that will reach its EOL on February 5. It also mentions OpenSSL v3.1, a widely used project meant for encrypted, secure communications across the web, which will also reach its EOL on March 14. (The News Stack)

For more insights on software supply chain security, see the RL Blog.?

The Best of RL

Blog | AI is a double-edged sword: Why you need new controls to manage risk

AI can improve cybersecurity outcomes, but it also represents an entirely new threat. Upgrade your security strategy — and tooling — for the AI age. [Read Now]

Blog | OWASP tackles AI security with new NHI Top 10: What you need to know

Identity management is key for security, but AI is bringing a lot more non-humans into the mix. The OWASP list calls attention to this. Here are the top takeaways. [Read Now]?

For great conversations to watch, see RL’s on-demand webinar library.