Threat actors jailbreak DeepSeek, Qwen AI models to generate ‘malicious’ content: Report

The Qwen series of AI models developed by Chinese tech giant Alibaba has also shown potential for misuse with minimal restrictions, the report said.

Since the meteoric rise of DeepSeek, experts have expressed concerns about safety and risk mitigation taking a backseat in the high-stakes race for AI supremacy. In October last year, OpenAI had confirmed that its popular AI chatbot, ChatGPT, was used by threat actors to write new malware and make existing ones more efficient.

Key findings

Presenting blurred screenshots as evidence, the report highlighted the following ways in which AI models developed by DeepSeek and Alibaba have been used for malicious purposes.

Developing infostealers: “Threat actors have been reported creating infostealers using Qwen, focusing on capturing sensitive information from unsuspecting users.”

Bypassing banking protections: “Multiple discussions and shared techniques on using DeepSeek to bypass banking system anti-fraud protections have been found, indicating the potential for significant financial theft.”

Mass spam distribution: “Cyber criminals are using three AI models—ChatGPT, Qwen, and DeepSeek together —to troubleshoot and optimise scripts for mass spam distribution.”

However, the Check Point report did not specify the research methodology used to uncover these incidents or divulge the scale of the operations and other details.

After finding ways to manipulate DeepSeek and Qwen models to show uncensored content, threat actors also shared the information with others online, as per the report.

This information included jailbreaking prompts such as the “Do Anything Now” and “Plane Crash Survivors” methods to manipulate responses by DeepSeek’s AI models.

Jailbreaking is an umbrella term but in this context, it refers to the various techniques that allow users to manipulate AI models to generate uncensored or unrestricted content. “This tactic has become a preferred technique for cyber criminals, enabling them to harness AI capabilities for malicious intent,” the report said.

It pointed out that newer AI models have attracted interest from different levels of attackers, especially low-skilled ones who are capable of exploiting scripts or tools without a deep understanding of AI. The report did not mention the identities of the threat actors or their country of origin.

We have reached out to DeepSeek, OpenAI, and Alibaba for comment. This report will be updated with their responses if we hear back.