A new academic report has found that most popular AI-powered chatbots are easily tricked into producing harmful and illegal content. The study highlights what it describes as a “tangible and deeply concerning” threat, as “jailbroken” chatbots can now deliver illicit information they absorbed during training.

Despite built-in safety systems intended to block inappropriate or dangerous queries, researchers say these protections can be bypassed through jailbreaking, specially crafted prompts designed to override safety guardrails.

Chatbots such as ChatGPT, Gemini, and Claude rely on large language models (LLMs) trained on massive troves of internet data, which contain material related to illegal activities, including hacking, money laundering, drug manufacturing, and bomb-making. While efforts have been made to filter such content from training datasets, the underlying models still retain knowledge of it.

According to the researchers, jailbreaks exploit a fundamental tension in LLM design: while the primary objective of these models is to fulfill user commands, their safety measures are secondary, making them vulnerable to manipulation.

Researchers develop a universal jailbreak

The study was led by Professor Lior Rokach and Dr. Michael Fire of Ben Gurion University of the Negev in the occupied Palestinian territories. Their team developed a universal jailbreak that successfully compromised multiple leading AI chatbots, compelling them to respond to nearly any prompt, including those that would normally trigger a refusal.

“It was shocking to see what this system of knowledge consists of,” said Fire. The researchers documented instances in which chatbots, once jailbroken, provided detailed guidance on hacking, drug manufacturing, and other criminal activities.

Rokach emphasized the unprecedented danger posed by this capability. “What sets this threat apart from previous technological risks is its unprecedented combination of accessibility, scalability, and adaptability,” he said.

Some of the dark LLMs identified in the study are explicitly marketed online as having "no ethical guardrails" and promoting their willingness to assist in cybercrime, fraud, and other illegal activities.

AI companies fail to respond adequately

After identifying the vulnerability, the researchers alerted major LLM providers. However, they described the response as “underwhelming,” while several companies either failed to respond or dismissed the jailbreak issue as outside the scope of their security bounty programs, which are designed to reward ethical hackers for reporting vulnerabilities.

The authors argue that without a serious shift in how tech firms address model-level risks, AI security will remain fragile. They call for comprehensive screening of training data, the deployment of robust query firewalls, and the development of "machine unlearning" methods to help chatbots forget dangerous material.

According to the report, dark LLMs should be classified as severe security threats, comparable to unlicensed weapons or explosives, and providers should face accountability for any misuse.

Dark LLMs compared to unlicensed weapons

External experts have shared the same concerns raised by the study. Dr. Ihsen Alouani, a researcher in AI security at Queen’s University Belfast, warned that jailbreak attacks on LLMs could enable real-world harm, from disinformation and social engineering to automated scams and weapon-making guides.

“A key part of the solution is for companies to invest more seriously in red teaming and model-level robustness techniques, rather than relying solely on front-end safeguards,” Alouani said, adding, “We also need clearer standards and independent oversight to keep pace with the evolving threat landscape.”

Commented for @Forbes on the dangers of poorly trained #AI

The recent story of Google Gemini outputting harmful content to its users highlights the #risks of operating #LLMs and the critical need for continuous testing of models and their guardrails https://t.co/pxv0hYLyoW

— Peter Garraghan (@DrGarraghan) November 27, 2024

Professor Peter Garraghan of Lancaster University, who specializes in AI security, agreed that current safeguards are insufficient. “Organisations must treat LLMs like any other critical software component, one that requires rigorous security testing, continuous red teaming and contextual threat modelling,” he said.

Garraghan added that meaningful AI security demands not only responsible disclosure but also responsible design and deployment.

Industry responds to growing pressure

OpenAI, the company behind ChatGPT, responded by pointing to improvements in its latest “o1” model, which it says demonstrates increased resilience to jailbreak attempts due to its ability to reason about internal safety policies.

Microsoft, another major AI provider, shared a blog post outlining its ongoing work to strengthen protections against such exploits. Other tech giants, including Meta, Google, and Anthropic, have yet to publicly respond to the study’s findings.