jailbreak
A prompt crafted to make an AI model bypass its safety training and produce output it was trained to refuse, often through role-play or a disguised request.
- The DAN prompt, which recast ChatGPT as a character without rules
- A harmful request encoded in Base64 to slip past safety training
- An algorithm-found string of characters appended to a request
- A long prompt packed with fake dialogues of an assistant complying
- Earliest AI use
- 2022
- Prompts that got ChatGPT past its refusals within its first days.
- Current meaning
- 2025
- NIST fixes the standard definition; the AI sense dates from 2022.
In plain words
In AI, a jailbreak is a prompt that gets a model to ignore its safety training and produce content it was built to refuse.
A jailbreak is an attempt to talk an AI model out of its own rules. Developers train chatbots to refuse some requests, and a jailbreak is a prompt written to get around that training. Merriam-Webster’s definition covers AI models as well as electronic devices.1 The AI sense borrows from phone hacking, where owners jailbreak an iPhone to install apps Apple has not approved.2
Jailbreaks appeared within a day of ChatGPT’s launch in November 2022. Many early ones were role-play prompts. The best known, DAN, short for “Do Anything Now”, recast ChatGPT as a character exempt from its rules.3 Later methods hide requests in encodings or pack a prompt with fake conversations. Others use strings of characters found by search algorithms. When OpenAI launched GPT-4 in March 2023, it said jailbreaks still existed.
Simon Willison, a developer who writes widely about these attacks, says the most common harm is embarrassment. A chatbot is tricked into offensive output, and a screenshot of it spreads. He calls a model helping someone commit a real crime the theoretical worst case.4 Labs also attack their own models on purpose, and invite outsiders to, in a practice called red teaming.
A jailbreak is often confused with prompt injection. A jailbreak targets the safety training built into the model, and it usually comes from the person typing. Prompt injection targets an application built on a model. Untrusted text, such as an email or a web page, overrides the developer’s instructions. Sources draw this line differently. OWASP’s security guidance calls jailbreaking a form of prompt injection. NIST’s attack taxonomy defines the two separately.
How it happens
A model learns to refuse in a stage often called safety training, which comes after pretraining. This stage shapes what the model tends to say. It does not erase what the model learned before.
Wei, Haghtalab and Steinhardt describe two ways safety training fails. In the first, goals compete. The model is also trained to be helpful and to follow instructions. A prompt that rules out the usual refusals, or dictates how the answer must begin, can tip it toward complying. In the second, safety training covers less ground than the model’s abilities. A request encoded in Base64, which the model can read, may fall outside anything its safety training covered.
Attacks can also be automated. Search algorithms can find strings of characters that make a model comply, and strings found on open models have worked on commercial ones. Many-shot jailbreaking fills a long context window with fake exchanges in which an assistant complies. The model follows the pattern, much as it learns from examples in few-shot prompting.
Defenses come in layers. Labs retrain models to refuse known attacks, but Anthropic found this only delayed the many-shot attack. Classifying and modifying prompts before they reached the model cut its success rate from 61% to 2% in one case.5 Classifiers that check inputs and outputs cut jailbreak success in Anthropic’s automated tests from 86% to 4.4%.6 Microsoft’s guidance adds a protective system prompt and monitoring for abuse.
Origin and first known use
A compound of jail and break. As a noun meaning an escape from jail, it is recorded from 1843. The verb dates from 2007, the year the iPhone went on sale, in the sense of removing a device’s built-in limits. Owners did this to add apps the maker had not approved. Bloomberg describes the AI sense as a nod to that phone hacking. It appeared within days of ChatGPT’s release in November 2022, when users traded prompts that got the chatbot past its refusals. By 2025 NIST had defined the term in its taxonomy of attacks on AI systems.
- 1843 Jailbreak, a noun for a forcible escape from jail, is first recorded.Merriam-Webster, “jailbreak”, First Known Use (noun) merriam-webster.com →: source for 1843
- 2007 The verb is first recorded, for removing the built-in limits of an electronic device, in the year the iPhone went on sale.Merriam-Webster, “jailbreak”, First Known Use (verb) merriam-webster.com →: source for 2007
- 2022 Zvi Mowshowitz’s post “Jailbreaking ChatGPT on Release Day” gathers prompts that got past ChatGPT’s refusals, several found by the end of its first day.Zvi Mowshowitz, “Jailbreaking ChatGPT on Release Day”, Don’t Worry About the Vase, 2 Dec 2022 thezvi.wordpress.com →: source for 2022
- 2023 Wei, Haghtalab and Steinhardt trace jailbreaks to two failures of safety training, competing objectives and mismatched generalization. Attacks built on them break GPT-4 and Claude v1.3.Wei, Haghtalab and Steinhardt, “Jailbroken: How Does LLM Safety Training Fail?”, NeurIPS 2023 arXiv 2307.02483 →: source for 2023
- 2023 Zou and colleagues automate jailbreaks: a search finds a text suffix on open models that also draws objectionable content from ChatGPT, Bard and Claude.Zou, Wang, Carlini, Nasr, Kolter and Fredrikson, “Universal and Transferable Adversarial Attacks on Aligned Language Models” arXiv 2307.15043 →: source for 2023
- 2024 Simon Willison argues that jailbreaking targets a model’s own safety filters, while prompt injection targets applications mixing untrusted input with a developer’s prompt.Simon Willison, “Prompt injection and jailbreaking are not the same thing”, 5 Mar 2024 simonwillison.net →: source for 2024
- 2025 Anthropic guards a Claude model with input and output classifiers and opens a public challenge. Four of 339 participants pass all eight levels, one with a universal jailbreak.Anthropic, “Constitutional Classifiers: Defending against universal jailbreaks”, 3 Feb 2025, updated 13 Feb 2025 anthropic.com →: source for 2025
- 2025 NIST’s attack taxonomy defines a jailbreak as a direct prompting attack that gets around limits on model outputs, and defines prompt injection separately.Vassilev et al., “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations”, NIST AI 100-2 E2025, March 2025 nist.gov →: source for 2025
In the wild
-
“Additionally, there still exist “jailbreaks” to generate content which violate our usage guidelines.”
-
“Sidestepping ChatGPT’s guardrails ‘like a video game’ for jailbreak enthusiasts—despite real-world dangers”
-
“Large language models trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of “jailbreak” attacks…”
-
“…so-called “jailbreaks” against LLMs… these attacks have required significant human ingenuity and are brittle in practice.”
-
“Jailbreaking is the class of attacks that attempt to subvert safety filters built into the LLMs themselves.”
-
“The basis of many-shot jailbreaking is to include a faux dialogue between a human and an AI assistant within a single prompt for the LLM.”
-
“An AI jailbreak is a technique that can cause the failure of guardrails (mitigations).”
Microsoft Threat Intelligence, Microsoft Security blog microsoft.com →: Microsoft Threat Intelligence, Microsoft Security blog, 4 Jun 2024Archived →: copy of Microsoft Threat Intelligence, Microsoft Security blog, 4 Jun 2024 -
“Jailbreaking is a form of prompt injection where the attacker provides inputs that cause the model to disregard its safety protocols entirely.”
OWASP Gen AI Security Project, Top 10 for LLM Applications 2025 genai.owasp.org →: OWASP Gen AI Security Project, Top 10 for LLM Applications 2025, 18 Nov 2024Archived →: copy of OWASP Gen AI Security Project, Top 10 for LLM Applications 2025, 18 Nov 2024 -
“Anthropic has a new way to protect large language models against jailbreaks”
Not to be confused with
-
Prompt injection
Prompt injection targets an application built on a model: untrusted text, such as an email or web page, overrides the developer’s instructions. A jailbreak targets the model’s own safety training, usually from the person typing.
-
Red teaming
Red teaming is authorized testing in which people attack a model on purpose to find its weaknesses, often by trying jailbreaks. A jailbreak is the attack itself, whoever carries it out.
-
A phone jailbreak
Jailbreaking a phone removes the maker’s limits on the software it runs, so its owner can add unapproved apps. An AI jailbreak changes nothing in the model; it is a prompt, usually in plain language.
Questions
What is the difference between a jailbreak and prompt injection?
A jailbreak targets the safety training built into a model, usually by the person typing, to get output the model was trained to refuse. Prompt injection targets an application built on a model: untrusted text, such as an email or web page, overrides the developer’s instructions. Simon Willison and NIST treat them as separate attacks. OWASP counts jailbreaking as a form of prompt injection.
Why do AI jailbreaks work?
Safety training shapes what a model tends to say but does not erase what it learned before. Wei, Haghtalab and Steinhardt describe two gaps. The model is also trained to be helpful and follow instructions, and those goals can win out over safety. Safety training also covers less ground than the model’s abilities, so a request in an unusual form, such as Base64, can slip past.
How do AI companies defend against jailbreaks?
AI companies use several layers of defense. Labs retrain models to refuse known attacks, though Anthropic found this only delayed its many-shot attack. Classifiers screen what goes into and comes out of a model. In Anthropic’s automated tests they cut jailbreak success from 86% to 4.4%. In its 2025 public challenge, four of 339 participants still passed all eight levels, one with a universal jailbreak.
What was the DAN jailbreak?
DAN, short for “Do Anything Now”, was a role-play prompt for ChatGPT that recast the chatbot as a character exempt from its rules. A study of 1,405 jailbreak prompts shared online from December 2022 to December 2023 calls DAN the original jailbreak prompt. Close variants followed, and asking the model to play a character remained a common jailbreak strategy.
Where does the word jailbreak come from?
The noun, meaning an escape from jail, is recorded from 1843. The verb dates from 2007, the year the iPhone went on sale. It meant removing a device’s built-in limits so its owner could add unapproved apps. The AI sense appeared within days of ChatGPT’s launch in November 2022, when users shared prompts that got the chatbot past its refusals.
Watch
Cite this entry
“Jailbreak.” SUPERR_dictionary, Superrduperr, 24 Sept. 2026, superrduperr.com/
Superrduperr. (2026, September 24). Jailbreak. In SUPERR_dictionary. https://superrduperr.com/
Superrduperr. “Jailbreak.” SUPERR_dictionary. September 24, 2026. https://superrduperr.com/