Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated
A jailbreak is a technique that gets an AI to bypass the restrictions placed on it through clever wording. NIST's material on generative AI describes it as crafting a prompt (the instruction text given to an AI) in order to get around restrictions on the output.
How does a jailbreak work?
It is close to a gate where only authorized people may pass, and someone tells a plausible story to make the guard forget the rule. The gate and the guard stand for the safety mechanisms built into the AI. The rules themselves are not broken. The wording leads the AI into not following them.
OWASP places the jailbreak as a form of prompt injection. Its feature is steering the AI toward ignoring all of its safety rules. The document also says the two words are often used with the same meaning.
To resist it, the model's training and safety mechanisms are said to need continuing updates. It is not something that is finished once it has been built. The whole picture of prompt injection is on the prompt injection term page, and the vulnerability entry is on the prompt injection vulnerability page.
Why does a jailbreak matter?
Not a few operators think an AI is safe once told "do not answer this" or "do not reveal this information". OWASP, however, says that even when you limit the types of information by instruction, prompt injection can bypass the limit.
So information you want to protect in an AI's answers should not rest on the AI's restrictions alone. Design so that the information is never handed to the AI in the first place. Narrowing the range of documents the AI can touch is another review that works.
NIST also points out that tests which only try jailbreaks may not systematically confirm an AI's accuracy or reliability. Testing is to be combined with other checks. The outside view of a site that embeds an AI feature is something the free URL check can show.
Related terms
- Prompt injection: a technique that abuses an AI's instructions to make it behave in unintended ways
- Guardrails: safety mechanisms that put a brake on an AI's output and behavior
Frequently asked questions
What is the difference between a jailbreak and prompt injection?
OWASP treats a jailbreak as a form of prompt injection, and describes it as trying to make the AI ignore all of its safety rules. The two words are often used with the same meaning.
If I tell the AI "do not do this", am I safe?
That alone is not enough. OWASP says that limiting a type of data by instruction can still be bypassed by prompt injection. In our view, narrowing permissions and checking the output are a realistic addition.
Next step
Write down separately the rules you ask the AI to follow and the information you hand it. Then decide a policy to keep the handed-over information to a minimum. A detailed breakdown is on the prompt injection vulnerability page, and for a discussion with us, contact us.
Frequently asked questions
What is the difference between a jailbreak and prompt injection?
OWASP treats a jailbreak as a form of prompt injection, and describes it as trying to make the AI ignore all of its safety rules. The two words are often used with the same meaning.
If I tell the AI "do not do this", am I safe?
That alone is not enough. OWASP says that limiting a type of data by instruction can still be bypassed by prompt injection. In our view, narrowing permissions and checking the output are a realistic addition.