Prompt injection meaning, in plain words

Prompt injection abuses the instructions given to an AI so that it acts in unintended ways. What OWASP LLM01 and NIST AI 600-1 say, and why no sure fix exists.

Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated

Prompt injection is a technique that abuses the instructions given to an AI so that it behaves in ways nobody intended. OWASP describes it as a weakness in which a user's input changes the behavior or output of a large language model (LLM, an AI that converses in text) in an unintended way.

How does prompt injection work?

Picture asking a receptionist to guide each visitor according to the manual, and then finding that a note a visitor handed over hides a different order. If the receptionist follows the order in the note, the manual is no longer kept. An AI can go wrong in the same way when it cannot tell the original instruction from text that came in from outside.

OWASP divides the problem into two. The direct type is where text typed by the user changes the AI's behavior. The indirect type is where an instruction contained in a web page or file the AI took in changes the AI's behavior. NIST also describes the indirect type as an attack that plants instructions from outside in the data an AI-connected app is likely to retrieve.

Note that it works even in a form a person cannot read by eye, as long as the AI can read it. No sure method of prevention is known, and OWASP's countermeasures are described as ways to reduce damage.

Checking features that include AI is covered in the AI and LLM assessment. For an outside view of your site, the free URL check is available. The weakness itself is organized on the prompt injection vulnerability page.

Why does prompt injection matter?

More companies are adding automated answers to inquiries, or AI that reads documents and answers, to their sites. The more an AI connects to internal documents and outside services, the wider the effect if its behavior is taken over.

As countermeasures, OWASP lists defining the AI's role and range in its instructions, checking the form of the output with a program, and limiting input and output. Designs such as narrowing permissions and requiring human approval are organized on the prompt injection vulnerability page.

The document also adds the idea of treating the AI like an untrusted user. A periodic check that tries the system from an attacker's viewpoint is one of the countermeasures as well.

Related terms

  • Jailbreak: a technique that gets an AI to bypass its restrictions through clever wording
  • Excessive agency: a state where the permissions given to an AI are too large
  • LLM data leak: a problem where secrets leak through what is fed to an AI or what it answers
  • Insecure output handling: the flaw of trusting an AI's answer and acting on it directly

Frequently asked questions

Can prompt injection be fully prevented?

No method that can be said to prevent it exists at present. OWASP says no sure way to prevent it is known, and that the countermeasures are meant to reduce damage. Prepare with a combination: define the AI's role and range, narrow the permissions it holds, and have a person approve important actions.

Is it the same as a jailbreak?

OWASP describes a jailbreak as a form of prompt injection. Its feature is trying to make the AI ignore all of its safety rules, and the two words are often used interchangeably.

Next step

If a feature on your site includes AI, write down which documents and services the AI can touch. The scope of a check is on the AI and LLM assessment page. For a discussion before you introduce it, contact us.

Frequently asked questions

Can prompt injection be fully prevented?

No method that can be said to prevent it exists at present. OWASP says no sure way to prevent it is known, and that the countermeasures are meant to reduce damage. Prepare with a combination: define the AI's role and range, narrow the permissions it holds, and have a person approve important actions.

Is it the same as a jailbreak?

OWASP describes a jailbreak as a form of prompt injection. Its feature is trying to make the AI ignore all of its safety rules, and the two words are often used interchangeably.

See what an attacker can see, today

The free URL check returns a grade in under a minute. For costs and next steps, book a free consultation.