Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated
An LLM data leak is the problem of internal confidential information escaping through what is typed into an AI or what it answers. OWASP organizes, as sensitive information disclosure, the cases in which an LLM (an AI that converses in text), or an application that uses one, exposes private information or confidential data through its output.
How does an LLM data leak work?
It resembles showing confidential company documents in full to an adviser to ask for an opinion, and then finding that the adviser has told someone else what was in them. With AI, information can leave by two routes: how the information you type in is handled, and what gets mixed into the AI's answer.
According to OWASP, the information that can leak includes personal information, financial and health records, business secrets, credentials and legal documents, and it also covers non-public information related to how the model is built. Confidential content may also get mixed into an answer unintentionally. The countermeasures OWASP shows include removing confidential content from training data, minimizing the range of external data the AI can touch, and educating users not to enter confidential information. The encyclopedia entry is on the LLM data leak vulnerability page, and the order of countermeasures is in information leak countermeasures.
Why does an LLM data leak matter?
Because AI is convenient, it is a tool that is easily used on the judgment of people on the floor. If the people typing do not know what must not go in, information leaves without anyone noticing. The AI Guidelines for Business (Ministry of Internal Affairs and Communications and Ministry of Economy, Trade and Industry) list, as an effort for businesses that use AI, paying attention not to enter confidential information and the like inappropriately.
The first thing an operator can do is to put in writing the line between information that may be entered and information that may not. The problem of AI being used without the company knowing is explained under shadow AI. The free URL check shows your site's outside view, but not what staff type into AI services.
Related terms
- Shadow AI: generative AI that employees use without the company knowing.
- Prompt injection: a technique that abuses instructions to an AI to make it behave in unintended ways.
Next step
First, make a list of information that may be entered into AI and information that may not. An example of the organization is on the LLM data leak vulnerability page, and the order of the countermeasures as a whole is in information leak countermeasures. To draw up the rules, contact us. Your own site's outside view is on the free URL check.
Frequently asked questions
Does information we type into AI leave the company?
It depends on the service and its settings. As countermeasures, OWASP lists educating users not to enter confidential information, and stating clearly the policy on data retention and opt-out (the choice not to let your data be used for training). Check the terms and settings of the service you use first.
If we instruct the AI "do not answer with this information", will it not leak?
That alone is not enough. OWASP says that restricting the kinds of data by instruction can be bypassed by prompt injection, so you should not rely on it alone. Narrowing the information you give the AI in the first place is more reliable.
Frequently asked questions
Does information we type into AI leave the company?
It depends on the service and its settings. As countermeasures, OWASP lists educating users not to enter confidential information, and stating clearly the policy on data retention and opt-out (the choice not to let your data be used for training). Check the terms and settings of the service you use first.
If we instruct the AI "do not answer with this information", will it not leak?
That alone is not enough. OWASP says that restricting the kinds of data by instruction can be bypassed by prompt injection, so you should not rely on it alone. Narrowing the information you give the AI in the first place is more reliable.