Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated
A training data leak is a problem where an AI shows, in its answers, confidential information it learned. The report of NIST (the US National Institute of Standards and Technology) sorts as one kind of information extraction the risk that personally identifiable information (PII) and the like, which a generative AI met during training, can be drawn out as answers.
How does a training data leak work?
It is like a student with a good memory who repeats part of a name list seen in class word for word. An AI learns from a huge amount of text, and if it remembers some of it down to the phrasing, that can mix into an answer. According to NIST's report, for AI that writes text, email addresses, phone numbers, and locations have been reported to come out as the continuation of sentences about personal information.
The same report also touches on the fact that what comes out cannot be relied on. The answer can be inaccurate, for example by confusing whom the information belongs to. OWASP's LLM02 names the risk that sensitive data, proprietary algorithms, and secret details leave through the output of an AI app. The whole picture of information leaks through generative AI is also organized on the LLM data leaks page.
Why does a training data leak matter?
If you use customer lists or internal meeting minutes to train or tune an AI, a route appears by which their content can leave in the form of answers. NIST's report introduces techniques that remove the influence of particular data from a trained model, and says that the more certain the method, the higher the computing cost. Choosing what goes in before you put it in is the first preparation.
OWASP's LLM02 names, as headings of its measures, sanitizing data, access control, privacy-preserving training techniques, and user education and transparency. Under sanitizing, it shows making sure users' data does not enter the trained model. If you use an outside AI service, confirm by contract and settings whether what you enter is used for training.
To set rules on what may be entered into AI, contact us.
Related terms
- LLM data leak: a problem where an LLM includes learned information or entered secrets in its answers.
- Shadow AI: AI services that employees use without the company's knowledge.
Next step
List, inside the company, the information that may be entered into AI and the information that must not, and share it. The free URL check shows the state of the public side of your site, and for setting up rules on AI use, contact us.
Frequently asked questions
Can personal information an AI produced be taken as real?
NIST's report says an AI's answers can be unreliable, for example by confusing who a piece of information belongs to. Whether or not it is real, treat output that contains personal information as a suspected leak and tell the response contact in the company.
Why does an AI memorize training data?
NIST's report explains that neural networks have a tendency to memorize training data. It also introduces the view that a model with larger capacity can more easily reproduce what it learned as it is.
Frequently asked questions
Can personal information an AI produced be taken as real?
NIST's report says an AI's answers can be unreliable, for example by confusing who a piece of information belongs to. Whether or not it is real, treat output that contains personal information as a suspected leak and tell the response contact in the company.
Why does an AI memorize training data?
NIST's report explains that neural networks have a tendency to memorize training data. It also introduces the view that a model with larger capacity can more easily reproduce what it learned as it is.