Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated
Data poisoning is an attack that mixes poison into the data an AI learns from or refers to. OWASP explains it as illegitimately altering the data used to build or tune an LLM (an AI that converses in text), so as to plant hidden weaknesses, backdoors or biased behavior.
How does data poisoning work?
If extra ingredients are written into some pages of a recipe book, the dishes made by that book taste different. An AI is the same: if the material it learns from or refers to is contaminated, its answers are distorted.
OWASP lists three stages at which it can happen. They are pre-training, which learns from broad general data, fine-tuning, which learns additionally to fit a purpose, and embedding, which turns text into a string of numbers. The risk is said to be especially high when the source is external data that has not been checked.
There are also cases in which a backdoor is planted that makes the AI behave strangely only when a specific trigger appears. It works normally the rest of the time, which makes it hard to find in tests. NIST lists, as one of the cyber risks to AI, an attack in which training data is tampered with and the AI's output or behavior is manipulated.
The case in which the source data of RAG is contaminated is covered on the RAG glossary page. The organization of the topic is on the data poisoning vulnerability page.
Why does data poisoning matter?
An AI's answers depend on the quality of its source material. If you build in an AI that uses your own documents or documents taken in from outside, you need to know who can change that material.
The measures OWASP shows include tracing the origin and the change history of data, carefully judging the providers of data, and isolating the AI from data that has not been checked. It also adds a method of managing versions of data to find unauthorized changes.
Even a small company can start right away on reviewing where the documents an AI reads are kept and limiting who can write to them. The free URL check shows what is visible from outside.
Related terms
- RAG: a mechanism that searches internal documents and uses them in an AI's answers.
- Supply chain attack: an attack that uses the weak points of business partners or parts as a foothold.
Next step
Decide where the documents the AI refers to are kept, and keep the number of people who can write to them to a minimum. The organization of the weak points is on the data poisoning vulnerability page, and what is visible from outside on your own site is covered by the free URL check. You can talk through how to proceed with checks in a free consultation.
Frequently asked questions
At which stage does data poisoning happen?
OWASP explains that it can happen at the stages of pre-training, fine-tuning (additional learning to fit a purpose) and embedding (the processing that turns text into a string of numbers). The risk rises when external data that has not been checked is used.
Is it relevant even if we do not build AI ourselves?
It may be. OWASP lists the risk that the source data for the knowledge of RAG (a mechanism that searches internal documents and uses them in answers) is contaminated by insiders, by instructions, by deliberately mixed-in data, or by unchecked providers. You need to check where the documents you use come from.
Frequently asked questions
At which stage does data poisoning happen?
OWASP explains that it can happen at the stages of pre-training, fine-tuning (additional learning to fit a purpose) and embedding (the processing that turns text into a string of numbers). The risk rises when external data that has not been checked is used.
Is it relevant even if we do not build AI ourselves?
It may be. OWASP lists the risk that the source data for the knowledge of RAG (a mechanism that searches internal documents and uses them in answers) is contaminated by insiders, by instructions, by deliberately mixed-in data, or by unchecked providers. You need to check where the documents you use come from.