Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated
RAG (retrieval-augmented generation) is a mechanism that searches internal documents and uses them in an AI's answer. OWASP explains that it is a technique for adapting a trained language model by combining it with an external knowledge source, and that it uses vectors (numbers that represent the features of a text).
How does RAG work?
It resembles an exam in which you are allowed to bring reference materials. When a question comes in, the system first looks on the shelf of documents for the relevant passages, puts them at hand, and then writes the answer. It is easier to produce an answer with grounds than by relying on the AI's memory alone.
According to OWASP, the purpose is to give outside information at the time of the question, in order to raise the quality of an LLM-based app's answers (an LLM is an AI that converses in text) and how well they fit the context. The AI Guidelines for Business (Ministry of Internal Affairs and Communications and Ministry of Economy, Trade and Industry) explain that, at inference time, when the AI works out an answer, it uses information supplemented with external knowledge through RAG and similar methods.
It is convenient, but the place where the reference documents are kept becomes a new weakness. OWASP lists the danger of information leaking when permission checks are weak, and mixing when several users share the same storage. The problem of documents being contaminated also relates to data poisoning.
Checks of features with AI built in are covered in the AI and LLM assessment. The risk of contaminated documents is organized on the data poisoning vulnerability page.
Why does RAG matter?
Features that let an AI read internal rules and product documents are spreading in customer-inquiry handling and elsewhere. If the range of documents the AI can read is not matched to people's permissions, documents that should not be shown may appear in answers.
The countermeasures OWASP shows are a design that checks permissions finely and separates where data is kept logically. It also lists checking how documents are taken in, authenticating where they come from, and inspecting for contamination and hidden content. Keeping records of retrieval is also part of the countermeasures.
The AI Guidelines for Business also note that RAG may make answers more likely to converge on similar ones, and may not suit work that needs diversity or originality. You need to choose where to use it. The free URL check reads your site's public pages; it does not show which documents an AI can retrieve.
Related terms
- Data poisoning: an attack that mixes poison into the data an AI learns from or refers to.
- Hallucination: the phenomenon in which AI gives a plausible-sounding false answer.
Next step
Make a list of the documents you let the AI read, and decide the range of people who may see them and who can rewrite them. The summary is on the data poisoning vulnerability page, and feature checks are in the AI and LLM assessment. To work on the design, contact us. The free URL check is limited to what a public page shows from outside.
Frequently asked questions
If we use RAG, will the AI's mistakes disappear?
It cannot be said that they go away. The AI Guidelines for Business expect RAG and similar methods to curb hallucinations and to make the grounds for an answer easier to follow. If the reference documents are wrong, the answer will be wrong too.
Can internal secrets leak through RAG?
They can. OWASP lists the danger that, if access control is weak, the AI may pull out personal or confidential information and answer with it. A design that divides, document by document, who may see what is needed.
Frequently asked questions
If we use RAG, will the AI's mistakes disappear?
It cannot be said that they go away. The AI Guidelines for Business expect RAG and similar methods to curb hallucinations and to make the grounds for an answer easier to follow. If the reference documents are wrong, the answer will be wrong too.
Can internal secrets leak through RAG?
They can. OWASP lists the danger that, if access control is weak, the AI may pull out personal or confidential information and answer with it. A design that divides, document by document, who may see what is needed.