Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated
Hallucination is the phenomenon in which AI gives a plausible-sounding false answer. NIST's guidance on generative AI organizes it under the name "confabulation". It describes the risk as users being misled by confidently stated errors or falsehoods.
How does hallucination work?
Picture a new hire who answers every question with confidence, even ones they do not know well. AI does not always stop and say "I don't know". It may assemble a plausible answer from the patterns in the text it learned from.
NIST says the danger lies in the confident tone of the answer. Users may believe the error, act on it, or pass it on. Its example is a summary of patient information. If the summary contains an error, a doctor could choose the wrong diagnosis or treatment.
A note in the NIST document adds that some people regard the terms "hallucination" and "fabrication" as a problem in themselves, because they treat AI like a person. Checks of features that use AI answers are covered on the AI and LLM diagnosis page.
Why does hallucination matter?
An error made by AI comes out as text that is hard to tell apart from the real thing by sight. If it is used as is in a notice to customers, or in an explanation of a contract or a price, it affects the company's credibility.
The AI Guidelines for Business say that using RAG (retrieval-augmented generation, a mechanism that searches internal documents and uses them in the AI's answer) is expected to suppress hallucinations and improve the transparency of the grounds for an answer. For the use of generated output, they also set out the measure of keeping human judgment in the loop.
In practice, the basic rule is not to publish AI output as it is. A person checks facts, figures and proper names against the source before using them. When you bring generative AI or an LLM into your work, decide first who does this checking. We take consultations on setting up the checking flow through a free consultation.
Related terms
- Generative AI: AI that creates new text, images and other content.
- RAG: a mechanism that searches internal documents and uses them in the AI's answer.
Next step
Before AI-written text goes out, set a step in which a person checks the figures and proper names. Checks of features that embed AI are in the AI and LLM diagnosis. To settle who does the checking, contact us; the free URL check covers only what is visible from outside.
Frequently asked questions
Is a hallucination the same as an AI mistake?
The meanings are close, but a hallucination is marked by an error stated in a confident tone. NIST describes the phenomenon as confidently stated erroneous or false content. The danger is that users believe it.
Does using RAG make hallucinations disappear?
No, it cannot be said to remove them. The AI Guidelines for Business say that using RAG and similar methods is expected to suppress hallucinations and make the grounds for an answer more transparent. It is a method expected to reduce them, and checking by a person is still needed separately.
Frequently asked questions
Is a hallucination the same as an AI mistake?
The meanings are close, but a hallucination is marked by an error stated in a confident tone. NIST describes the phenomenon as confidently stated erroneous or false content. The danger is that users believe it.
Does using RAG make hallucinations disappear?
No, it cannot be said to remove them. The AI Guidelines for Business say that using RAG and similar methods is expected to suppress hallucinations and make the grounds for an answer more transparent. It is a method expected to reduce them, and checking by a person is still needed separately.