Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated
AI red teaming is a check that tries to find an AI's weaknesses from the attacker's point of view. NIST's material for generative AI describes it as a still-developing approach that finds undesirable behavior and results of an AI, and how they could arise, and puts load on its safety mechanisms to check them.
How does AI red teaming work?
It resembles checking the security of a building, where someone other than the designer pretends to be an intruder and looks for weak points in the locks and windows. With AI, you pretend to be an attacker and test whether you can make the AI behave unexpectedly.
According to NIST, such exercises are often carried out in a controlled environment in cooperation with the people who build the AI, and can be done both before and after the AI is released. The same material also says that the quality of the results is related to the background and expertise of the team that carries it out.
In Japan, the AI Safety Institute (AISI) published the "Guide to Red Teaming Methodology on AI Safety" in September 2024. Version 1.10, published on March 31, 2025, is described as adding the procedure that was actually carried out on an AI system implementing RAG, together with a document of the results. RAG is a mechanism that searches internal documents and uses them in the AI's answers.
Checks of AI features are covered by the AI and LLM assessment. One of the weak points is prompt injection.
Why does AI red teaming matter?
An AI returns slightly different answers even to the same question each time, so a check with a fixed procedure alone can miss problems. A method that tries in many ways from the attacker's side plays a complementary role.
The Hiroshima Process international code of conduct, which the AI Guidelines for Business (Ministry of Internal Affairs and Communications and Ministry of Economy, Trade and Industry) refer to, says to combine evaluation methods such as red teaming and to adopt diverse internal and external tests. OWASP's material on prompt injection also includes, among its measures, treating the AI as an untrusted user and simulating attacks on a regular basis.
It is not limited to companies that build their own AI. Even when you build in a ready-made AI, how it is built in needs to be checked. The scope and the way to proceed are decided according to the purpose. The free URL check shows what is visible from outside.
If you want help scoping an AI check, contact us.
Related terms
- Red team: a team that plays the part of an attacker to test an organization's defenses.
- Penetration test: a check that actually tries to break in to find weaknesses.
Next step
If you have a feature with AI built in, first write down the points you want to test. The scope of the check is gathered in the AI and LLM assessment. What is visible from outside can be seen with the free URL check, and we take questions on how to proceed through a free consultation.
Frequently asked questions
Does red teaming guarantee that an AI is safe?
It does not. NIST positions AI red teaming as an approach still under development. It also says the quality of the results depends on the background and expertise of the team carrying it out. It is only one means of finding weaknesses.
How is it different from an ordinary vulnerability assessment?
An ordinary assessment looks for known weaknesses in a site or system. AI red teaming is characterized by trying, from the attacker's side, whether an AI behaves unexpectedly or gives undesirable output. It is used together with assessments of the non-AI parts.
Frequently asked questions
Does red teaming guarantee that an AI is safe?
It does not. NIST positions AI red teaming as an approach still under development. It also says the quality of the results depends on the background and expertise of the team carrying it out. It is only one means of finding weaknesses.
How is it different from an ordinary vulnerability assessment?
An ordinary assessment looks for known weaknesses in a site or system. AI red teaming is characterized by trying, from the attacker's side, whether an AI behaves unexpectedly or gives undesirable output. It is used together with assessments of the non-AI parts.