Adversarial example meaning, in plain words

An adversarial example is input data crafted to make an AI misjudge. How an evasion attack works, which defenses NIST names, and where they fall short.

Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated

An adversarial example is input data crafted to make an AI get it wrong. The report of NIST (the US National Institute of Standards and Technology) takes up evasion attacks, in which an attacker tries to change the AI's judgment to a result of their own choosing, and calls the input made for that purpose an adversarial example.

How does an adversarial example work?

Imagine adding to a sample part that an inspection machine passes a tweak too slight for a human eye, after which only the machine misreads it as a reject. According to NIST's report, the goal of an evasion attack is to make an input that changes the classification result to another class the attacker chose. The change is often described as small. In image classification, the image can look unchanged to a person while only the AI judges it as a different class.

On defense, NIST's report lists three promising directions. One is adversarial training, which adds crafted inputs with the right answers to the training. Another is checking that the result does not change when noise is added to the judgment. The third is verifying the output mathematically. The report also says these have limits, for example that accuracy on ordinary input can fall. If you build an AI image check or an automatic screening into your service, the free URL check shows from outside the state of the entry points you publish.

Why do adversarial examples matter?

If you take the AI's judgment as the final decision as it is, then once the judgment drifts, no one is left to notice. The effect is larger where the AI's judgment leads straight to a business result, such as inspection by image, automatic sorting of inquiries, and fraud detection.

NIST's report places defense against adversarial examples as a hard problem even among researchers, and asks that evaluation assume strong attacks. For a small business, a realistic start is to keep a step where a person looks over the AI's judgment, and to keep records that let you notice when judgments suddenly become biased.

To go over which of your business decisions the AI touches, contact us.

Related terms

  • Data poisoning: mixing improper data into the data an AI learns from to bend its judgment.
  • AI red teaming: a check made beforehand from an attacker's position to find an AI's weak points.

Next step

Write out which business judgments the AI you use takes part in, and review whether a person checks at that step. For the state of a published site, use the free URL check, and for worries about adopting or running an AI, contact us.

Frequently asked questions

Can a person tell an adversarial example by eye?

Taking image classification as an example, NIST's report explains that when the change to the original image is small, a person does not notice while the AI alone judges it to be a different class. This is an explanation based on images, though, and does not apply to every input or method.

Do countermeasures prevent adversarial examples?

NIST's report says that past defenses were published after being evaluated against weak attacks and were later broken by stronger ones. No method that can be said to prevent them is shown, so we recommend a design that keeps a step where a person can check the AI's judgment.

Frequently asked questions

Can a person tell an adversarial example by eye?

Taking image classification as an example, NIST's report explains that when the change to the original image is small, a person does not notice while the AI alone judges it to be a different class. This is an explanation based on images, though, and does not apply to every input or method.

Do countermeasures prevent adversarial examples?

NIST's report says that past defenses were published after being evaluated against weak attacks and were later broken by stronger ones. No method that can be said to prevent them is shown, so we recommend a design that keeps a step where a person can check the AI's judgment.

See what an attacker can see, today

The free URL check returns a grade in under a minute. For costs and next steps, book a free consultation.