Model inversion attack meaning, in plain words

A model inversion attack uses an AI's answers to infer the data it was trained on. What NIST and OWASP say about the risk and how to reduce it.

Written by: Mio Kanzaki(Security Analyst (AI))Reviewed by: 千葉圭介Updated

A model inversion attack is an attack that infers the data used for training from an AI's answers. The report of NIST (the US National Institute of Standards and Technology) introduces it as an attack that recreates representative examples of each class from the data a machine learning classification model learned.

How does a model inversion attack work?

It is like comparing many stamp impressions to work out roughly the shape of the original stamp. The idea is to trace back, from the impressions that are the AI's answers, the features of the data used in training. According to NIST's report, a model inversion attack can create images similar in meaning to the training data, yet it cannot reproduce the training data directly. The same report names the tendency of AI to memorize training data as part of why reconstructing data is possible.

The measure NIST gives is differential privacy. It is an idea that makes it hard to read from the output whether one particular piece of data was included in training. NIST says it helps against reconstruction attacks but also points out a trade-off between the strength of privacy and the accuracy of the AI. The whole picture of information leaks from services that use generative AI is organized on the LLM data leaks page.

Why does a model inversion attack matter?

If you train or tune an AI with your customer data or internal documents, a route appears by which that data can leave in the form of answers. If you remove personal information and partners' secrets from the training data, the very things that could be inferred become fewer.

OWASP's LLM02 names sanitizing data, so that users' data does not enter the trained model, as one measure against disclosure of sensitive information. Even if you do not plan to build an AI yourself, it is worth confirming before you sign whether an AI service you use takes your input into training.

To review how your AI services treat your input, contact us.

Related terms

  • Data poisoning: mixing improper data into the data an AI learns from to bend its judgment.
  • LLM data leak: a problem where an LLM includes learned information or entered secrets in its answers.

Next step

Make a list of the data you use to train or tune an AI, and check whether personal information or secrets are mixed in. For a review on the side that uses AI services, start from the free URL check, which examines the state of your published site, or contact us.

Frequently asked questions

Does a model inversion attack restore training data exactly?

NIST's report explains that a model inversion attack can create data close in meaning to each class of the training data, but it cannot reproduce the training data itself directly. For a different kind of data reconstruction attack, though, it says individuals' data can be restored under some conditions.

Which kinds of AI does a model inversion attack target?

The example NIST's report introduces is a machine learning model that classifies images and the like. For generative AI that writes text, extraction of training data is covered under a separate item.

Frequently asked questions

Does a model inversion attack restore training data exactly?

NIST's report explains that a model inversion attack can create data close in meaning to each class of the training data, but it cannot reproduce the training data itself directly. For a different kind of data reconstruction attack, though, it says individuals' data can be restored under some conditions.

Which kinds of AI does a model inversion attack target?

The example NIST's report introduces is a machine learning model that classifies images and the like. For generative AI that writes text, extraction of training data is covered under a separate item.

See what an attacker can see, today

The free URL check returns a grade in under a minute. For costs and next steps, book a free consultation.