---
title: "17. If you train an AI on your company's data, can that data be pulled back out — model inversion"
url: https://blog.tyrano.dev/en/17-if-you-train-an-ai-on-your-company-s-data-can-that-data-be-pulled-back-out-model-inversion
lang: en
site: Tech AI News - 테카이
category: "Know Before You Use"
tags: ["AI","AI security","model inversion","fine-tuning","data privacy","LLM","OWASP"]
published: 2026-11-09T00:32:49.164Z
updated: 2026-11-10T12:17:20.998Z
sources:
  - https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting
  - https://arxiv.org/abs/2311.17035
  - https://genai.owasp.org/llmrisk/llm02/
  - https://arxiv.org/abs/2508.14062
  - https://arxiv.org/abs/2506.20856
  - https://www.protecto.ai/blog/model-inversion-attacks-in-llms/
---

# 17. If you train an AI on your company's data, can that data be pulled back out — model inversion

> If you fine-tuned an AI on internal company emails so it could mimic a sender's name or the company's tone, the vendor's phone number or the contract terms written in that email went into the model along with it. Keep probing it with varied questions, and that number can come back out.

---

## Background — why this question is coming up now

More companies are fine-tuning general-purpose models on internal data — support logs, contracts, internal code, HR records — to sharpen response quality. The natural question that follows is: does this data get stored as-is inside the model's weights, and if so, can it be pulled back out later?

This concern isn't hypothetical. Researchers have repeatedly shown that training data can actually be extracted from public services using nothing but repeated queries — no access to the weights or the training infrastructure at all. A [November 2023 paper from a team including Google DeepMind](https://arxiv.org/abs/2311.17035) reported extracting training data from ChatGPT using nothing more than an attack that made it repeat a single word endlessly.

The problem is that for a model a company fine-tunes itself, the target of such an attack is no longer "a vast pile of data scraped from the internet" — it's "our company's narrow, specific data." And many companies assume they're immune to this risk simply because "we don't touch the weights directly, we only call the API."

## Key data & current state

| Case / study | Result | Source |
|---|---|---|
| GPT-2 extraction verification | Of 1,800 candidates, over 600 matched actual training data exactly (67% at the best setting), including names, phone numbers, and emails that were public PII | Carlini et al., USENIX Security 2021 |
| ChatGPT (gpt-3.5-turbo) repetition attack | Extracted over 10,000 pieces of verbatim training data for about $200 worth of queries; reported to OpenAI on 2023.8.30 and patched | Nasr et al. (joint research incl. Google DeepMind), 2023.11 arXiv |
| Repeated sensitive records during fine-tuning | Repeatedly exposing the same sensitive record during fine-tuning raised the leak rate from a baseline of 0–5% to 60–75% | arXiv 2508.14062, 2025 |
| LoRA fine-tuning comparison | Keeps performance roughly on par with full fine-tuning while meaningfully lowering memorization risk (though it's unverified whether it blocks advanced extraction attacks) | arXiv 2506.20856, 2025 |

> Interpretation: What all four cases share is that the attacker never touched the model's internals (weights) or the original database. That means training data can be statistically reconstructed just by repeating queries — which is exactly why the assumption "we're safe because we only use the API" doesn't hold.

## Deeper analysis

### 1) What exactly is model inversion

Model inversion isn't an attack that breaches the original database. It's a method where an attacker repeatedly queries a model and analyzes its outputs (or output probabilities) to infer or reconstruct the original data used in training. The [OWASP GenAI Security Project](https://genai.owasp.org/llmrisk/llm02/) describes it as a risk where "exposed training data can leave a model open to inversion attacks," citing CVE-2019-20634, where exposed training data led to model extraction and inversion that let attackers bypass a real email filter.

A related but weaker concept, membership inference, only confirms whether a given piece of data was used in training. Model inversion goes a step further — it attempts to reconstruct the content itself.

### 2) Why fine-tuning raises the risk

General-purpose pretrained models are trained on billions of diverse documents, so the influence of any single sentence on the whole model gets diluted. The datasets companies use for their own fine-tuning, by contrast, are relatively small and narrow. The more a given customer record or contract term repeats within that dataset, the more sharply the probability rises that the model will "memorize" that specific record. In an experiment that repeatedly exposed sensitive records during fine-tuning, the leak rate jumped from a baseline of 0–5% to as high as 60–75%.

The AI data privacy company Protecto frames this as an architectural problem rather than a flaw in the model itself: "if raw customer records, employee information, or confidential documents enter training pipelines," it creates future privacy exposure. No matter how carefully you handle the model afterward, if sensitive raw data goes in at the start, the risk is baked in from the design stage.

### 3) A black box behind an API isn't safe either

The premise that "we don't touch the model weights directly, we only call the API" doesn't guarantee safety. The attack that extracted training data from ChatGPT involved no access whatsoever to the model's internal structure — it was done purely by feeding text into the public API and reading the output. The same principle applies even if a company hides its own fine-tuned model behind its own API. What the attack needs isn't access to the weights — only query access.

### 4) Full prevention is hard — mitigation is a trade-off

Techniques like differential privacy, which add noise during training to dampen any single sample's influence, are effective but also reduce model accuracy. Rate limiting can raise the cost and time required for bulk extraction enough to make it impractical, but a targeted attack that probes for one specific piece of information with a small number of queries can stay under the request-count threshold and pass right through.

Analysis: In the end, no single safeguard closes this out. The overall risk only comes down when you stack controls across training-data governance, output filtering, and query-pattern monitoring.

## In practice — 5 principles to check before training an AI on internal data

1. **Strip or mask sensitive information before it enters the training pipeline.**
   Tokenize or anonymize data that can identify a specific individual — customer information, employee information, contract terms — before it goes into the fine-tuning dataset.
2. **Deduplicate so the same sensitive record isn't exposed repeatedly.**
   Repeated exposure raises memorization risk sharply rather than linearly, so deduplication alone has an outsized effect.
3. **Consider a parameter-efficient method like LoRA before full fine-tuning where possible.**
   It keeps performance roughly comparable while lowering memorization risk — but don't assume it blocks advanced extraction attacks too.
4. **Put pattern-matching filters on model outputs as well.**
   Don't just control the training stage — detect and filter patterns like phone numbers or account numbers at the stage before a response reaches the user.
5. **Drop the assumption that "we're safe because we only call the API," and monitor query patterns.**
   Rate limiting raises the cost of bulk extraction but doesn't catch targeted attacks, so pair it with detection that looks at query patterns themselves, not just request counts.
