AI Model Security: Threats to the Model Itself, Not Just the Data

What is AI model security

AI model security is the practice of protecting a trained machine learning model, not just the data it learned from. That includes the model’s weights, its architecture, and the way it behaves when queried.

Many conversations about AI security start and end with the training data. That makes sense. Data breaches are familiar, well understood, and heavily regulated. But a trained model is its own asset.

It represents months of engineering time, compute spend, and tuning. In many cases, it also encodes patterns from data that was never meant to be exposed.

If someone can copy your model, reconstruct what it learned from, or pull the raw weights off a server, the data protections you built elsewhere in your stack don’t help. The model itself needs its own defenses.

For a closer look at protecting the data side of this equation, see Duality’s guide to AI data security.

Why attackers target the model, not just the data

A trained model is valuable on its own terms. Building one takes real investment: labeled data, compute time, specialized engineering, and often years of iteration. An attacker who can copy that model gets the benefit of all that work without paying for any of it.

There’s a second reason models are attractive targets. A model can act as a proxy for the data it was trained on. Query it enough times, in the right way, and it’s sometimes possible to reconstruct pieces of the underlying training set, including records that were supposed to stay private.

This is why model security and data security overlap, but they aren’t the same problem. Protecting the training set doesn’t automatically protect the trained model, and protecting the model doesn’t automatically stop every form of data exposure.

How to protect intellectual property during data collaboration

This overlap gets more complicated the moment more than one organization is involved. In collaborative AI projects, a hospital system, a bank, or a government agency may need to train a model using data from several partners.

Each partner has its own IP concerns: the data owners don’t want their raw records exposed, and whoever built the model doesn’t want it copied or reverse-engineered by a collaborator.

A few practices help keep IP protected in these setups:

●       Keep raw data at its source instead of centralizing it, using an approach such as federated learning.

●        Encrypt model weights and gradients while they’re in use, not only at rest or in transit.

●       Set clear, auditable access controls over who can query the model and how often.

●       Define ownership and usage rights for the trained model up front, before training starts, not after.

None of this replaces a legal agreement between parties. But technical controls make those agreements enforceable in practice, rather than relying on trust alone.

Duality’s approach to privacy-protected machine learning is built around exactly this kind of shared-model, shared-data scenario, where model IP protection and data protection have to hold at the same time.

Three attacks target the model’s confidentiality: extraction, inversion, and theft

These three attacks go after different things and call for different defenses.

Model extraction attacks

In a model extraction attack, someone with no special access, just the ability to send queries to a model and read the responses, tries to build a working copy of it.

They send carefully chosen inputs, record the outputs, and use those pairs to train a substitute model that behaves close enough to the original to be useful.

This isn’t a theoretical risk. Researchers at Cornell Tech and EPFL demonstrated working extraction attacks against several types of models exposed through prediction APIs, showing that black-box access alone was often enough to reconstruct a functionally similar model, without ever touching the original weights or training data  (Tramèr et al., USENIX Security 2016).

Any model exposed through a public or partner-facing API, especially one that returns confidence scores alongside predictions, carries some version of this risk.

Any model exposed through a public or partner-facing API, especially one that returns confidence scores alongside predictions, carries some version of this risk.

Model inversion attacks

Model inversion works differently. Instead of copying the model’s behavior, an attacker uses the model’s outputs to work backward toward the data it was trained on.

Given enough queries and the right technique, it’s sometimes possible to reconstruct sensitive attributes, or in some cases entire records, about individuals in the training set.

This matters most for models trained on regulated data: patient records, financial histories, government datasets.

A model that memorizes too much of its training data, and exposes that memorization through its outputs, can become a privacy liability even if the original dataset was never breached directly.

Model theft

Model theft is the most direct threat: someone gets a copy of the model artifact itself, rather than reconstructing it through queries. This can happen through compromised storage, an over-permissioned pipeline, or a departing employee walking out with model files.

Once someone has the actual weights, they can deploy the model elsewhere, resell it, or study it offline to find further weaknesses at their own pace.

This threat often gets less attention than extraction or inversion because it looks like a standard data breach on the surface. The difference is what’s inside the stolen files: years of tuning, evaluation, and design decisions, not simply records.

Whoever has those files can redeploy the model directly, with no further access to your systems required.

For readers who want a structured way to think through the full range of ML-specific risks, the OWASP Machine Learning Security Top 10 is a useful community-maintained reference that covers extraction, inversion, and several other attack types side by side.

Where models are exposed across the lifecycle

Model security isn’t a single control you add at the end of a project. Risk shows up at different points, and each stage needs its own approach.

●       During training: This is where distributed and federated learning setups introduce risk that centralized training doesn’t have. When multiple parties train a shared model without pooling their raw data, they typically exchange gradients or model updates instead. Those updates can themselves leak information about the underlying data if they aren’t protected. This is known as gradient leakage, and it is a separate attack from model inversion.

●        In storage and deployment: Trained models sit in registries, storage buckets, and deployment pipelines. Any of these can be misconfigured, over-permissioned, or left without proper access logging, which opens the door to straightforward model theft.

●       At inference and runtime. Once a model is live and answering queries, whether through an internal tool or a public-facing API, it’s exposed to both extraction and inversion attempts. The more queries an endpoint allows, and the more detail it returns in its responses, the more useful that exposure is to an attacker.

Organizations building collaborative AI across institutions face all three of these at once, since the model is being trained, stored, and queried by more than one party.

That’s the specific problem Secure Collaborative AI is built to address: training and deploying models across organizational boundaries without exposing the model or the data behind it to any single participant.

How do you protect model weights during distributed training

Distributed and federated training move the model to the data instead of the other way around, which limits raw data exposure. But it introduces a new question: how do you protect the model updates that move between participants during training?

A few approaches address this directly:

• Secure aggregation. Instead of any single party seeing individual updates from other participants, updates are combined using cryptographic protocols so only the aggregate result is revealed.
• Encrypted gradient exchange. Gradients can be encrypted in transit and processed without being decrypted, reducing the risk that a compromised network step exposes them.
• Secure multiparty computation. Multiparty computation lets multiple parties jointly compute a result, in this case a trained model, without any party seeing another’s raw inputs or intermediate values.

How to protect AI models in training and inference

Beyond the training stage specifically, a few categories of privacy-enhancing technologies apply protection across the model’s full lifecycle:

• Compute on encrypted data and models. Fully homomorphic encryption, or FHE, allows computations to run directly on encrypted data without ever decrypting it. Applied to model training or inference, this means the model can learn from or serve predictions on data that stays encrypted the entire time, which limits what an attacker gains even with full access to the infrastructure.
• Protect data and models while they’re in use. Confidential computing uses hardware-based secure environments to keep both the data and the model isolated during processing, not only when stored or transmitted. This closes a gap that encryption at rest and in transit leaves open: the moment data or a model is actually being used.
• Add noise to reduce what outputs reveal. Differential privacy introduces controlled statistical noise into a model’s outputs or training process, making it harder for an attacker to reconstruct specific training records through inversion, while preserving useful accuracy at the aggregate level.
• Limit and monitor access. Rate-limiting queries, restricting how much detail is returned in predictions, and monitoring for unusual query patterns all raise the cost of an extraction attempt. None of these stop a determined attacker outright, but they make large-scale extraction slower and easier to detect.

Organizations in regulated sectors evaluating these controls often align them against an existing framework rather than building a checklist from scratch. 

The NIST AI Risk Management Framework is a widely referenced starting point for structuring that evaluation, particularly for government, healthcare, and financial services teams that already report against other NIST guidance.

How Duality protects the model and the data

Duality’s platform is built around the idea that the model and the data behind it both need protection, and that neither should have to be exposed to make AI useful.

Each organization’s raw data stays where it is. Model updates are combined inside protected hardware environments (trusted execution environments), and a model whose design must stay private can run inside one of these environments, out of other parties’ reach.

This works for training a shared model, running inference, or building retrieval-based applications.

For organizations in government, defense, healthcare, and financial services, where both the data and the models built on it are often subject to strict compliance requirements, this means AI initiatives can move forward without exposing either the data or the model.

A defense agency training a shared model across departments, a hospital network pooling insights across sites, or a bank collaborating with partners on fraud detection all face the same underlying question: how do you get the value of collaborative AI without any single party having to expose their data or their model to the rest.

The platform is built to be the secure environment the AI pipeline runs in, from training through deployment.