Quick Take
- Generative AI security has three layers: the prompt, the model, and the data behind it.
- Prompt and model risks (injection, poisoning, theft) are well documented. The data layer is the one many guides skip.
- If the underlying data can be exposed, no amount of prompt or model security fully closes that gap.
Generative AI moved from pilot projects to production systems fast. Large language models now draft code, summarize regulated documents, and sit in front of internal knowledge bases that used to require a login and a reason to access.
Security teams are asking a reasonable question: where does the risk actually live?
The honest answer is that it lives in three places. The prompt, where an attacker or a careless employee can manipulate what the model does. The model itself, where training data and outputs can be tampered with or extracted. And the data layer, where the sensitive information behind the whole system either stays protected or doesn’t.
Many generative AI security content stops at the first two. This one goes to the third, because for organizations working with regulated, classified, or otherwise sensitive data, that’s where model risk management earns its name.
What generative AI security actually means
Generative AI security covers the practices, controls, and architecture decisions that keep large language models and the systems around them from becoming a new source of data exposure, manipulation, or compliance failure.
It’s a different problem than traditional application security for a few reasons. A generative AI system doesn’t run fixed code paths. It interprets language and produces output probabilistically, which makes its behavior harder to predict with a fixed set of test cases. It’s also rarely standalone.
Many enterprise deployments connect a model to internal documents, databases, or automation, so a weakness anywhere in that chain can expose what’s behind it. And the model itself is often trained or fine-tuned on organizational data, which raises its own set of exposure and intellectual property questions.
That combination is why generative AI security risks split naturally into the three layers below.
The prompt layer: where attacks get in
The prompt is often the most exposed part of any generative AI system, because it’s the one part a user or an outside attacker can directly touch.
Prompt injection
Prompt injection happens when someone embeds instructions inside input text to redirect a model away from its intended behavior. This can be direct (a user typing “ignore your previous instructions”) or indirect (hidden text inside a document, email, or webpage that the model reads and treats as a command).
The current OWASP Top 10 for LLM Applications (2026) still ranks prompt injection as the number one risk for LLM applications, a position it has held since the list’s first version. That consistency says something. Prompt injection isn’t a bug that gets patched once. It’s a structural property of systems that treat natural language as an instruction set.
- Prompt injection comes in two forms. Direct injection is a user typing something like “ignore your previous instructions” straight into a chat box.
- Indirect injection is more dangerous in practice: an attacker hides instructions inside a document, email, or web page, and the model picks them up the moment something else tells it to read that content.
Common defenses include separating instructions from untrusted content at the architecture level, filtering and sanitizing inputs and outputs, and training models to weight system-level instructions above anything found in retrieved or pasted content. None of these close the gap completely, which is why layered controls matter more than any single fix.
Sensitive information disclosure through prompts and outputs
The second common prompt-layer risk is simpler and more common in practice: a model surfacing information it shouldn’t, either because a user asked a cleverly worded question or because the system wasn’t built to distinguish what a given user is allowed to see.
Did you know?
OWASP’s 2026 revision broadened what used to be called “system prompt leakage” into a wider category covering tool schemas, internal policy logic, and other operational details, not just the system prompt text itself. The scope of what can leak through a prompt interaction keeps expanding as these systems get more capable.
The model layer: where the AI itself becomes the target
Once an attacker looks past the prompt, the model itself becomes the target. This is a different category of risk, because it targets the system’s training, integrity, or intellectual property rather than a single interaction.
Training data poisoning
If an attacker can influence the data a model is trained or fine-tuned on, they can influence its behavior at inference time, sometimes in ways that are hard to detect until the model is already in production.
Retrieval-augmented systems introduce a related version of this problem: poisoning the retrieval database rather than the model itself can skew what the model treats as ground truth.
Model theft and extraction
Model weights and architecture represent real intellectual property and, in some cases, a proxy for the sensitive data used to train them. Unauthorized access to a model, whether through a compromised API, an exposed endpoint, or a more exotic extraction technique, can hand a competitor or attacker something that took significant investment to build.
Hallucination-driven security failures
A hallucination isn’t an attack. It’s the model producing a plausible but incorrect answer with the same confident tone it uses for a correct one. That becomes a security problem the moment the output influences a real decision: a hallucinated compliance answer, a flawed remediation step, or a wrong configuration recommendation that someone acts on without checking.
Pro tip: Treat every generative AI output as untrusted input by default, the same way you’d treat data from an external API. Add human review for anything high stakes, and cross-check outputs against an authoritative source before they drive a decision that matters.
The data layer: the risk many frameworks gloss over
This is where many generative AI security guidance thins out. Prompt and model risks get detailed mitigation advice: filtering, rate limiting, monitoring, testing. Data risk usually gets a policy recommendation and a reminder to classify sensitive information.
That’s a real gap, because the data layer behaves differently from the other two.
Exposure that doesn’t depend on the model or the prompt
Prompt injection requires a malicious or careless input. Model poisoning requires access to training pipelines. But data exposure can happen even when the prompt is clean and the model is behaving exactly as designed, simply because the underlying data was never protected from the systems and people processing it.
An employee pasting a client contract into an AI tool, a retrieval system pulling from an improperly segmented database, or a fine-tuning job that touches more data than it should are all failures that have nothing to do with an attacker’s cleverness.
Cross-border and data sovereignty complications
For organizations in government, defense, healthcare, and financial services, this gets more complicated. Data often can’t move freely between countries, cloud providers, or organizations, whether for legal, contractual, or classification reasons.
A generative AI project that needs to train or run inference across that data hits a wall that no amount of prompt filtering solves, because the constraint isn’t about what the model says. It’s about whether the data can be touched at all outside its original environment.
Each reduces exposure in a different way: confidential computing processes data inside sealed hardware, homomorphic encryption works on data while it stays encrypted, and federated learning keeps records where they are and shares only model updates, which need extra protection of their own. That changes what’s possible for sectors where data can’t be pooled or moved in the first place. Duality runs LLM inference and secure RAG applications inside protected hardware environments, so sensitive data stays shielded while the model uses it.
Model risk management: connecting the three layers
Model risk management works best when it treats the prompt, model, and data layers as one connected surface rather than three separate checklists.
A control at one layer often supports the others: strong data access controls reduce what a successful prompt injection can actually reach, and output validation reduces the downstream damage of a poisoned or hallucinating model.
| Risk category | Primary layer | Typical control |
| Prompt injection | Prompt | Input validation, output filtering, guardrail testing |
| Sensitive information disclosure | Prompt / Data | Access controls, output redaction, data classification |
| Training data poisoning | Model | Data provenance checks, retrieval database monitoring |
| Model theft or extraction | Model | API authentication, rate limiting, endpoint monitoring |
| Hallucination-driven errors | Model | Human-in-the-loop review, cross-validation against source data |
| Sensitive data exposure during processing | Data | Encryption in use, data segmentation, privacy-enhancing technology |
| Cross-border and sovereignty constraints | Data | Federated architectures, jurisdiction-aware data controls |
A practical model risk management program covers governance (who approves AI use cases and what data they can touch), testing and validation (both for the model’s outputs and for how it handles adversarial input), ongoing monitoring (of prompts, outputs, and API usage), and data-layer controls that hold even if the first two fail.
Securing generative AI in regulated industries
Regulated industries face a version of this problem that general enterprise guidance doesn’t fully address.
Frameworks like the NIST AI Risk Management Framework and the EU AI Act give useful structure for governance and risk classification, and existing regulations like HIPAA and GDPR apply to AI systems that handle patient health information for healthcare providers, insurers and the companies working for them. None of them solve the underlying technical problem of using sensitive data for AI without exposing it.
The specifics differ by sector. Healthcare organizations answer to HIPAA and, in many cases, state-level health data laws on top of it, which means patient data used for AI training or inference needs the same protections it would need anywhere else in the organization.
Financial services already run mature model risk management programs under frameworks like SR 26-2, the Fed, OCC, and FDIC’s current guidance, which replaced SR 11-7 in 2026. SR 26-2 leaves generative and agentic AI out of scope, calls them novel and rapidly evolving, and points banks to their broader risk management practices.
Government and defense agencies face the strictest bar: FedRAMP authorization for cloud-hosted AI, CJIS requirements where criminal justice data is involved, and classification rules that can rule out cloud processing entirely for certain workloads.
That’s where the gap between compliance and actual security shows up. An organization can be fully compliant on paper, with the right policies and the right sign-offs, and still be one misconfigured retrieval pipeline away from a data exposure incident. Compliance tells you what’s required. It doesn’t make the data itself harder to expose.
Our experts’ advice: If your organization can’t legally or contractually centralize its data, don’t wait for policy to catch up before addressing the technical gap. Look at architectures that let you train, analyze, or run inference on the data where it already sits, under whatever jurisdiction it’s already subject to. That’s a smaller technical lift for many teams than it sounds, and it reduces the exposure itself, not only the policy around it.
Generative AI security vs. AI governance
These two terms get used interchangeably, and that’s a mistake. They cover different parts of the same problem.
AI governance sets the rules: which AI use cases are approved, what data can touch a given model, who signs off before something goes to production, and how risk gets classified against frameworks like the NIST AI RMF or the EU AI Act. Governance answers what should happen.
Generative AI security enforces those rules in real time and protects the system from things governance can’t fully anticipate on paper: a prompt injection attempt, a poisoned retrieval database, an exposed inference endpoint. Security answers what actually can’t be bypassed, regardless of what the policy says.
Shadow AI is a useful example of why the split matters. A governance policy can say employees may only use approved AI tools. That policy does nothing on its own to stop someone from pasting a client contract into an unapproved chatbot, because governance has no visibility into what’s actually happening at the point of use.
Security is what closes that gap: monitoring, access controls, and technical enforcement that hold whether or not everyone follows the policy.
A useful third piece some organizations add is compliance: the evidence that governance policies are actually being enforced, not just documented. A policy that says “no sensitive data in prompts” means little without a technical control and an audit trail proving it holds.
Treating governance, security, and compliance as one connected program, rather than three separate budget lines, is what closes the gap between what an organization intends and what its AI systems actually do.
