LLMs and Data Privacy: How to Protect Sensitive Information

Your legal team just approved a new AI project. The model is powerful, the use case is clear, and everyone is excited. Then someone asks the question nobody wants to answer: “What happens to the sensitive data we feed into this thing?”

That moment of silence is familiar to almost every enterprise AI team right now. LLMs are genuinely transformative tools, but their relationship with sensitive data is complicated in ways that most vendors gloss over. The risks are real, they are not always obvious, and the regulatory consequences of getting it wrong are getting steeper by the year.

Working at the intersection of privacy-enhancing technologies and enterprise AI, we have seen organizations at every stage of this journey, from those who discovered the risks before deployment to those who discovered them afterward.

This article covers what you actually need to know: where the risks sit, how the main privacy technologies work, and what a solid LLM data privacy strategy looks like in practice.

What Are the Biggest Data Privacy Risks with LLMs?

LLM data privacy is not one problem. It is a cluster of related problems that show up at different points in the AI lifecycle. Understanding where each risk lives is the first step to addressing it.

• Training data exposure is the most widely discussed risk. LLMs are trained on enormous datasets, and that data often contains personally identifiable information (PII), credentials, proprietary content, and other sensitive material.
  
  Research from Truffle Security published in 2025 found approximately 12,000 live API keys and passwords in a single major training dataset crawl, with 63% of those secrets appearing across multiple pages. That is not a theoretical risk. It is data that models can and do reproduce.

• Inference-time leakage is a different category of problem. This happens when a model generates outputs that reveal information from prior conversations, internal documents, or fine-tuning data it was exposed to. In enterprise settings with RAG (retrieval-augmented generation) architectures, this is a particularly acute concern.
  
  An improperly scoped RAG system can surface confidential customer records, internal financial data, or legal documents in response to a question it was never meant to answer.

• Prompt injection and adversarial extraction involve attackers crafting inputs specifically designed to trick an LLM into revealing information it would not normally share. These attacks have become increasingly sophisticated, and standard output filtering often does not catch them reliably.

• Third-party API risk applies to organizations using cloud-hosted LLMs. When sensitive data is sent to an external API for inference, questions about data retention, logging, and cross-customer data handling become compliance issues, not just technical ones.

Each of these risks sits in a different layer of the stack, which is why a single-layer defense never works well.

How Do LLMs Memorize and Leak Sensitive Data?

This is the question that surprises most non-technical stakeholders. People intuitively assume that training a model is like averaging data into a statistical blur. The reality is more nuanced and more concerning.

During training, an LLM encodes patterns from its training corpus into billions of numerical parameters (weights). For the vast majority of data, this encoding is indeed a lossy compression. But for data that appears repeatedly, or that appears in contexts the model finds “memorable” (highly distinctive, low-perplexity sequences), the encoding can be close to verbatim.

Research has consistently shown that larger models memorize more. As model size increases, the probability of verbatim reproduction of training examples grows with it. Emails, phone numbers, addresses, and even fragments of private correspondence have been demonstrated to leak from models trained on web-scale datasets.

Fine-tuning introduces a separate and underappreciated risk. When an organization fine-tunes a base model on its own data, that proprietary data can become memorized in the fine-tuned weights.

Unlike pre-training memorization (where the data came from public sources), fine-tuning memorization typically involves the organization’s own confidential information: customer records, clinical notes, internal communications, or financial data.

It is also worth understanding inference leakage, which is subtler than memorization. A model does not need to reproduce text verbatim to leak sensitive information. Statistical patterns in the training data can allow a sufficiently motivated attacker to infer private details about individuals or groups by probing the model with carefully constructed prompts.

This is sometimes called a membership inference attack, and it is a genuine concern for models trained on sensitive datasets.

How Can You Use LLMs Without Exposing PII or PHI?

The good news is that there are practical, proven approaches to reducing PII and PHI exposure in LLM workflows. None of them is a silver bullet on its own, but in combination they provide real protection.

• Input scrubbing and PII masking is often the first line of defense. Before any data enters an LLM pipeline, automated detection systems identify and redact or replace sensitive entities: names, dates of birth, social security numbers, medical record numbers, account numbers, and so on.
  
  Named entity recognition (NER) models handle this detection, and the replacements can be consistent pseudonyms (so context is preserved) or simple blanks.

The limitation of this approach is that it is only as good as the detection model. Novel or context-dependent PII (a patient described by their unique combination of conditions rather than their name, for example) can slip through.

• Role-based access controls and prompt-level scoping matter enormously in RAG architectures. If the retrieval layer can surface documents that a given user should not have access to, the LLM becomes an inadvertent disclosure mechanism. Proper access controls at the retrieval layer are not optional; they are fundamental.

• Output filtering provides a second pass, screening LLM outputs for patterns that match sensitive data formats before they are returned to the user. This catches some leakage that input scrubbing misses, but it is reactive rather than preventive.

• Encrypted inference using privacy-enhancing technologies like fully homomorphic encryption (FHE) represents the more robust end of the spectrum. With FHE, computations run on encrypted data, meaning sensitive information is never exposed to the model in plaintext.
  
  This is technically demanding but increasingly practical for enterprise deployments, particularly in financial services and healthcare.

For PHI specifically, organizations operating under HIPAA need a combination of these approaches plus formal business associate agreements with any third-party LLM providers, audit logging, and documented data handling procedures.

What Is the Difference Between Data Anonymization and Differential Privacy for LLMs?

This distinction matters practically, because the two approaches offer very different levels of protection and are suited to different stages of the LLM pipeline.

• Data anonymization means removing or transforming identifying information from a dataset so that individuals can no longer be identified.
  
  In practice this involves removing names, replacing exact ages with ranges, generalizing locations, and so on. It sounds straightforward, and for many years it was considered the gold standard for privacy-protecting datasets.

The problem is that modern AI models, including LLMs, are extraordinarily good at re-identification. A 2025 European Data Protection Board report explicitly noted that large language models rarely achieve true anonymization standards.

When a model is trained on supposedly anonymized data and then queried with auxiliary information, it can often reconstruct identities that the anonymization was supposed to protect. The combination of attributes that seemed innocuous individually can uniquely identify a person.

• Differential privacy (DP) takes a mathematically rigorous different approach. Instead of trying to remove identifying information, DP adds carefully calibrated statistical noise to the training process itself.
  
  This noise is designed so that no individual data point can be reliably attributed to the model’s outputs, while the aggregate patterns (which is what makes the model useful) are preserved.

The formal guarantee is this: the probability of any particular model output should be roughly the same whether or not any specific individual’s data was included in training. This is provable mathematics, not an empirical claim.

The trade-off is utility. Adding noise to the training process degrades model performance, and the strength of the privacy guarantee (expressed as the epsilon value) inversely correlates with model accuracy.

Tight privacy (epsilon near 1) produces a significantly less capable model. Most enterprise deployments find a practical middle ground around epsilon 8 to 10, which offers meaningful protection without unacceptable performance degradation.

For most enterprise LLM deployments, anonymization is a useful preprocessing step, but differential privacy is the more rigorous technical guarantee. They are not alternatives; they work best together.

How Do Privacy-Enhancing Technologies Protect Data Used by LLMs?

Privacy-enhancing technologies (PETs) are a family of cryptographic and statistical techniques designed to allow computation on sensitive data without exposing that data. Each one addresses a different aspect of the LLM data privacy problem.

• Fully homomorphic encryption (FHE) is arguably the most powerful. It allows a model to perform computations on encrypted data and return an encrypted result, meaning the raw data is never decrypted during processing.
  
  For LLM inference, this means sensitive queries and documents can be processed without the model (or the infrastructure running it) ever seeing the plaintext.
  
  Duality Technologies is a leading developer of FHE technology, having co-created OpenFHE, the widely used open-source FHE library that Google has also integrated into its cryptographic infrastructure.

• Secure multi-party computation (SMPC) allows multiple parties to jointly compute a function over their combined data without any party seeing the other parties’ inputs. This is particularly relevant for collaborative AI use cases: two competing banks wanting to train a shared fraud-detection model, for instance, without either institution exposing its customer data to the other.

• Trusted execution environments (TEEs) are hardware-level security enclaves, such as Intel SGX or AMD SEV, that create isolated, encrypted processing environments.
  
  Code running inside a TEE cannot be inspected or tampered with, even by the host operating system or cloud provider. For LLM inference, TEEs allow sensitive data to be processed in an environment that is verifiably isolated from external access.

• Federated learning (covered in detail below) allows model training to happen on distributed data without centralizing that data.

The practical reality for enterprise AI deployments is that no single PET covers every attack surface. FHE addresses inference privacy. Federated learning addresses training data centralization. TEEs address infrastructure trust.

A genuinely secure LLM deployment typically combines two or three of these approaches depending on the threat model.

Can You Fine-Tune LLMs on Sensitive Data Safely?

Yes, but the default approach of simply uploading your data to a fine-tuning API and letting it run is not safe for sensitive information. Fine-tuning on sensitive data requires deliberate technical controls.

• Differential privacy training is the most widely adopted safeguard. Adding DP to the fine-tuning process bounds the influence any individual data point can have on the final weights. This directly limits the risk of verbatim memorization of specific sensitive records.

• Federated fine-tuning allows a model to be updated using data that never leaves its original location. Instead of sending data to a central training server, each participating node computes gradient updates locally and sends only those updates (not the raw data) to be aggregated.
  
  This is particularly relevant in healthcare, where patient data cannot practically or legally be centralized.

• Synthetic data generation is increasingly used as a preprocessing step. Rather than fine-tuning on real customer or patient records, organizations generate a synthetic dataset that has the same statistical properties as the real data but does not contain actual individuals’ information.
  
  The model is then fine-tuned on the synthetic data. This approach has improved significantly with better generative models and is now a practical option for many use cases.

• Trusted execution environments for fine-tuning ensure that training computations happen in an isolated environment where even the cloud provider cannot observe the data or intermediate model states.

It is also worth noting the data residency question. For organizations subject to GDPR or sector-specific regulations, fine-tuning on a cloud provider’s infrastructure may constitute a data transfer with compliance implications, depending on where the infrastructure is located and how the provider processes data.

What Does GDPR Require for LLM Deployments?

GDPR compliance for LLM deployments is one of the more complex areas of current data protection law, in part because the regulation predates large-scale generative AI and some of its requirements require interpretation when applied to LLM-specific contexts.

The core principles are clear. Data must be processed lawfully, fairly, and transparently. Only data necessary for the specified purpose may be processed (data minimization).

Personal data must be accurate, stored only as long as needed, and processed securely. The organization deploying the LLM is accountable for all of this.

Several GDPR provisions create specific operational challenges for LLM deployments.

• The right to erasure (Article 17) requires that an organization be able to delete an individual’s personal data on request. For a traditional database, this is straightforward. For an LLM whose weights may have encoded information from that individual’s data during training, it is technically challenging.
  
  Current approaches include machine unlearning techniques, full model retraining (expensive but definitive), and output filtering that prevents the model from generating outputs related to the erased individual’s data.

• Data Protection Impact Assessments (DPIAs) are mandatory under Article 35 for processing that poses high risk to individuals’ rights. Deploying an LLM that processes personal data at scale almost certainly meets this threshold. A DPIA must be conducted before deployment, not after.

• Automated decision-making under Article 22 restricts decisions based solely on automated processing when those decisions have legal or similarly significant effects. LLM-based systems used in hiring, credit, insurance, or similar contexts need human oversight mechanisms built in.

• The lawful basis question is particularly thorny for LLM training. Organizations typically rely on legitimate interests (Article 6(1)(f)), but the 2025 EDPB guidance has clarified that this requires a genuine balancing test and that LLMs trained on scraped public data are not automatically compliant just because the data was technically public.

Cumulative GDPR fines reached €5.88 billion by 2026. This is not a compliance area where organizations can afford to take a casual approach.

How Does Federated Learning Help with LLM Privacy?

Federated learning (FL) is one of the most practically significant privacy-enhancing technologies for enterprise LLM deployments, particularly in regulated industries.

The core idea is simple: instead of bringing data to the model for training, you bring the model to the data. Each participating node (a hospital, a bank branch, a device) trains the model locally on its own data, then sends the resulting model updates (gradients) to a central aggregator.

The aggregator combines these updates to improve the global model, then distributes the updated model back to participants. At no point does raw data leave its original location.

This architecture addresses one of the most fundamental LLM data privacy risks: the centralization of sensitive data in a training pipeline. In traditional centralized training, creating a single large training dataset means aggregating sensitive records from multiple sources into one location, which creates both a compliance burden and a high-value target for attackers.

Federated learning is not a complete solution on its own. Research has demonstrated that gradient updates themselves can leak information about the underlying training data under certain conditions. These gradient inversion attacks represent a real risk in FL deployments.

The standard mitigation is to combine federated learning with differential privacy at the gradient level. Before a participant sends gradient updates to the aggregator, noise is added to those gradients that provide DP guarantees, limiting what can be inferred about the participant’s local data.

Secure aggregation protocols add a further layer: the aggregator can compute the sum of participants’ updates without seeing individual participants’ updates, using cryptographic techniques. This means even a compromised aggregation server cannot reconstruct individual contributions.

For industries like healthcare, financial services, government, insurance, and data service providers, federated learning with differential privacy and secure aggregation represents a genuinely practical path to training useful LLMs on sensitive data without centralizing that data or violating regulatory requirements. 

What Does a Strong LLM Data Privacy Strategy Actually Look Like?

LLM data privacy is not a problem you can solve with a terms-of-service clause or a blanket API disclaimer. It requires real technical controls at every stage of the AI pipeline: training, fine-tuning, inference, and retrieval.

The technologies exist to deploy LLMs on genuinely sensitive data safely. Differential privacy, federated learning, fully homomorphic encryption, and trusted execution environments each address different attack surfaces. Most robust deployments use two or three in combination.

Regulatory requirements are also tightening. GDPR enforcement around AI processing is intensifying, and the EDPB’s 2025 guidance makes clear that “we used public data” and “we tried to anonymize it” are not sufficient defenses for LLM deployments handling personal data.

The organizations that get this right are not waiting for regulators to force their hand. They are building privacy into the architecture from the start, because that is what genuine trust requires.

How Can Duality Help You Deploy LLMs with Genuine Data Privacy?

Most organizations building LLM applications are not cryptographers. They should not need to be. But they do need infrastructure that gives them real privacy guarantees, not just compliance checkboxes.

Duality Technologies was founded by world-renowned cryptographers and data scientists to make privacy-enhancing technologies practical for enterprise use. The secure data-collaboration platform operationalizes FHE, federated learning, and TEEs as production-ready tools, not research prototypes.

For LLM deployments, that means private inference, secure fine-tuning, and privacy-preserving RAG, all available within trusted execution environments on Google Cloud Confidential A3 infrastructure. The platform integrates with Microsoft Azure, Intel, IBM, and DARPA, giving enterprise teams the partnerships and governance controls that regulated industries demand.

If data privacy is a real constraint for your LLM project, not just a theoretical one, Duality is the practical next step.