Sensitive Information Disclosure in AI: How LLMs Leak Your Data (OWASP LLM Top 10 #2)
AI systems can expose personally identifiable information, credentials, confidential business data, and proprietary content through their outputs. Sensitive Information Disclosure has risen to the #2 position in the 2025 OWASP Top 10 for LLM Applications, reflecting the growing recognition that the same capabilities that make language models useful also make them dangerous repositories and transmitters of sensitive data. This guide explains how AI data leakage happens, why it is so difficult to detect, and what your organization can do to protect the information entrusted to you.
Every time your organization feeds data into an AI system, you are making a trust decision. You are trusting that the information you provide, whether it is donor records, client intake forms, internal strategy documents, or financial data, will remain confidential and will not be exposed to unauthorized parties. Sensitive Information Disclosure, ranked #2 in the OWASP Top 10 for LLM Applications 2025, is what happens when that trust is violated. It covers every scenario in which an AI system reveals information that it should not, whether through its generated outputs, its training data memorization, or the data flows that surround its operation.
This vulnerability category moved from #6 in the original OWASP LLM Top 10 to #2 in the 2025 edition, a significant jump that reflects the growing number of real-world incidents in which AI systems have leaked confidential information. The rise in ranking is not because the vulnerability itself has changed. It is because organizations are now deploying AI in contexts where the data at stake is far more sensitive than what early chatbot experiments involved. When a nonprofit uses AI to process client case notes, analyze donor giving patterns, or draft grant applications that contain proprietary program data, the potential consequences of disclosure are far more severe than when AI was simply used to generate marketing copy.
This is the second article in our series covering the full OWASP Top 10 for LLM Applications. The first article on prompt injection examined how attackers manipulate AI systems by crafting malicious inputs. Sensitive Information Disclosure is closely related: prompt injection is often the mechanism through which an attacker triggers unauthorized data exposure. But information disclosure also occurs without any attack at all, through careless data handling, poor system architecture, or the inherent memorization behaviors of language models themselves.
In this article, we will explore the mechanisms through which AI systems leak sensitive information, examine the types of data most at risk, explain why traditional data loss prevention tools fail to catch AI-specific disclosure, and provide practical defense strategies that organizations can implement at multiple layers. Whether your organization is just beginning to explore AI or has already deployed AI-powered applications, understanding this vulnerability is essential for protecting the people and data entrusted to your care.
What Sensitive Information Disclosure Actually Is
Sensitive Information Disclosure in the context of AI and large language models refers to any situation where an AI system exposes data that should remain confidential. This includes personally identifiable information (PII) like names, addresses, Social Security numbers, and health records. It includes organizational secrets like strategic plans, financial projections, and proprietary methodologies. It includes technical credentials like API keys, database passwords, and internal system configurations. And it includes any other information whose exposure could cause harm to individuals or organizations.
What makes this vulnerability distinct from traditional data breaches is that the disclosure often happens through the AI's natural language generation process. A database breach typically involves an attacker exploiting a technical vulnerability to extract structured data in bulk. AI-based information disclosure, by contrast, can happen conversationally. A user asks a question, and the AI responds with information it should not have shared. The disclosure can be partial, contextual, and difficult to detect because it is embedded in otherwise legitimate-looking text. An AI assistant might answer a donor's question about their account and inadvertently include details about another donor's giving history. A chatbot trained on internal documents might reveal competitive strategy when a user asks a seemingly innocent question.
It is also important to understand that disclosure does not require a malicious attacker. Many of the most significant information disclosure incidents involve employees using AI tools in ways that expose sensitive data unintentionally. When Samsung engineers pasted proprietary source code into ChatGPT to debug it, they were not acting maliciously. They were trying to do their jobs more efficiently. But by feeding confidential code into a third-party AI system, they exposed it in ways that traditional data loss prevention never anticipated. Similar dynamics play out every day across organizations of all sizes, including nonprofits, where staff members paste client information into AI tools to draft emails, summarize case notes, or generate reports.
Traditional Data Breach vs. AI Information Disclosure
Traditional Data Breach
- •Exploits technical vulnerabilities (SQL injection, misconfigurations)
- •Extracts structured data in bulk
- •Usually detected by network monitoring and access logs
- •Clear forensic trail of what was accessed
AI Information Disclosure
- •Exploits the AI's learned knowledge and generation process
- •Leaks data conversationally, embedded in natural language
- •Often invisible to traditional security monitoring tools
- •Difficult to determine exactly what was disclosed and to whom
The distinction matters because it determines which defenses are effective. Organizations that rely solely on traditional security measures, firewalls, encryption at rest, access control lists, and network monitoring, may have a false sense of security when it comes to AI-based disclosure. The data does not leave your systems through a network exploit. It leaves through the AI's own outputs, often in a form that looks completely normal to traditional monitoring tools.
How Sensitive Information Disclosure Works in Practice
There are several distinct mechanisms through which AI systems disclose sensitive information. Understanding each one is critical because they require different defensive strategies. An organization that addresses only one mechanism while ignoring the others remains vulnerable.
Training Data Memorization and Extraction
The AI remembers sensitive data from its training and reproduces it on request
Language models learn by processing vast amounts of text data. During this process, they inevitably memorize portions of the training data verbatim, especially data that appears frequently or has distinctive patterns. Researchers have demonstrated that models like GPT can be prompted to reproduce email addresses, phone numbers, API keys, and even passages of proprietary code from their training data. This is not a bug in the traditional sense. It is a fundamental characteristic of how neural networks learn. The model does not "choose" to memorize sensitive data; memorization is a side effect of the learning process itself.
For organizations that fine-tune AI models on their own data, this risk is amplified significantly. If you train a model on donor records, client case files, or internal communications, those specific details can be extracted from the model later. Even if the model was fine-tuned to be helpful in general and was not intended to discuss specific records, the right sequence of prompts can cause the model to reproduce memorized data. This applies to both the base model's training data and any additional data used for fine-tuning.
- Models trained on smaller or highly specific datasets are more prone to memorization because data patterns are repeated more during training
- Extraction attacks use techniques like divergence prompting, where a repeated token causes the model to fall back on memorized training data
- Even models with safety training can be tricked into reproducing sensitive memorized content through carefully crafted prompt sequences
Conversational Context Leakage
Information from one conversation or user session bleeds into another
When AI systems process multiple users' requests, there is always a risk that context from one interaction leaks into another. This can happen at the application level when session management is poorly implemented, when conversation histories are shared across users, or when the AI system's context window contains information from prior interactions that it should not reference. In a multi-tenant AI application, where multiple organizations share the same deployed model, one organization's data could surface in responses to another organization's queries.
This risk is especially relevant for nonprofits using AI-powered customer relationship management tools or case management systems. If a social worker asks the AI about one client's situation and the system's context window still contains details from a previous client's case, the AI might blend those details into its response without any indication that it has crossed a confidentiality boundary.
- Shared context windows across users can inadvertently expose one user's information to another
- RAG systems that retrieve documents without proper access controls may pull confidential records into any user's session
- Conversation history features can retain and resurface sensitive information across sessions
Data Exposure Through User Inputs
Staff members inadvertently feed sensitive data into third-party AI systems
One of the most common and least technical vectors for AI-related data disclosure is simply employees pasting sensitive information into AI tools. When Samsung engineers pasted proprietary source code into ChatGPT, that code became part of the training data pipeline for future model versions. The same dynamic applies to any organization whose staff uses public AI tools for work tasks. A program manager who pastes client intake data into an AI to draft a summary, a development officer who feeds donor records into an AI to personalize solicitation letters, or an HR director who uploads employee performance reviews into an AI for analysis, each of these actions transmits sensitive data to a third party.
The challenge is that this behavior is incredibly common and difficult to prevent through technical controls alone. Staff members are often motivated by efficiency and may not understand the data handling implications of using public AI tools. They see the AI as a productivity tool, not as a third-party service that ingests and potentially retains everything they provide. This is why data privacy governance and clear AI usage policies are essential components of an organization's defense strategy.
- Data submitted to public AI tools may be used for model training unless explicitly opted out
- Even enterprise AI plans may retain data for limited periods for safety monitoring and debugging
- Once data is submitted to a third-party AI service, the organization typically cannot control its deletion or usage
System Prompt and Configuration Exposure
AI systems reveal their own configuration details, including credentials and business logic
AI applications frequently contain sensitive information within their system prompts, configuration files, and connected service credentials. System prompts may include database connection strings, API keys for third-party services, internal business rules that reveal competitive strategy, or access tokens that grant elevated privileges. When an attacker uses prompt injection techniques to extract the system prompt, they gain access not only to the AI's behavioral instructions but also to any credentials or configuration details embedded within them.
This is compounded by the fact that many developers embed sensitive values directly in system prompts for convenience, treating the system prompt as a secure configuration file when it is anything but. The system prompt is part of the AI's operational context and can be extracted through various techniques. Any credentials or sensitive configuration stored there should be considered compromised.
- API keys, database credentials, and access tokens embedded in system prompts can be extracted by attackers
- Business logic in system prompts can reveal proprietary processes and competitive advantages
- Connected tool configurations may expose internal network topology and service architecture
RAG and Knowledge Base Leakage
Retrieval-augmented generation systems expose documents beyond a user's authorization level
Many organizations deploy AI systems that use retrieval-augmented generation (RAG) to ground their responses in organizational knowledge. A RAG system works by retrieving relevant documents from a knowledge base and feeding them to the AI as context for generating a response. If the retrieval mechanism does not enforce proper access controls, the AI can pull documents that the requesting user should not have access to and include their contents in the response. A volunteer asking the AI about program schedules could receive a response that incorporates details from a board-only financial document that happened to mention those programs.
The challenge with RAG-based disclosure is that the AI does not indicate which source documents it used. The response appears seamless, blending authorized and unauthorized information into a single coherent answer. Without explicit source attribution and access control enforcement at the retrieval layer, users have no way to know that the AI's response includes information from documents they should not have seen.
- Document-level access controls are often bypassed at the embedding and retrieval layer
- Semantic similarity search retrieves documents based on content relevance, not authorization level
- AI responses blend information from multiple sources without clear attribution
Why Traditional Security Tools Fail
Organizations invest heavily in data loss prevention (DLP) systems, network monitoring, encryption, and access controls. These are valuable and necessary security measures, but they were designed for a world where data moves in predictable, structured ways. AI-based information disclosure breaks the assumptions that traditional tools rely on, creating blind spots that can leave organizations exposed even when they believe they are well-protected.
Traditional DLP tools work by scanning data in transit for patterns that match known sensitive data formats: credit card numbers, Social Security numbers, specific file types, or keyword patterns. When an AI system discloses sensitive information, it typically does so by paraphrasing, summarizing, or contextualizing the data rather than transmitting it verbatim. A DLP system scanning for the pattern "123-45-6789" will not flag an AI response that says "the client's social security number starts with 123 and ends in 6789." The information is disclosed, but not in a format that triggers pattern-based detection.
Network monitoring tools face a similar limitation. They can detect unusual data transfer volumes or connections to suspicious destinations, but AI-based disclosure happens through legitimate API calls and normal application traffic. When an employee sends a query to an AI service and receives a response that includes leaked data, the traffic pattern looks identical to any other AI interaction. There is no anomalous behavior to flag. The data loss happens at the semantic level, within the content of the AI's response, not at the network level.
Access controls, the foundation of most security architectures, are also insufficient on their own. Access controls determine who can query the AI system, but they do not control what the AI includes in its response. A user with legitimate access to the AI application may receive information that exceeds their authorization level because the AI draws on knowledge sources that span multiple access tiers. This is the core challenge: AI systems collapse access boundaries by processing information from diverse sources and generating unified responses that may blend authorized and unauthorized data.
This is precisely why organizations need specialized AI security testing that evaluates these AI-specific disclosure vectors. Testing methodologies designed for traditional web applications will not uncover the ways that language models can be coaxed into revealing sensitive information. AI security assessments use techniques specifically developed to probe for memorization, context leakage, access control bypasses, and the other mechanisms described in this article.
Who Is at Risk
Sensitive Information Disclosure affects any organization that uses AI systems in contexts where confidential data is processed, stored, or referenced. The risk level depends on the type of AI deployment, the sensitivity of the data involved, and the access controls in place. The following categories of AI applications are particularly vulnerable.
Customer-Facing Chatbots
AI chatbots that handle donor inquiries, client intake, or general information requests. These systems often have access to broader knowledge bases than any single user should see. A donor-facing chatbot connected to a CRM system could inadvertently disclose another donor's giving history, contact details, or communication preferences.
Document Processing Systems
AI tools that summarize, analyze, or search through organizational documents. When these systems process documents with varying confidentiality levels, they can surface restricted information in summaries or search results. Grant applications, board minutes, personnel files, and client records may all feed into the same knowledge base.
AI-Powered Internal Tools
Internal AI assistants that help staff with tasks like drafting communications, analyzing data, or generating reports. These tools often have broad access to internal systems and data, and any user query could trigger the AI to include information from data sources that user should not access. AI agents that connect to multiple systems amplify this risk further.
Fine-Tuned or Custom Models
Organizations that train or fine-tune models on their own data face the highest risk of training data memorization. The model literally learns from your sensitive data, and that knowledge can be extracted through targeted prompts. Smaller fine-tuning datasets increase the memorization risk because the model sees each data point more frequently during training.
Why This Matters More for Nonprofits
Nonprofits often handle some of the most sensitive categories of personal data: health information for healthcare nonprofits, immigration status for refugee services, housing instability for homeless services, abuse histories for domestic violence shelters, and financial hardship details for economic assistance programs. The people served by nonprofits are frequently in vulnerable situations, and the exposure of their information can cause harm that goes beyond financial loss. It can endanger their safety, undermine their trust in organizations they depend on, and violate regulatory requirements including HIPAA, FERPA, and state privacy laws.
At the same time, nonprofits typically operate with smaller technology teams, fewer dedicated security resources, and tighter budgets than their for-profit counterparts. This combination of highly sensitive data and limited security capacity makes the nonprofit sector particularly vulnerable to AI-based information disclosure. The organizations that stand to benefit most from AI efficiency gains are also the ones most at risk when AI systems mishandle the data entrusted to them.
Defense Strategies: A Layered Approach
Protecting against Sensitive Information Disclosure requires defenses at multiple layers, from the data that enters your AI systems to the outputs they produce and the policies that govern their use. No single control is sufficient on its own. A comprehensive defense strategy addresses the problem from the ground up, starting with the most fundamental controls and building toward more sophisticated protections.
Layer 1: Data Governance and Minimization
Control what data the AI system can access in the first place
The most effective way to prevent an AI system from disclosing sensitive information is to ensure it never has access to that information in the first place. Data minimization, the practice of limiting AI system access to only the data strictly necessary for its function, is the foundational defense layer. Before deploying any AI application, conduct a thorough assessment of what data the system will access and whether each data source is truly necessary for the intended functionality.
- Inventory all data sources that your AI systems can access and classify them by sensitivity level
- Apply data masking and anonymization before feeding sensitive data into AI systems for training or retrieval
- Implement strict data retention policies for AI interactions, including automatic deletion of conversation histories containing sensitive data
- Separate high-sensitivity data stores from general-purpose AI knowledge bases entirely
- If you fine-tune models on organizational data, sanitize all training data to remove PII, credentials, and confidential identifiers before training begins
Layer 2: Access Controls and Authorization
Enforce user-level permissions at every layer of the AI application
Traditional access controls determine who can access the AI system, but for AI applications, you also need to control what the AI can access on behalf of each user. This means implementing authorization at the retrieval layer (in RAG systems), at the tool/action layer (for AI agents), and at the output layer (filtering responses based on the requesting user's permissions). The principle of least privilege applies not just to human users but to AI systems themselves.
- Implement document-level access controls in RAG systems so retrieval respects user authorization boundaries
- Use separate AI deployments or contexts for different sensitivity levels rather than a single system with broad access
- Enforce session isolation so that one user's conversation context is never accessible to another user
- Limit the AI system's own permissions to only the data and tools required for its current task, following zero trust principles
Layer 3: Output Monitoring and Filtering
Scan AI outputs for sensitive data before they reach the user
Even with strong input controls and access management, the AI may still generate outputs that contain sensitive information. Output filtering provides a final defense layer by scanning AI responses before they are delivered to the user. This requires purpose-built tools that go beyond traditional DLP pattern matching. Effective AI output monitoring needs to understand the semantic content of responses, not just look for specific data patterns, because AI systems paraphrase and contextualize information rather than reproducing it verbatim.
- Deploy AI-aware output scanning that can detect sensitive information even when paraphrased or partially disclosed
- Implement automated redaction for known sensitive data categories (PII, credentials, financial details) in AI outputs
- Log all AI interactions for audit purposes, enabling retroactive detection and investigation of disclosure incidents
- Establish alerting thresholds for suspicious patterns, such as repeated queries that appear designed to extract specific information
Layer 4: Organizational Policies and Training
Establish clear rules for how staff interact with AI systems and handle data
Technical controls are only effective when paired with organizational policies that govern how people interact with AI systems. Many information disclosure incidents stem not from technical failures but from staff members using AI tools in ways that expose sensitive data. Clear policies, regular training, and organizational awareness are essential for closing these human-factor gaps. The goal is not to restrict AI usage but to ensure that AI usage happens within boundaries that protect sensitive data.
- Create clear, specific AI usage policies that define what data can and cannot be shared with AI tools, with concrete examples relevant to each department's work
- Provide approved, organization-sanctioned AI tools that have appropriate data privacy protections in place, reducing the temptation to use uncontrolled public tools
- Train staff on the specific risks of sharing sensitive data with AI systems, including the distinction between enterprise and consumer AI service tiers
- Establish an incident response process specifically for AI-related data disclosure, including how to report suspected incidents and what remediation steps to take
Common Mistakes Organizations Make
Even organizations that take AI security seriously often make mistakes that leave them vulnerable to Sensitive Information Disclosure. These mistakes typically stem from underestimating how AI systems process and expose data, or from applying traditional security thinking to fundamentally new risks.
Relying on System Prompt Instructions to Prevent Disclosure
Many organizations add instructions to their AI's system prompt telling it not to disclose certain types of information: "Never share donor financial data," "Do not reveal personal information about clients," or "Keep internal strategy confidential." While these instructions provide some behavioral guidance, they are not security controls. System prompt instructions can be bypassed through prompt injection, they are inconsistently enforced across different query phrasings, and they provide no guarantee that the AI will comply in all situations. Treating system prompt instructions as your primary defense against information disclosure is like putting a "Please do not enter" sign on an unlocked door.
Trusting That Enterprise AI Plans Eliminate All Risk
Enterprise tiers of AI services like ChatGPT Team, Copilot for Business, and Claude for Enterprise typically promise not to use your data for model training. This is an important protection, but it does not eliminate Sensitive Information Disclosure risk. The AI can still memorize and reproduce data within a session, conversation histories may be retained for safety monitoring, and the fundamental issue of AI-based context leakage exists regardless of the service tier. Enterprise plans reduce one specific vector (training data incorporation) while leaving others unaddressed.
Failing to Test AI Systems for Information Disclosure
Organizations frequently deploy AI applications after testing for functional correctness but not for security. They verify that the AI produces helpful and accurate responses but never systematically test whether it can be induced to disclose sensitive information. This is understandable, as testing for information disclosure requires specialized knowledge of extraction techniques and adversarial prompting, but it leaves a critical gap. A professional AI application security assessment specifically tests for these disclosure vectors using techniques designed to probe what the AI knows and what it can be tricked into revealing.
Overlooking Data Flows in RAG and Agent Architectures
When organizations build RAG systems or deploy AI agents that interact with multiple data sources, they often focus on the AI model's behavior while overlooking the data flows surrounding it. The retrieval pipeline, the tool integrations, the logging systems, and the response caching mechanisms all represent potential points where sensitive data can be exposed. A secure AI model connected to an insecure retrieval pipeline is still vulnerable. Every component in the data flow needs to enforce the same access controls and data handling policies as the front-facing application.
What a Professional Assessment Covers
A comprehensive AI Application Security assessment evaluates your organization's exposure to Sensitive Information Disclosure across all the vectors described in this article. Here is what a thorough assessment examines.
Training Data Extraction Testing
Systematically probing the AI model for memorized sensitive data using divergence attacks, completion prompting, and other extraction techniques. Determining whether the model can be induced to reproduce PII, credentials, proprietary content, or other confidential data from its training set.
Context Isolation Verification
Testing whether information from one user's session can leak into another user's session through shared context, conversation history, or cached responses. Evaluating multi-tenant isolation in environments where multiple organizations or user groups share the same AI deployment.
RAG Access Control Assessment
Evaluating whether the retrieval layer enforces document-level access controls. Testing whether users can extract information from documents they should not have access to through carefully phrased queries that trigger retrieval of restricted content.
Configuration and Credential Review
Examining system prompts, environment configurations, and connected tool credentials for sensitive values that could be exposed through prompt extraction techniques. Verifying that credentials are stored securely and not accessible through the AI's response generation process.
Output Filtering Evaluation
Testing the effectiveness of any output filtering or redaction mechanisms in place. Determining whether sensitive data can bypass filters through paraphrasing, encoding, or partial disclosure techniques. Evaluating whether monitoring and logging systems capture enough detail to detect and investigate disclosure incidents.
Data Flow and Architecture Review
Mapping the complete data flow from user input through retrieval, processing, generation, and output delivery. Identifying points where sensitive data could be logged, cached, or transmitted without appropriate protections. Evaluating whether the architecture design minimizes the blast radius of a disclosure incident.
The Value of Systematic Assessment
Information disclosure risks are often subtle and interconnected. A system that appears secure when tested casually may reveal significant vulnerabilities under systematic adversarial testing. Professional assessment brings the specialized knowledge needed to test for extraction techniques, access control bypasses, and architectural weaknesses that are unique to AI applications. It also provides prioritized remediation guidance so your team knows which fixes deliver the most risk reduction for the least effort.
For organizations handling regulated data, including health information under HIPAA, educational records under FERPA, or personal data under state privacy laws, a comprehensive AI security review can also demonstrate due diligence in protecting sensitive data, which is increasingly relevant as regulators turn their attention to AI-related data handling practices.
The OWASP Top 10 for LLM Applications: Full Series
This article is part of our comprehensive series covering every vulnerability in the OWASP Top 10 for LLM Applications. Each article provides a deep dive into a specific risk category with practical defenses for your organization.
Prompt Injection
Published: February 25, 2026
Sensitive Information Disclosure
You are here
Supply Chain Vulnerabilities
Coming soon
Data and Model Poisoning
Coming soon
Insecure Output Handling
Coming soon
Excessive Agency
Coming soon
System Prompt Leakage
Coming soon
Vector and Embedding Weaknesses
Coming soon
Misinformation
Coming soon
Unbounded Consumption
Coming soon
Protecting What People Trust You With
Sensitive Information Disclosure sits at the #2 position in the OWASP Top 10 for LLM Applications because it strikes at the heart of what makes AI both powerful and dangerous: its ability to process, understand, and generate content based on the data it has access to. Every time an AI system produces a response, it is drawing on its training data, its context window, and its connected data sources. If any of those sources contain information that should not be exposed to the requesting user, the AI may disclose it without any awareness that it is crossing a confidentiality boundary.
For nonprofits, the stakes are particularly high. The data entrusted to your organization represents the lives, circumstances, and vulnerabilities of real people. Donors trust you with their financial information. Clients trust you with their most personal challenges. Staff trust you with their employment records. When an AI system leaks this information, the harm extends beyond the technical incident. It breaks the trust that is essential to your mission and your relationships with the communities you serve.
The good news is that Sensitive Information Disclosure is addressable through a layered defense strategy. Data minimization reduces the blast radius of any potential disclosure. Access controls at the retrieval and output layers prevent unauthorized information from reaching users. Output monitoring catches disclosure attempts that bypass other controls. And clear organizational policies close the human-factor gaps that no technical solution can fully address. No single layer is sufficient, but together they create a defense in depth that significantly reduces your risk.
The most important step you can take is to understand your current exposure. What data do your AI systems have access to? What controls are in place to prevent unauthorized disclosure? Have those controls been tested against the specific extraction techniques that AI systems are vulnerable to? If you are not confident in the answers to these questions, your organization is operating with unknown risk. An AI security assessment can provide the clarity you need to make informed decisions about your AI deployment strategy and the data you put at stake.
Is Your AI Exposing Sensitive Data?
Sensitive Information Disclosure is the #2 vulnerability in the OWASP Top 10 for LLM Applications, and traditional security tools cannot detect it. Our AI Application Security assessments test your systems for training data memorization, context leakage, RAG access control bypasses, and all other disclosure vectors covered in this article.
Start with a free consultation to understand your current exposure and the right assessment scope for protecting your organization's sensitive data.