Ethical AI Procurement: Questions Every Nonprofit Should Ask AI Vendors
Before you sign any AI vendor contract, your organization deserves clear answers about how your data will be used, how bias is tested and monitored, what happens when things go wrong, and what rights you retain. This guide gives you the framework and specific questions to ask.
More than 85% of nonprofits have now explored AI tools, yet only 9% feel ready to adopt AI responsibly. This gap tells a revealing story: the speed of AI adoption in the sector has significantly outpaced the rigor of how organizations evaluate the vendors they're trusting with sensitive data and mission-critical operations.
The stakes for nonprofits are uniquely high. Your organization likely holds some of the most sensitive data in any sector: beneficiary health records, immigration status, financial hardship details, mental health histories, and housing instability information. You hold donor giving histories with personal significance. You operate under public trust that took years to build. When you hand any piece of that data ecosystem to an AI vendor, you are making a governance decision, not just a technology purchase.
The 2026 regulatory environment has added another layer of urgency. The EU AI Act, new U.S. federal guidance on AI procurement, and a wave of state-level algorithmic accountability laws all point in the same direction: the days of informal AI adoption are ending. Funders, boards, and regulators are beginning to ask what due diligence nonprofits conducted before deploying AI tools that affect people.
Yet most nonprofits still approach AI vendor selection the same way they might evaluate a new project management tool: a free trial, a few demos, a quick comparison of pricing tiers, and a signature on a terms-of-service agreement that almost nobody reads carefully. This guide is designed to help your organization do better, without requiring a legal department or a dedicated IT security team to do it.
What follows is a structured due diligence framework organized around the domains that matter most: how vendors handle your data, how transparent they are about how their AI works, how they test for and address bias, what contract terms you should require, and how to identify red flags that should give you pause before proceeding. This isn't about creating bureaucratic barriers to innovation. It's about making sure the AI tools you adopt actually serve your mission rather than quietly undermining it.
Start with Risk Classification, Not Vendor Comparison
The most common mistake organizations make in AI procurement is jumping straight to vendor comparison before asking a more fundamental question: what level of risk does this particular AI use case carry? Not all AI tools require the same depth of scrutiny, and treating every tool as equally high-risk leads to either analysis paralysis or, more commonly, a frustrated team that shortcuts the process entirely.
A practical approach is to classify every intended AI use by three factors: whether the AI makes or influences decisions affecting people, whether it processes sensitive personal data, and what the consequence of an incorrect or biased AI output would be. These three questions will usually tell you which tier of due diligence applies.
Tier 1: Lower Risk
Basic review required
- Productivity tools (meeting notes, scheduling)
- No personally identifiable information involved
- All outputs reviewed by humans before use
- Easy to reverse or override
Tier 2: Medium Risk
Full questionnaire required
- PII is processed or stored
- Outputs inform decisions about people
- Donor or staff data involved
- Moderate reputational exposure if system fails
Tier 3: Higher Risk
Deep review and legal sign-off
- Decisions affecting service eligibility or delivery
- Sensitive beneficiary data (health, legal status, mental health)
- Hiring or HR-related AI functions
- Regulatory exposure or grant compliance implications
Once you've classified the risk level, you know how much due diligence is appropriate. A Tier 1 tool might warrant a quick privacy policy review and a security certification check. A Tier 3 tool deserves a full vendor questionnaire, independent bias documentation review, legal review of the contract, executive sign-off, and a post-deployment monitoring plan. Matching the depth of scrutiny to the level of risk protects your organization without creating unnecessary friction.
Questions to Ask About Data Practices
How a vendor uses your data is the most fundamental question in AI procurement. The answers reveal not just technical practices, but the company's fundamental values around your organization's information and the people it serves.
Training Data and Model Use
- What data was used to train this model, and how was it sourced?
- Will our data (inputs, outputs, fine-tuning) be used to train your models or improve products for other customers?
- Do you have a zero-retention policy for inference data, meaning prompts and outputs are not logged after the session?
- How is this controlled contractually, not just in your privacy policy?
Data Storage and Sub-Processors
- Where is our data stored? Which geographic regions and cloud providers?
- Who are your sub-processors? Will you provide a full list and notify us before adding new ones?
- What is your data retention schedule, and how is data deleted at contract end?
- Can you guarantee logical isolation of our data from other customers?
These questions matter more for nonprofits than for most commercial organizations. When your beneficiary data includes immigration status, mental health history, or financial hardship details, the standard business risks associated with a data breach become risks to real people with real vulnerabilities. You owe it to the communities you serve to know exactly where their information goes and who else has access to it.
The sub-processor question is frequently overlooked, yet it can be where significant risk hides. An AI vendor may have reasonable data practices, but rely on a third-party infrastructure provider or analytics tool that doesn't. Requiring a full sub-processor list and advance notice of changes ensures you maintain visibility into the entire data supply chain, not just your direct relationship with the vendor.
Data portability and deletion guarantees at offboarding are also worth securing before you sign. A vendor who is vague about what happens to your data when you leave the relationship is a vendor who may have different incentives than they've represented. Always ask: how do we get our data back, in what format, and what happens to copies of it after we leave?
Transparency and Explainability: What Should Vendors Be Able to Show You
Transparency is not just a philosophical value in AI procurement; it is a practical safeguard. When an AI system produces an unexpected or harmful output, you need to be able to understand why it happened, communicate about it credibly to stakeholders, and fix it. That's impossible if you've signed on to a black box.
Documentation That Responsible Vendors Provide
A credible AI vendor should be able to provide all of the following without hesitation.
- A model card or system card documenting training data, intended use, known limitations, and performance benchmarks
- A plain-language explanation of how the model generates its outputs
- The version of the model you'll be using and a change notification policy
- Published bias evaluation results and the methodology used
- A responsible AI policy that is publicly accessible
- A named individual or team accountable for AI ethics decisions
Model cards and system cards are the equivalent of nutrition labels for AI systems. They tell you what the model was trained on, what it's designed to do, where it performs well, and where it's known to struggle. A vendor that hasn't produced this documentation either hasn't thought carefully about their model's limitations or doesn't want you to know about them. Neither is reassuring.
The model change notification question is particularly important for nonprofits with compliance obligations. If you've vetted a specific version of a model for use with beneficiary data, and the vendor silently updates to a new version with different behavior, your due diligence is retroactively invalidated. Require advance notice of material changes in writing, not just in a changelog blog post you'd have to find on your own.
Watch for vendors who describe their AI as "explainable" without providing any actual explanation. The term has become a marketing buzzword. What you actually want to know is whether the model can surface reasoning for its outputs in your specific use case, and whether that reasoning is accurate rather than a post-hoc rationalization. Ask for a demonstration with your actual data types.
Bias and Fairness: What "We've Tested for Bias" Actually Means
Nearly every AI vendor will tell you their system has been tested for bias. What varies enormously is what that actually means. Vague assurances of fairness are not due diligence. They are marketing. Your job is to ask specific enough questions that you can evaluate whether the vendor's approach is credible for your specific use case and the populations your organization serves.
Specific Bias Questions to Ask
These questions distinguish vendors who have done genuine fairness work from those making unsupported claims.
- What specific bias metrics were used in testing? (Look for named approaches: demographic parity, equalized odds, counterfactual fairness, predictive rate parity.)
- Which protected attributes were included in testing? Race, gender, age, disability status, geography?
- Were the test datasets representative of the populations our organization actually serves?
- Have any third-party auditors independently assessed the model for fairness?
- How do you monitor for model drift and bias post-deployment? What triggers a reassessment?
- What is your remediation pathway when bias is detected after deployment?
The model drift question is often the most revealing. A vendor may have performed thorough pre-deployment bias testing, but AI models that are continuously retrained on new data can develop new biases over time. An organization that has a rigorous deployment-phase testing process, with clear triggers for reassessment and a transparent remediation pathway, demonstrates that they understand fairness as an ongoing operational responsibility rather than a one-time compliance checkbox.
For nonprofits serving specific underserved communities, the question about test dataset representativeness is critical. An AI model tested primarily on demographic profiles that don't match your beneficiary population may perform poorly or unfairly for your specific users, even if aggregate fairness metrics look acceptable. Ask the vendor how they would validate performance for your population specifically, and whether they offer any customization or retraining for under-represented groups.
Be especially careful with AI tools that will be used in case management, service eligibility, or hiring contexts. These are exactly the scenarios where AI bias in decisions affecting people carries the highest stakes, and where even well-intentioned systems can produce discriminatory outcomes if bias testing wasn't done rigorously.
Security Verification: What Certifications Actually Mean
Security certifications are the most verifiable form of vendor assurance in AI procurement. Unlike bias testing or responsible AI policies, security certifications from recognized standards bodies require independent audit and produce documented evidence that you can request. They're not a perfect guarantee, but they're a meaningful signal that the vendor takes security infrastructure seriously.
Certifications to Ask About
- SOC 2 Type II (covers security, availability, and confidentiality)
- ISO 27001 (information security management systems)
- FedRAMP (for nonprofits with federal grant compliance requirements)
- ISO 42001 (AI-specific management system standard, emerging)
Security Questions to Ask
- How do you protect against prompt injection attacks?
- What is your breach notification timeline and process?
- How is our data logically isolated from other customers' data?
- What encryption standards apply to data at rest and in transit?
The prompt injection question is specific to AI systems and worth understanding. Prompt injection attacks occur when a malicious actor embeds instructions into content that the AI processes, causing it to behave in unintended ways. For nonprofits using AI to analyze external documents, parse survey responses, or process incoming communications, this represents a real attack vector. Asking a vendor how they test for and mitigate prompt injection tells you whether they understand the security landscape specific to AI, not just general software security.
Don't accept references to security certifications in marketing materials without verification. Request the actual SOC 2 report summary, or ask when their most recent audit was completed and whether the report is available to customers. A vendor with genuine certification will be accustomed to this request. One who becomes evasive should raise concern.
Contract Clauses You Should Require
The contract is where vendor commitments become legally enforceable. Yet most AI vendor contracts are written to protect the vendor, not the customer, and many nonprofits sign them without any negotiation. Legal analysis of AI vendor contracts in 2025 found that broad data usage rights were claimed in a large majority of agreements, liability caps were commonly limited to monthly subscription fees, and IP indemnification was absent from most contracts by default.
You don't need to be a lawyer to negotiate better terms. You need to know which clauses matter most and what language to request. Here are the seven clauses that should be non-negotiable for any Tier 2 or Tier 3 AI tool.
1. Data Use Restriction Clause
Explicitly prohibits the vendor from using your data to train models for other customers, for competitive intelligence, or for any purpose outside direct service delivery to your organization. This must be stated explicitly in the contract, not implied from a privacy policy that can be changed unilaterally.
2. Data Portability and Exit Clause
Guarantees you can export all your data in a usable, documented format upon contract termination, and that the vendor will delete all copies within a defined and legally binding timeframe. Without this clause, you may find yourself locked into a vendor relationship you can't exit cleanly, or discover that your data persists in systems beyond your control.
3. Sub-Processor Disclosure Clause
Requires the vendor to provide a full list of current sub-processors and give advance written notice (typically 30 days) before adding new ones. You should have the contractual right to object or terminate for cause if a new sub-processor doesn't meet your standards. This clause is standard under GDPR and increasingly expected even for U.S.-only operations.
4. Audit Rights Clause
Gives you or a designated third party the right to audit model behavior, data handling, and compliance on a periodic or triggered basis. Model drift, a retraining event, or a compliance concern should automatically trigger audit rights. This clause is often negotiated away first by vendors; holding firm signals that you take governance seriously and may encourage the vendor to as well.
5. Indemnification Clause
Requires the vendor to indemnify your organization for IP infringement claims, data breaches caused by vendor negligence, and regulatory penalties arising from vendor non-compliance. IP indemnification is absent from most AI vendor contracts by default and must be actively negotiated. Without it, if an AI vendor's training data use triggers a copyright lawsuit, your organization could be implicated along with the vendor.
6. Incident Notification Clause
Defines what constitutes a reportable incident: security breach, significant model failure, bias discovery, regulatory action against the vendor. Specifies notification timelines (typically 72 hours for breach notification, matching GDPR requirements). Also covers what remediation steps the vendor is obligated to take and in what timeframe.
7. Model Change Notification Clause
Requires advance notice of model updates, version changes, or retraining cycles that may affect performance, bias characteristics, or compliance posture. You vetted a specific model; you're entitled to know before that model changes in ways that could affect your operations or your obligations to the people you serve.
Red Flags That Should Give You Pause
Not every red flag means a vendor is untrustworthy, but patterns of evasion, vagueness, or resistance to reasonable questions are meaningful signals. An organization that is genuinely committed to responsible AI will welcome your scrutiny. They've likely answered these questions many times before and have developed clear, credible responses.
Transparency Red Flags
- Cannot or will not explain how the model works or was trained
- No published privacy policy, Data Processing Agreement, or responsible AI policy
- Cannot identify their sub-processors or refuses to disclose them
- No model card, system card, or technical documentation of any kind
- Vague or unsubstantiated "AI-powered" claims with no specifics (often called "AI washing")
Contract Red Flags
- Broad data usage rights claimed without limitation
- Liability caps limited to the monthly subscription fee
- No indemnification for IP infringement or data breaches
- No audit rights or data portability guarantees
- Complete unwillingness to negotiate any terms
Evasive behavior during the sales process is a particularly telling signal. If you ask a vendor detailed questions about their bias testing methodology and receive only a generic response about their commitment to responsible AI, that's not a documentation gap; it's a credibility gap. A vendor who has done the work can describe it specifically. Sales representatives who escalate your technical questions to engineering or legal staff are a positive sign, showing that the organization has people who actually understand these issues.
Starting with a small pilot before a full commitment is always wise for higher-risk tools. It allows you to observe how the vendor behaves as a partner before you've made a significant organizational investment, and gives you evidence of how the system performs in your actual operating environment before making decisions that affect the people you serve.
For a deeper look at how to structure your AI governance process and keep your board informed, see our guide to AI governance dashboards. For organizations working on updating their broader AI policy frameworks, the guidance on updating your AI policy covers complementary ground.
Building Vendor Accountability Into Your Culture
Ethical AI procurement is not a one-time evaluation; it's an ongoing relationship. The vendor you approved after thorough due diligence in 2025 may look quite different in 2027 after several rounds of model updates, a change in leadership, a merger, or a strategic pivot. Building vendor accountability into your organizational culture means treating AI vendor relationships with the same ongoing monitoring you'd apply to any other significant third-party dependency.
Ongoing Vendor Accountability Practices
- Schedule annual vendor reviews for all Tier 2 and Tier 3 AI tools, re-asking the key due diligence questions with an eye toward what has changed
- Establish an internal process for staff to report unexpected or concerning AI outputs, so patterns can be identified and escalated to vendors
- Maintain a simple AI vendor register that tracks which tools are in use, at what tier, what data they access, and when the last review occurred
- Designate a staff member responsible for monitoring vendor communications about model updates, policy changes, and security notices
- Include AI vendor accountability in your organization's existing risk management and compliance frameworks, not as a separate silo
The vendor register concept is particularly valuable for organizations that have adopted multiple AI tools across different departments without centralized oversight. When every team is independently adopting AI tools, the organization's total exposure is the aggregate of all those decisions, even if no single decision looks risky in isolation. A simple register, even a spreadsheet, gives leadership visibility into the full picture.
This kind of ongoing governance connects directly to the broader AI ethics work that responsible nonprofits are building into their operations. Procurement due diligence is the entry point; ongoing monitoring is how those standards are maintained over time as both the technology and the vendor landscape continue to evolve rapidly.
Procurement as a Mission-Aligned Practice
Ethical AI procurement is ultimately an expression of your organization's values. When you protect the data of the people you serve, you are acting in alignment with the trust they've placed in you. When you demand transparency from vendors, you are modeling the accountability you expect in your own operations. When you insist on fair bias testing, you are honoring your commitment to equitable service delivery.
None of this requires a large IT department or legal team. It requires a structured approach, a set of specific questions, and the organizational will to ask them. Vendors who do good work will welcome the scrutiny. The ones who don't are giving you valuable information before you've made a commitment.
The nonprofit sector has spent decades building trust with communities, donors, and funders. AI adoption that is thoughtful, transparent, and accountable is how you protect that trust in an era of rapid technological change. The questions in this guide are the practical tools for doing exactly that.
Ready to Build Responsible AI Practices?
Our team helps nonprofits develop AI governance frameworks, evaluate vendors, and build the internal capacity to adopt AI responsibly.