AI Auditing for Bias: How to Test Your Nonprofit's AI Tools for Discrimination

Historical bias is arguably the most pervasive and difficult to address because it is embedded in the training data itself. When an AI system learns from data that reflects past discrimination, it treats that discrimination as a pattern to replicate. Consider a nonprofit using AI to identify neighborhoods for a new after-school program. If the model uses historical program enrollment data, it may favor neighborhoods where the organization already has a presence, perpetuating geographic inequities in who gets served. The data is technically accurate (those neighborhoods did have higher enrollment) but reflects access patterns shaped by decades of unequal investment, not actual need.

Detecting historical bias requires comparing model outputs against ground-truth measures of need or eligibility that are independent of the biased historical record. For the after-school program example, this might mean comparing the AI's neighborhood rankings against census data on child population density, poverty rates, and existing program availability rather than past enrollment figures.

Selection bias occurs when the data used to train an AI model does not represent the full population the model will serve. This is extremely common in nonprofit contexts because data collection often depends on who walks through your door, who fills out a survey, or who has reliable internet access. A mental health nonprofit that trains a needs-assessment chatbot on data from English-speaking clients who accessed services online will produce a tool that works poorly for clients who prefer other languages, have lower digital literacy, or historically accessed services through walk-in visits.

The fix for selection bias starts before the model is built: by auditing the training data for representativeness. During a bias audit, you should ask vendors what data their models were trained on, how diverse that data is, and what populations might be underrepresented. If you are building models internally, compare the demographics of your training data against the demographics of the community you serve.

In the context of AI, confirmation bias refers to feedback loops where an algorithm's decisions influence the data it learns from in ways that reinforce its initial assumptions. A predictive policing tool deployed in a social services context illustrates this clearly: if a model flags certain clients as "high risk" and those clients then receive more scrutiny, they are more likely to have issues detected, which generates data that confirms the model's original prediction. The model becomes increasingly confident in its biased assessments over time, even though the underlying pattern is an artifact of differential scrutiny rather than differential behavior.

Nonprofits that use AI for case management, risk assessment, or fraud detection should be especially vigilant about confirmation bias. The key indicator is whether model accuracy varies across demographic groups: if the model appears more "accurate" for one group, it may simply be reflecting that group receiving more intensive monitoring.

Proxy discrimination is one of the most technically subtle forms of AI bias, and it is the reason simply removing protected characteristics like race or gender from a model does not eliminate discrimination. A proxy variable is a seemingly neutral data point that is highly correlated with a protected class. Zip code is the classic example: because residential segregation remains prevalent in the United States, using zip code as a feature in an AI model can produce outcomes that closely mirror what you would get if you used race directly. Other common proxies include first name, language preference, school attended, commute distance, and internet service provider.

Detecting proxy discrimination requires statistical analysis that goes beyond checking whether protected characteristics are explicitly included in the model. Auditors must examine the correlation between model features and protected classes, and test whether removing suspected proxy variables changes model outcomes for different demographic groups. This is one of the areas where specialized auditing tools become especially valuable.

Originally developed for employment discrimination cases by the Equal Employment Opportunity Commission, the four-fifths rule provides a simple, powerful threshold for detecting disparate impact. The rule states that if the selection rate for a protected group is less than 80% (four-fifths) of the selection rate for the group with the highest rate, the process has a disparate impact that warrants further investigation. For example, if an AI resume screener advances 60% of male applicants to the interview stage but only 40% of female applicants, the ratio is 40/60 = 0.67, which falls below the 0.80 threshold, indicating potential gender bias.

This method works well for any AI system that produces binary outcomes (approved/denied, selected/not selected, high priority/low priority). To apply it in a nonprofit context, define the "positive outcome" for the system you are testing (such as "approved for services" or "advanced to interview"), calculate the rate of that outcome for each demographic group, and compare each group's rate against the highest rate. Any ratio below 0.80 flags a potential problem that requires deeper investigation.

The four-fifths rule is a starting point, not a definitive verdict. A ratio below 0.80 does not prove illegal discrimination, and a ratio above 0.80 does not guarantee fairness. But it provides a clear, quantitative threshold that makes bias conversations concrete rather than abstract. It is especially valuable for nonprofits because it requires no specialized software; you can calculate it in a spreadsheet.

Counterfactual analysis is one of the most intuitive bias testing methods. You create test cases that are identical in every way except for the protected characteristic you want to test, then observe whether the AI system produces different outputs. For a grant application screening tool, you might submit two identical applications where the only difference is the name of the applicant (using names associated with different racial or ethnic groups) and check whether the scores differ. For a chatbot, you might submit identical queries in English and Spanish and compare the quality and completeness of responses.

This method is particularly powerful for testing text-based AI systems like chatbots, email generators, and document review tools where you can directly control the inputs. Create a test suite of at least 50 paired scenarios covering the protected characteristics most relevant to your context (race, gender, age, disability status, language). Document any systematic differences in output quality, tone, accuracy, or recommendations.

One limitation of counterfactual analysis is that it tests for explicit sensitivity to protected characteristics but may miss proxy discrimination. An AI system might produce identical outputs for "Maria Garcia" and "John Smith" but still discriminate based on zip code, which is correlated with ethnicity. That is why combining counterfactual analysis with proxy variable detection (described below) produces a more complete picture.

A/B testing for bias compares AI-driven decisions against a human baseline to identify where the algorithm diverges in ways that correlate with demographic characteristics. In this approach, you take a sample of decisions (such as 200 client intake assessments) and have both the AI system and trained human reviewers evaluate them independently. You then compare the outcomes, looking for cases where the AI and humans disagree, and checking whether those disagreements cluster around particular demographic groups.

This method is valuable because it anchors the bias discussion in a comparison that staff can understand intuitively. Rather than debating whether a statistical threshold is meaningful, you can point to specific cases where the AI treated a client differently than a trained human would have. The human baseline is not assumed to be perfect (humans have biases too), but systematic patterns of AI-human disagreement along demographic lines are a strong signal that warrants investigation. This approach requires more effort than the four-fifths rule but provides richer insights.

Proxy variable detection examines the correlation between model features and protected characteristics to identify inputs that effectively act as stand-ins for race, gender, or other protected classes. The process involves calculating the statistical correlation (such as Pearson correlation or mutual information) between each feature the AI uses and each protected characteristic. Features with correlations above a defined threshold (commonly 0.5 or higher) are flagged as potential proxies.

Common proxy variables in nonprofit contexts include zip code (correlates with race and income), educational institution (correlates with race and socioeconomic status), language preference (correlates with national origin and ethnicity), device type or internet speed (correlates with income), and time of day for service requests (correlates with employment type and socioeconomic status). Simply removing these variables is not always the answer, because doing so may reduce model accuracy for everyone. The goal is to understand these correlations and make deliberate, documented decisions about which features to include and why.

For vendor-provided tools where you do not have access to model internals, proxy detection can be performed on the input-output level. Analyze the data you send to the system and the decisions it returns, checking whether output patterns correlate with proxy variables in your input data. If clients from certain zip codes consistently receive lower priority scores, that is a signal worth investigating regardless of how the model works internally.

Enacted in 2021 and enforced since July 2023, NYC Local Law 144 requires any employer or employment agency using automated employment decision tools (AEDTs) in New York City to conduct an independent bias audit annually, publish the audit results on their website, and notify candidates that an AEDT is being used. The law defines an AEDT broadly as any tool that uses machine learning, statistical modeling, data analytics, or AI to substantially assist or replace discretionary decisions related to employment.

For nonprofits based in or hiring in New York City, compliance is mandatory. Even nonprofits outside NYC should pay attention, because the law has become a de facto national standard. Many vendors now offer Local Law 144 compliance as a baseline feature, and auditing firms have adopted its requirements as a starting framework. If your nonprofit uses any AI-powered hiring tool, conducting audits aligned with Local Law 144 standards positions you well for compliance regardless of your jurisdiction.

Colorado's AI Act, which took effect in February 2026, goes significantly further than NYC's law. It applies to "high-risk AI systems" across multiple domains, not just employment. The law requires deployers (organizations that use AI systems) to implement a risk management policy, conduct impact assessments before deploying high-risk AI, notify consumers when AI is used to make consequential decisions, and provide an opportunity for human review of AI-driven decisions that adversely affect individuals.

Nonprofits operating in Colorado or serving Colorado residents should assess whether their AI tools qualify as "high-risk" under the Act's definitions, which include systems used in education, employment, financial services, healthcare, housing, insurance, and government services. Many nonprofit service-delivery tools fall squarely within these categories. The impact assessment requirements align closely with the five-stage audit framework described in this article, making proactive bias auditing an effective path to compliance.

Beyond New York and Colorado, at least a dozen states have introduced or passed AI-related legislation as of early 2026. Illinois requires disclosure when AI is used in video interviews. Maryland prohibits facial recognition in hiring without consent. California's proposed AI accountability bills continue to advance through the legislature. Texas, Virginia, and Connecticut have all passed or introduced algorithmic accountability measures with varying scopes and requirements.

For nonprofits that operate across multiple states, this patchwork creates complexity. The practical response is to audit against the most stringent applicable standard (currently Colorado's) and document your process thoroughly. Organizations that adopt a proactive, comprehensive approach to bias auditing will find it straightforward to demonstrate compliance as new regulations take effect, rather than scrambling to meet each new requirement individually. This proactive posture also strengthens grant applications, because funders increasingly ask about AI governance practices.

The foundation of any monitoring program is a set of defined fairness metrics with established baselines. During your initial audit, calculate and record the key metrics for each AI system: selection rates by demographic group, four-fifths rule ratios, error rates by group, and any domain-specific measures relevant to your programs. These baselines become the standard against which future measurements are compared. When a metric moves more than a predetermined threshold from its baseline, that triggers a review.

Choose metrics that are meaningful in your context, not just technically convenient. For a workforce development nonprofit, the most important metric might be job placement rates by race and gender for clients who used an AI-powered job matching tool. For a healthcare nonprofit, it might be appointment no-show prediction accuracy across age groups and insurance types. The metrics should reflect the outcomes that matter most to the people you serve.

Different systems need different review frequencies based on their risk level and how quickly their behavior might change. High-risk systems that make consequential decisions about service access or employment should be reviewed quarterly at minimum. These reviews should include both automated metric checks and human review of a sample of recent decisions. Medium-risk systems can be reviewed semi-annually, and low-risk systems annually. Any system should get an immediate review after a major vendor update, a significant change in the population served, or a staff or client complaint about potentially biased behavior.

• Quarterly: High-risk systems (service access, hiring, financial decisions)
• Semi-annually: Medium-risk systems (prioritization tools, recommendation engines)
• Annually: Low-risk systems (content tools, internal scheduling, newsletter optimization)
• Immediately: After vendor updates, population changes, or bias complaints

Automated metrics are essential but insufficient. Some forms of bias are best detected by the people experiencing them. Create clear, accessible channels for both staff and clients to report suspected bias in AI-driven decisions. Staff on the front lines of service delivery are often the first to notice when an AI system seems to be treating certain clients differently. Clients who feel they have been unfairly assessed or deprioritized should have a straightforward way to raise concerns and receive a timely response.

These feedback channels should be simple to use (a form, an email address, a section on the client portal), clearly communicated (mentioned during intake, posted in service areas, included in digital communications), and connected to a defined response process. Every report should be logged, investigated within a defined timeframe, and resolved with documentation. Even reports that do not reveal bias provide valuable data about how clients perceive and experience your AI systems. Over time, patterns in feedback reports can identify emerging bias issues that quantitative metrics have not yet captured.

Every element of your monitoring program should be documented: the metrics you track, the baselines you established, the review cadences you follow, the findings from each review, the remediation actions taken, and the outcomes of those actions. This documentation serves multiple purposes. It creates institutional memory that survives staff turnover. It provides evidence of due diligence for regulators, funders, and the public. And it enables your organization to identify long-term trends in AI fairness that would be invisible without historical records.

Assign clear accountability for monitoring tasks. Someone specific should own each AI system's bias monitoring, with backup assignments for coverage during absences. Include monitoring responsibilities in job descriptions and performance evaluations to ensure they do not get deprioritized when other work demands attention. Report monitoring results to your board or AI equity committee at least annually, and consider publishing summary findings to demonstrate your commitment to responsible AI and build trust with the communities you serve.