A/B Testing Donation Landing Pages: Which AI-Generated Variants Are Worth Running

How an A/B Test Works on a Donation Page

An A/B test is a controlled experiment. You keep your current page as the control (version A) and create one alternative (version B) that changes something specific. Testing software randomly assigns each visitor to one version and remembers the assignment, so a returning visitor sees the same page each time. Because the assignment is random, the two groups of visitors are, on average, alike in every way except the page they saw. Any reliable difference in how they behave can therefore be attributed to the change you made, rather than to the time of day, the email that sent them, or the kind of donor they happen to be.

On a donation page, the behavior you measure is usually some combination of whether a visitor gave (the conversion rate), how much they gave (the average gift), and whether the gift was one-time or recurring. These measures can move in opposite directions. A version that removes a high default amount might convert more visitors while lowering the average gift, and the net effect on revenue could be positive, negative, or zero. Deciding before the test which measure counts as the "win" is one of the most important steps, and we return to it in detail below.

The final ingredient is statistics. Even two identical pages will show different conversion rates in any given week, simply because of chance. Statistical testing asks whether the gap you observed is larger than chance alone would plausibly produce. Most tools report this as a confidence level, a p-value, or a "probability to beat control." The details vary by tool, but the underlying point is the same: a small gap measured on a small number of donors is usually indistinguishable from noise, and acting on it is not much better than flipping a coin.

A related method, the multivariate test, changes several elements at once and tries to estimate the effect of each combination. It sounds efficient, but it divides traffic among many more versions and needs far more visitors than a simple A/B test. For nearly every nonprofit outside the largest national organizations, a straightforward two-version test is the right tool.

Tests Worth Running: Changes Big Enough to Measure

Given the traffic constraint, the best tests share a trait: they change something that directly shapes the donor's decision or the effort it takes to give. Fundraising research organizations such as NextAfter have published years of donation page experiments, and the recurring theme is that changes to the offer, the friction, and the value proposition tend to matter more than cosmetic details. The categories below are the strongest candidates for a nonprofit with limited traffic.

Ask arrays and default amounts. The suggested gift amounts, and which one is pre-selected, influence both how many people give and how much. Testing a higher or lower set of amounts, fewer buttons, or a different default is a direct test of donor behavior with a plausibly large effect on revenue per visitor. Some platforms now personalize ask amounts based on a donor's history, a topic we cover in our article on dynamic ask amounts. A test of personalized versus fixed arrays is often a worthwhile one.

Monthly versus one-time default. Whether the form opens on the monthly or one-time tab, and how the monthly option is framed, can shift the share of new donors who become sustainers. Because recurring donors are worth far more over time, this is one of the highest-value tests available, but it must be judged on long-term revenue rather than first-day conversions. Our guide to converting donors to monthly giving discusses the framing in depth.

Form length and fields. Every required field is a small reason to quit. Removing optional fields, moving a phone number request to the thank-you page, or splitting a long form into steps can produce meaningful changes in completion. These tests are also easy to state as clear hypotheses.

Full page versus embedded form or overlay. Sending visitors to a separate checkout page, embedding the form on the campaign page, or opening it as an overlay are structurally different experiences. Changes of this magnitude often produce effects large enough to see on modest traffic.

Value proposition and social proof. A page that explains concretely what a gift accomplishes, or that shows how many people have given, tests a different reason to give, not just different words. Social proof must be truthful, and the ethics section below covers where the line sits.

Matching gift messaging. Announcing an active match, or prompting donors to check whether their employer matches gifts, can affect both conversion and total dollars raised. See our article on matching gift revenue for how to identify and follow up on employer matches after the gift.

Using AI Well: Fewer, Bolder, Hypothesis-Driven Variants

The best use of AI in donation page testing is the opposite of its most obvious use. Instead of asking for many small variations, ask it to help you design one or two variants that are meaningfully different and grounded in a clear idea about donor behavior. A useful prompt names the hypothesis first and the copy second. For example: "Our current page leads with our organization's history. We believe first-time donors respond more to a specific, concrete outcome of a single gift. Write a complete alternative page that leads with what a $50 gift accomplishes, uses one short paragraph of supporting detail, and keeps the form above the fold on mobile." That produces a variant that tests an idea, not a phrase.

Write the test plan before the variant

AI is excellent at turning a loose idea into a structured plan. Give it your hypothesis, your baseline numbers, and your monthly traffic, and ask it to draft a one-page plan that states the change, the reason you expect it to work, the primary metric, the secondary metrics you will watch, the target sample size, the planned duration, the stopping rule, and what you will do if the result is a win, a loss, or inconclusive. This document protects you later from the temptation to reinterpret the test after the fact, and it gives your executive director or board a clear picture of what you are trying to learn.

Ask AI to argue against your variants

One of the most valuable and least used techniques is asking AI to predict which of your candidate variants is weaker and why, before any traffic is spent. Paste in your control and three possible challengers and ask the model to act as a skeptical fundraising strategist: which variant is least likely to beat the control, which adds friction, which might confuse a first-time donor, and which is too similar to the control to produce a measurable difference. The model's predictions are not evidence, and it will sometimes be wrong. But it is good at spotting variants that are vague, that bury the ask, or that are essentially the control with different adjectives. Cutting those before launch saves weeks of traffic for the variant with a real chance.

Use AI to read results, not to declare winners

After a test ends at its planned sample size, AI can help you summarize the results, explain the confidence interval in plain language for colleagues, check whether the effect held across mobile and desktop or across traffic sources, and draft the internal write-up. Ask it to point out what the data cannot tell you, such as long-term retention for a monthly-giving test that has only run for six weeks. Avoid pasting donor names, emails, or payment details into a general-purpose AI tool; aggregate numbers are all the analysis needs. And keep the final call with a person who understands the plan and the organization's goals.

Pitfalls That Produce False Winners

Peeking and stopping early. The most common mistake is checking results daily and stopping the test as soon as one version looks significantly ahead. Standard significance calculations assume you decided the sample size in advance and looked once at the end. Statistician Evan Miller's widely cited essay "How Not To Run an A/B Test" shows how repeated checking with the intention of stopping at the first significant result dramatically inflates the chance of a false win. The fix is simple: set the sample size in your plan and do not act until you reach it, or use a tool built for sequential testing that adjusts for repeated looks.

Testing during unusual periods. Donor behavior during GivingTuesday, the last week of December, or an emergency appeal is not typical. Visitors arrive with different intentions, from different channels, and in much greater numbers. A test that runs across those periods mixes very different audiences, and a result from the year-end rush may not hold in March. Either avoid starting tests that will span peak periods, or design a test specifically for that period and apply the result only to that period next year. Our guides to GivingTuesday and year-end campaigns describe how those audiences differ.

Novelty effects. Returning visitors, such as regular donors clicking through from your newsletter, may respond to a new design simply because it is new. That initial bump can fade over weeks. Running tests for at least one or two full weekly cycles, and checking whether the effect is stable across the test, helps separate lasting improvement from curiosity.

Measuring conversion but not revenue or retention. As discussed above, a version can win on the metric you measured and lose on the one you care about. Decide your primary metric in advance and track downstream outcomes for tests involving gift amounts or recurring giving.

Too many variants. Each additional version splits your traffic further and raises the odds that one looks good by chance. If AI has given you many promising ideas, rank them and test the strongest one or two, then queue the next idea for the following test.

Uneven traffic mixes. If one email sends visitors only to version B, or a paid campaign points at a URL that bypasses the randomization, the comparison breaks. Confirm that all sources reach the test through the same entry point and that the tool splits each source evenly.

Ethical Lines: Testing Without Manipulating Donors

Testing makes it easy to find what works in the short term, and some things that work are not things a mission-driven organization should do. AI makes this risk sharper, because a model asked to "maximize conversions" will cheerfully suggest countdown timers, invented scarcity, and guilt-heavy copy. The Federal Trade Commission's staff report Bringing Dark Patterns to Light describes design practices that trick or manipulate people, including fake countdown timers and false scarcity claims, hidden costs, and pre-checked boxes that sign people up for things they did not choose. The report is aimed at commerce generally, but donors deserve at least the same respect as shoppers.

In practice, that means urgency must be real. A match deadline that actually exists can be stated plainly; a timer that resets for every visitor cannot. Social proof must be accurate: if you show the number of donors to a campaign, it must be the true number. Recurring giving defaults deserve particular care. Defaulting the form to monthly giving is a legitimate design choice when the monthly option is clearly labeled and easy to switch, but a design that leads donors into recurring charges they did not intend erodes trust, generates refund requests and chargebacks, and can draw regulatory attention. Cover-the-fees checkboxes and add-on asks should likewise be transparent about what the donor is paying.

A simple test helps: would you be comfortable if a donor saw both versions side by side and understood exactly why you tested them? If a variant only works because donors misunderstand it, it is not a winner worth keeping. Write this standard into your test plan template so that every AI-generated variant is reviewed against it before launch.