Survey Design Before Survey Analysis: Using AI to Write Better Questions

Start With the Decision the Survey Must Inform

The most common survey mistake happens before a single question is written. A team decides it would be "good to hear from participants" and starts brainstorming questions. The result is a long instrument full of things that are interesting to know but connected to no decision anyone will make. Respondents sense this. Long, unfocused surveys produce lower completion and weaker answers, and staff end up with a pile of data nobody has a plan to use.

A better starting point is to write down, in one or two sentences, the decision the survey will inform. Should we move our evening tutoring sessions to Saturday mornings? Which of three newsletter formats should we keep? What are the top unmet needs in our service area that our next strategic plan should address? Why are first-year volunteers not returning for a second year? Each of those decisions implies specific information you need, and that information implies specific questions. Anything that does not trace back to a decision is a candidate for cutting.

It helps to go one step further and sketch the table or chart you expect to produce from each question before you write it. If you plan to compare satisfaction between participants at two sites, you need a site question and a satisfaction question that uses the same scale for everyone. If you want to know whether newer donors feel differently from long-time donors, you need to either ask about tenure or link responses to your donor database. This "analysis plan first" habit catches missing questions early and exposes questions that would produce numbers you could never interpret. It also connects survey work to the larger evaluation picture covered in our guide to AI for nonprofit research and evaluation.

Finally, ask whether a survey is the right tool at all. Some questions are better answered by data you already have, such as attendance records or giving history. Others need the depth of interviews or focus groups. Surveys are best for measuring how common something is across a group, comparing groups, and tracking change over time. They are weaker at explaining why people feel the way they do, and they are a poor fit when respondents cannot read the instrument comfortably or have no reason to trust you with honest answers.

Response Scales That Measure What You Mean

The answer options are part of the question. A rating question with options "excellent, very good, good, fair" has three positive choices and one mildly negative one, so it will produce flattering results no matter what people think. Balanced scales give equal weight to both directions around a neutral point: "very satisfied, somewhat satisfied, neither satisfied nor dissatisfied, somewhat dissatisfied, very dissatisfied." Label every point with words rather than labeling only the ends, because fully labeled scales are easier to interpret consistently, especially on phones and for respondents reading in a second language.

How many points? For most nonprofit purposes, four or five well-labeled options are plenty. Pew notes that it generally keeps response sets to about four or five options. Longer numeric scales, such as zero to ten, have their uses, including the familiar "how likely are you to recommend" item, but they ask respondents to make fine distinctions that many people do not have and that your decisions rarely depend on. Whichever you choose, keep the same scale direction and format for similar questions throughout the survey, so respondents are not tripped up by a scale that suddenly reverses.

The midpoint and the "don't know" option deserve deliberate choices. A neutral midpoint lets people with genuinely mixed or indifferent views say so. Leaving it out forces them to lean one way, which can be useful when you need a decision but can also manufacture opinions. A "don't know" or "no opinion" option is different from a neutral midpoint. It is for people who lack the information to answer, such as a donor asked to rate a program they have never seen. Without it, those people guess, and their guesses look like real data. A "does not apply to me" option is different again, and it matters for questions that only some respondents can answer, such as rating childcare at an event they attended without children. Mixing these up, or putting "N/A" at the middle of a scale, muddies the results.

Order within the answer list matters too. Pew describes primacy effects in self-administered surveys, where people are more likely to pick options near the top, and recency effects in phone surveys, where they pick options heard last. For lists without a natural order, such as a list of possible new services, most online survey tools can randomize the order for each respondent. Ordered scales should stay in order, though you can reverse the direction for half your sample if you want to spread out order effects. Also avoid "select all that apply" lists when you can. Pew has found forced-choice formats, where people answer yes or no to each item, tend to yield more accurate answers, because respondents skimming a checkbox list tend to stop early.

Plain Language, Translation, and Accessibility

A survey is only as inclusive as its hardest question. If items are written at the reading level of a grant proposal, the people who answer will skew toward those who read comfortably, and the people you most need to hear from will quietly drop out or guess. Plain language for surveys means short sentences, common words, one idea per question, active voice, and concrete examples where a concept might be unfamiliar. Readability formulas give a rough check, but they measure sentence and word length, not meaning, so a short question can still be confusing. Our guide to plain language forms with AI covers rewriting techniques that apply directly to survey items and instructions.

Translation is where many nonprofit surveys go wrong, because a literal translation can be accurate word by word and still ask a different question. Concepts like "household," "neighborhood," "case manager," or "satisfied" do not always map neatly across languages and cultures, and scale labels such as "somewhat" can carry different strength in different languages. The survey research field's recommended approach is team translation, often described by the acronym TRAPD: Translation, Review, Adjudication, Pretesting, and Documentation. The Cross-Cultural Survey Guidelines from the University of Michigan endorse this team approach and note that back translation alone does not meet their recommendations. In practice, that means at least one fluent translator produces a version, another fluent reviewer critiques it, someone who understands the survey's purpose resolves disagreements, and the translated version is tested with native speakers from your actual community.

Cultural adaptation goes beyond language. Examples in a question may not fit a respondent's life. Response formats that are familiar in one culture, such as rating your own performance, can feel uncomfortable in another. Some communities are reluctant to give negative feedback directly to a service provider, which pushes ratings upward regardless of wording. People with recent experiences of government surveillance may distrust any question about immigration status, income, or household members. The fix is to involve community members in design, not just in translation, and to treat every translated version as a new instrument that needs its own pilot. Our article on reviewing AI translation quality describes how to check machine-produced drafts.

Accessibility should be built in from the start. For online surveys, that means choosing a platform that works with screen readers and keyboard navigation, avoiding questions that depend on color or images alone, using sufficient contrast, providing text alternatives for any images, and avoiding complex matrix questions that are hard to navigate with assistive technology. The Web Content Accessibility Guidelines (WCAG) are the common reference standard. Offer other modes when you can: paper, phone, or an in-person option with a staff member or volunteer who reads questions aloud, which also helps people with limited literacy. Ask whether your survey is usable by someone with low vision, a motor impairment, a cognitive disability, or an older phone with limited data. Our piece on building accessible AI programs covers the broader principles.

Where AI Genuinely Helps in Questionnaire Design

The survey research profession is actively working out how AI fits into its methods. In May 2026, the American Association for Public Opinion Research released a task force report on responsible AI integration in survey research, which proposes evaluating AI uses across the survey lifecycle according to their risk and their effect on validity, reliability, and performance. The broad professional consensus is that AI is most valuable as a drafting and review assistant, with humans leading the design. For a nonprofit without a survey methodologist on staff, that assistant role is significant. It can give you a second set of eyes on every question, available at any hour.

Drafting items from your survey brief

Give an AI assistant your one-page brief and ask it to propose questions for each information need, with balanced, fully labeled response options. You will get a reasonable first draft in minutes, which is often easier to critique and improve than a blank page. Ask for two or three alternative wordings for key questions so you can compare. Be explicit about your audience: "Write for adults who may read at a sixth-grade level and may be reading in their second language" produces very different drafts from a generic request.

Critiquing a draft for bias and ambiguity

This is where AI earns its place. Paste in your draft and ask the model to review each item for leading language, loaded terms, double-barreled structure, unbalanced scales, missing "don't know" or "does not apply" options, vague time frames, jargon, and assumptions about the respondent. Ask it to explain each concern and suggest a fix. Models are consistent about checking every item, something human reviewers skimming page four of a draft are not. You will still need to judge each suggestion, because models sometimes flag problems that are not really there and occasionally propose "fixes" that change what the question measures.

Simulating respondent confusion

A useful technique is to ask the model to role-play different kinds of respondents and describe how they might misunderstand each question. "Read this question as a parent who works two jobs and has never attended our board meetings. What might you think it is asking?" can surface ambiguity you would otherwise discover only after fielding. Treat these outputs as hypotheses to check in your pilot, not findings. The model is guessing at how people might think, based on patterns in text, and its guesses about specific communities can be wrong or stereotyped. Our article on synthetic personas for program design discusses the same tension in another context.

Reading level checks and plain language rewrites

AI can rewrite items and instructions at a target reading level, suggest simpler synonyms, and break long questions into shorter ones. Ask it to preserve meaning and to flag any rewrite where simplifying might change what is being measured. Pair this with a standard readability tool for a rough numerical check, and remember that the real test is whether pilot respondents understand the question.

Translation drafts for human review

Machine translation can produce a solid first draft that speeds up the translation step of a team approach. It does not replace the review, adjudication, and pretesting steps, and it is weakest exactly where surveys are most fragile: scale labels, idioms, culturally specific examples, and the emotional weight of words. Always have fluent speakers from your community review and test translated versions.

Pilot Testing and Cognitive Interviewing With Real People

Professional survey organizations test new questions before fielding them, and Pew describes pretesting as essential for questions being asked for the first time. The federal government maintains an entire center for this work: the National Center for Health Statistics' Collaborating Center for Questionnaire Design and Evaluation Research, whose primary evaluation method is cognitive interviewing. Its public Q-Bank database of question evaluation reports is a useful resource when you want to see how a commonly used question has performed in testing. You do not need federal resources to apply the same idea. A handful of conversations with real respondents will catch most serious problems.

Cognitive interviewing means sitting with someone from your target audience while they work through the survey, and asking them to explain how they understood and answered each question. Two techniques are common. In think-aloud, the person narrates their thinking as they go. In verbal probing, you ask follow-up questions such as "What does the word 'services' mean to you here?", "How did you choose between 'somewhat' and 'very'?", or "Was there an answer you wanted to give that was not listed?" The goal is not to collect their answers but to learn whether the question is working. Five to ten interviews with people who resemble your respondents, including people who might struggle most, will usually reveal the biggest problems. If you have translated versions, interview speakers of each language.

After cognitive interviews and revisions, a small field pilot tests the logistics. Send the survey to a small group through the actual channel you will use, whether that is a text message link, a tablet at the front desk, or a paper form mailed with a return envelope. Check how long it really takes, where people stop, which questions get skipped, whether answers cluster suspiciously at one end of a scale, and whether the data exports in a usable format. This is also when you discover that the survey platform's mobile layout breaks your carefully designed grid, or that the link expires after 24 hours.

AI can support this stage without replacing it. It can help you write a cognitive interview guide with probes for each question, summarize notes from interviews you conducted (with identifying details removed), and help you track which issues came up across interviews. If your pilot or survey moves toward research intended to contribute generalizable knowledge, such as a study with a university partner, check our guide on IRB review and research ethics for nonprofits before you start.

Applying the Principles to Four Common Nonprofit Surveys

The same principles apply everywhere, but each type of nonprofit survey has its own characteristic traps. Knowing them in advance lets you point your AI critique prompts and your pilot at the problems most likely to matter.

Program participant surveys face the strongest power imbalance, because the people answering depend on the organization asking. Social desirability and fear of consequences push ratings up, so anonymity, clear assurances, and administration by someone other than direct service staff matter a great deal. Literacy, language, and access are often the biggest barriers, which makes plain language, assisted modes, and cognitive interviews especially valuable. Outcome questions should focus on specific, observable changes rather than general impressions. For the broader practice of gathering and using participant input, see our guide to collecting beneficiary feedback.

Donor surveys are prone to a different bias: the temptation to turn the survey into a cultivation or solicitation tool. Questions that double as appeals ("How important is it to you that every child in our county has a warm coat?") produce flattering data and can annoy donors who notice the tactic. Keep the survey clearly separate from asks, include honest options for reduced engagement ("less often," "I'd prefer not to be contacted this way"), and decide in advance whether responses will be linked to giving records, then tell donors which it is.

Volunteer surveys often suffer from timing. Surveying only current, active volunteers misses the people who left, who are usually the ones with the most useful feedback. Consider a short exit survey for those who stop serving, and keep questions specific to the volunteer experience: onboarding, scheduling, communication, recognition, and the meaningfulness of the work. Our piece on the volunteer retention challenge covers what tends to drive those departures.

Community needs assessments carry the highest stakes and the greatest risk of baked-in assumptions. If the survey lists only the services your organization already provides, it will confirm that those are the community's needs. Start with open-ended questions in early rounds or community conversations, involve residents in writing the instrument, and pay close attention to sampling so that the loudest or most connected voices do not stand in for everyone.