Building an Evaluation Set for Your AI Workflows: How to Know It Still Works

What an Evaluation Set Actually Is

An evaluation set is a fixed collection of inputs and the outputs you have already agreed are good, stored somewhere stable, run through your AI workflow on a schedule and after any change, and graded against a written standard. That is the whole concept. Practitioners sometimes call it a gold set or a golden dataset, which makes it sound more formal than it needs to be. For a nonprofit with one or two AI workflows in daily use, it is a spreadsheet with a few dozen rows.

The defining property is that it does not change. This trips people up, because the instinct when a test fails is to fix the test. If your test cases drift along with your prompts, you lose the only fixed reference point you had, and you can no longer answer the question the set exists to answer, which is whether today's output is worse than the output you approved. You can add new cases as new situations arise. You should essentially never edit an existing case just because it stopped passing. Changing a case is a deliberate decision made because the correct answer genuinely changed, for example because your organization renamed a program or updated a required disclosure, and it should be recorded as such with a date and a reason.

It is worth being precise about what an evaluation set is not. It is not a measure of whether AI is good at your task in some absolute sense. It is not a benchmark you can compare against other organizations. It is not an accuracy percentage you can put in a board report without heavy caveats. It is a tripwire. Its only job is to tell you that something changed, and to tell you soon enough that you can investigate before the change reaches a donor's inbox or a funder's desk. Holding that modest goal in mind keeps the exercise proportionate. You are not building a research instrument. You are building a smoke detector.

The second useful property is that an evaluation set makes changes cheap to try. Most organizations are cautious about editing a working prompt because they have no way to tell whether the edit helped, so prompts calcify and accumulate awkward workarounds nobody dares remove. With a set of cases you can rerun in twenty minutes, prompt editing becomes a normal activity rather than a risky one. You change something, you rerun, you see whether the score moved, and you keep or revert. This is the same reason software teams write tests, and it is the benefit people underestimate before they have one.

Finally, the set gives you an institutional record. Staff turn over, the person who built the workflow leaves, and what remains is a prompt nobody fully understands doing something nobody has verified. The evaluation set is the part of the documentation that is executable. It sits naturally alongside the practice of documenting AI workflows, and it answers the question a written description cannot: not what this workflow was supposed to do, but what it did the last time anyone checked.

Writing Expected Outputs and a Rubric Anyone Can Apply

Here is where most first attempts go wrong. Someone writes an expected output that is a single perfect paragraph, then discovers at grading time that the model produced a different paragraph which is equally good, and now the grader has to decide whether a different-but-fine answer counts as a pass. Do that thirty times and the exercise becomes exhausting and inconsistent. The fix is to stop treating the expected output as a target the model must match and start treating it as a description of what a correct answer must contain.

In practice this means writing each case as a short list of requirements rather than a model answer. For a grant summary, the requirements might be: names the funder correctly, states the deadline as March 14 2027, identifies the three stated funding priorities, notes the geographic restriction to the tri-county area, does not claim we are eligible or ineligible, and runs under 300 words. Any output meeting all six passes, regardless of how it is phrased. This is far easier to grade, far more consistent between graders, and it forces you to articulate what you actually need from the workflow, which is a useful exercise in its own right.

Keep a reference answer alongside the requirements anyway, but use it differently. Its purpose is not comparison, it is calibration. When a new person takes over grading, reading a few approved outputs teaches them what "good" looks like in your organization faster than any rubric can. It also gives you something concrete to point at when quality drops, because showing someone the old output next to the new one makes a case that a score change does not.

Write the rubric in plain language aimed at whoever will actually grade, who is probably a program manager or a development associate rather than anyone technical. Every check should be answerable yes or no by someone reading the output once. "Is the tone appropriate" is not answerable. "Does it avoid clinical jargon and address the reader as you" is. "Is the summary accurate" is not answerable. "Does every fact in the summary appear in the source document" is, though it takes longer. The discipline of converting vague quality attributes into observable checks is most of the work of building a usable rubric, and it is the part worth spending real time on because you write it once and use it for years.

Grade at the level of the individual check rather than the whole output. A case with six requirements yields six yes or no answers, which is much more informative than one pass or fail. When quality drifts, it almost never drifts everywhere at once. It shows up as one specific check failing across several cases, and that pattern is the thing that tells you what changed. If the "includes the required disclosure" check fails on four cases this month and passed on all of them last month, you have a specific, diagnosable problem rather than a vague sense that things are worse.

Using a Second Model as Grader, and Where That Breaks

The obvious labor-saving idea is to have a model do the grading. Give a second model the input, the output, and the rubric, and ask it to score each check. This technique is widely used and it genuinely works for parts of the job, particularly for checks that are mechanical but tedious, like confirming that every date in a summary appears in the source document. Used for that kind of work it can cut your human grading time substantially, and for a two-person organization that difference determines whether the monthly run happens at all.

It also fails in specific and well-documented ways that you need to know about before you rely on it. Models used as graders show self-preference bias, systematically rating output from the same model family more favorably, and research analyzing a broad set of mainstream models has found the counterintuitive pattern that stronger models often show larger self-preference bias, not smaller. They show verbosity bias, rating longer answers higher regardless of quality. They show position bias, favoring whichever candidate answer appeared first or last when comparing two. And their agreement with human judgment degrades notably outside controlled conditions, with reported gaps between controlled-test accuracy and production bias-test performance that should make anyone cautious about treating a model score as ground truth.

There is a deeper problem for our purposes specifically. The grader model is itself subject to exactly the drift you are trying to detect. If both your production model and your grader model update, and the grader's standards shift in the same direction as your workflow's output, the score can stay flat while the actual quality falls. You have built an instrument made of the same material as the thing it measures. This is not a reason to avoid model grading entirely, but it is a decisive reason never to let a model-graded score be the only thing standing between you and a silent regression.

The workable arrangement is a division of labor. Let a model handle high-volume mechanical verification and flag the cases it thinks failed. Have a person review every flagged case, plus a fixed random sample of the cases the model passed, which is what catches the grader missing things. Use a grader from a different provider than the one running your workflow, which reduces the self-preference problem, and pin the grader's version so it changes only when you choose. Never let a model grade the judgment checks unreviewed, because tone and appropriateness are precisely where model judgment diverges most from the judgment of someone who knows your community.

Calibrate the grader before you trust it. Take ten cases you have already graded by hand, run the model grader over them, and compare. If it agrees with you on most checks, you have a useful assistant. If it disagrees frequently, your rubric is probably too vague for a model to apply, which usually means it is also too vague for a new human grader, so the disagreement is telling you something useful either way. Recalibrate whenever you change the grader model. Running graders and production workflows from different providers is one practical argument for a deliberate multi-model strategy rather than standardizing everything on a single vendor.

Cadence, Ownership, and the Spreadsheet That Holds It All

Three triggers should cause a run, and the first is the calendar. Monthly is the right default for a workflow in regular use. Quarterly is defensible for something used occasionally and low stakes. Weekly is overkill for almost any nonprofit and guarantees the practice will be abandoned within two months. Put the monthly run on a specific date with a specific person's name on it, because a cadence with no owner is a cadence that happens twice and then stops.

The second trigger is any change you make. A prompt edit, however small, gets a run before and after. Adding a new instruction to fix one problem very often breaks something else, and the evaluation set is how you find out in twenty minutes rather than in six weeks. This applies equally to changing a document template that feeds the workflow, changing the tool the workflow runs in, or changing a setting you did not think was important. Organizations maintaining a shared prompt library should attach the current evaluation score to each stored prompt version, so anyone reusing it can see when it was last verified and against which model.

The third trigger is any change the provider makes. When you see a notice about a new model version, a deprecation, or a platform update, run the set. This is the trigger most likely to catch a real regression and the one most likely to be missed, because provider announcements arrive in a technical changelog nobody at a small nonprofit is reading. Subscribe someone to the release notes of the tools you depend on, and treat any version change as a reason to run, including the version changes that arrive without an announcement. If your platform lets you pin a specific model version, do that, so upgrades happen when you decide rather than when a provider decides, and understand that pinning buys time rather than permanence, since versions get retired eventually. Our guidance on model selection for nonprofits covers how to weigh that stability question when you choose a tool in the first place.

Ownership in a small organization should be explicit and singular. Name one person, give them ninety minutes a month, and make the run a line item in their role description rather than a favor. In a ten-person nonprofit this is usually the operations manager or whoever owns the workflow itself. Grading can be shared among two or three people, which is good practice anyway because it surfaces rubric ambiguity, but scheduling and follow-through need one name. Where an organization has cultivated internal AI capability, this is a natural responsibility for that person, and it is one of the more concrete things an AI champion can own that produces visible value.

The tooling can be a spreadsheet, and for most nonprofits it should be. One tab holds the cases, with columns for a case identifier, the input or a link to it, the requirements list, the reference output, and notes. A second tab holds the results, one row per case per run, with the date, the model version, the prompt version, a yes or no for each check, and space for the grader's comment. A third tab holds the summary: a row per run with the overall score, the deterministic score, the judgment score, and what you concluded. Charting the summary column gives you the single most useful artifact in the whole system, a line that should be flat and that you will notice when it is not. Working directly with AI assistance inside your spreadsheet, as covered in our piece on using AI in spreadsheets, can automate several of the deterministic checks without any additional software.

When the Set Starts Failing: A Triage Order

A run comes back eight points below baseline. The instinct is to start rewriting the prompt, and it is the wrong first move, because you do not yet know what changed. Work through the possibilities in order of how easy they are to rule out, which happens to be roughly the order of how likely they are to be the real cause.

Start with the measurement itself. Did the grader change? Is a new person grading, applying the rubric differently? Did someone edit a case or a requirement? Did a spreadsheet formula break when a column moved? A surprising share of apparent regressions are measurement artifacts, and they are the cheapest thing to check. Compare this run's failures against the same cases last month and read the actual outputs rather than the scores. If the outputs look comparable and the scores moved, the problem is in the measurement.

Next, check what changed on your side. Pull the prompt version and compare it to the one in the last passing run, including the small edits nobody logged. Check whether the input documents changed format, whether a connected system started returning data differently, whether a template was updated. Changes you made are easier to identify and easier to reverse than changes a provider made, so exhaust them before assuming the model is at fault. This is where version discipline pays for itself, and where organizations without it end up guessing.

Then check the provider. Look at release notes and status pages for the period between your last passing run and this one. Look at whether the model identifier you are using resolves to a different version than it did. If you find a version change, you have your explanation and a decision to make: revise the prompt for the new version, pin back to an older version if that is still available, or move the workflow to a different model. Run the set again after whichever you choose, because the fix needs verifying like anything else.

Read the failing outputs before you change anything. Scores tell you that something moved, but only the text tells you what. Look for the pattern: is it the same check failing, the same kind of input, the same part of the output? Is the model now hedging where it used to be direct, adding caveats, changing format, or truncating? These patterns point at a fix directly, and a prompt edit made after reading five failing outputs will be better than three edits made from the score alone.

While you investigate, change how the output is handled. Put the workflow back under full human review, or pause it for the highest-stakes uses, and say so to the staff who depend on it. Then consider the outputs already produced since the last passing run, because if quality dropped after a model change three weeks ago, three weeks of work may need spot-checking. That is the uncomfortable part of catching a regression, and it is also why the monthly cadence matters: the difference between a month of suspect output and a year of it is the whole argument for doing this at all. Workflows that involve outside language or sensitive judgment, such as the ones described in our article on translation quality review, warrant the most conservative handling during an investigation, because the people most affected by a quality drop are the least likely to be in a position to complain about it.