Turning Case Notes into Outcomes Data

The Data You Already Have and Cannot Currently Use

Before designing anything, it helps to be specific about what is actually sitting in your case notes and why it is valuable. In most human services settings, the structured fields in the case management system capture the transaction: date of contact, service type, duration, referral destination. The narrative captures the substance: why the client came in, what barriers surfaced, what the caseworker attempted, how the client responded, and what changed between visits.

That substance contains at least four categories of information that funders and boards repeatedly ask for and that most organizations cannot produce. The first is outcomes, meaning concrete changes in a client's situation such as securing housing, obtaining employment, resolving a legal matter, or completing a treatment course. The second is barriers, meaning the recurring obstacles that slow or block progress, which is the single most useful input for program design. The third is service pathways, meaning the actual sequences of support that clients move through, which almost never match the linear model in the logic model. The fourth is early warning signals, meaning the patterns that precede disengagement or crisis.

Consider a housing program that reports how many households received rental assistance. The case notes for those households describe which ones stayed housed six months later, which ones were displaced again and why, which landlords repeatedly refused vouchers, and which clients needed benefits advocacy before assistance could stabilize anything. The first number satisfies a grant report. The rest is what would let the organization redesign the program, and it is already written down.

It is worth naming the honest limitation up front. Case notes reflect what staff chose to write, filtered through their training, caseload pressure, and organizational norms. If caseworkers were never asked to note housing status at closure, extraction will not find it, no matter how good the model is. Extraction surfaces what is present in the record; it cannot conjure what nobody documented. That constraint shapes both what you should expect from a first pass and what you should change about documentation practice going forward.

Designing the Extraction Schema Before You Extract Anything

The most common way this work fails is that someone asks a model to "find the outcomes" in a set of case notes. What comes back is a plausible, readable, and entirely unusable summary, because the question had no definition behind it. Extraction only produces trustworthy numbers when you have decided in advance exactly what you are counting and what evidence would count as sufficient.

A workable schema is a short list of fields, each with a precise definition, a closed set of allowed values, and a rule about what constitutes evidence. Instead of "housing outcome," you define a field called housing status at case closure, with permitted values of permanent housing, transitional housing, temporary arrangement with family or friends, unsheltered, institution, or not documented. That last value is essential and frequently omitted. A field that cannot return "not documented" forces the model to guess, and a model that guesses will produce clean-looking data that is quietly wrong.

Each field should also specify what the model must point to. Requiring the model to return a short supporting quote alongside every extracted value transforms the reliability of the whole system. It lets a human spot-check a result in seconds rather than rereading a full case file, it makes disagreements resolvable, and it produces the audit trail you will need if a number ever ends up in a public report. Extraction that returns only a value with no provenance is unverifiable, which means it is not evidence.

Build the schema with the people who write the notes. Caseworkers know that "housed" means something different in your program than the funder's definition, that a particular phrase is shorthand for a specific situation, and that a certain kind of entry usually means a client stopped responding rather than declined services. Encoding that knowledge into the field definitions is the difference between a schema that fits your records and one that fits a generic template. It also builds the buy-in you will need when staff are asked to trust and review the results.

Validation: Proving the Numbers Deserve to Be Believed

If an extracted figure is going into a grant report, it needs the same standard of evidence as any other reported number. That means you must be able to answer a simple question from a program officer: how do you know this is right? "The AI read the notes" is not an answer. A documented agreement rate against independent human coding is.

The practical method is straightforward. Before running the full dataset, take a random sample of cases, have two experienced staff members code them independently using the same schema, and reconcile their disagreements. That reconciled set becomes your reference standard. Then run the model on the same cases and compare field by field. You will learn two things at once: how well the model performs, and how well your schema is written. When two experienced humans disagree substantially on a field, the field definition is ambiguous and the model's confusion is a symptom rather than the disease.

Look at the errors rather than only the headline agreement rate. An overall agreement figure hides the pattern that matters. A model might handle housing status almost perfectly while systematically misreading employment outcomes because your notes use an internal shorthand it does not understand. It might perform well on notes written by long-tenured staff and poorly on notes from a program with different documentation habits. Those patterns are actionable, and they also tell you which fields need a human review step permanently rather than only during the pilot.

Watch particularly for differential accuracy across client groups. If extraction is less accurate for notes involving clients who needed interpretation, or clients in a particular program, or notes written by a specific team, then any aggregate figure will misrepresent those groups. This is the same equity concern that arises throughout applied AI, and it is especially serious here because the resulting numbers will be used to make funding and program decisions about the people the errors affect. Our guide on mapping and mitigating program inequalities explains how to structure that analysis.

Finally, document the validation and publish the caveats alongside the findings. A report that says a figure was derived from AI-assisted extraction of case notes, validated against independent human coding on a sample, with the agreement rate stated and the known limitations named, is more credible than one that presents the number bare. Funders have become considerably more sophisticated about AI-derived evidence, and transparency about method is now a mark of rigor rather than a confession of weakness.

Consent, Governance, and Being Straight with Clients

There is a question underneath all of this that technical controls do not answer. Clients disclosed difficult things to a caseworker in a relationship of trust. Did they agree to a machine reading it? In most cases the consent they signed at intake describes service delivery, quality improvement, and program evaluation in general terms, and a reasonable reading may cover internal analysis. But "legally defensible" and "consistent with what the client understood" are different standards, and organizations that only meet the first one tend to regret it.

The practical response has three parts. Review your intake consent language and update it so it describes, in plain terms, that records may be analyzed, including with automated tools, for the purpose of understanding and improving services. Say plainly what will never happen, such as sharing identified records with vendors for training purposes. And make the notice genuinely readable, because consent obtained through dense text that nobody reads is a formality rather than an agreement. Our article on informed consent in an AI context works through how to write language that clients can actually act on.

Governance should sit above the individual project. A standing decision about who may authorize processing of client narratives, what categories of program are excluded outright, what the retention period is for extraction outputs, and who reviews the practice annually keeps this from becoming a series of one-off judgment calls. It also means the next staff member who wants to try something similar inherits a framework rather than reinventing one under deadline pressure.

Some categories of record should simply be excluded, and deciding that in advance is easier than deciding it in the middle of a project. Notes from domestic violence services, records covered by heightened federal confidentiality protections, immigration case files, and notes involving minors in sensitive circumstances all warrant an explicit exclusion or a much higher bar. The value of better outcomes data does not outweigh a risk to a client's physical safety, and an organization that has written that principle down in advance will find the decision easy when it arises.