Should AI Index Your Oral History Collection? Transcription and Access Decisions

Why Oral History Audio Is Hard for Speech Recognition

Most commercial speech recognition is tuned on audio that looks nothing like an oral history archive. It performs best on clear, modern recordings of a single speaker using a widely spoken variety of English, with a good microphone and little background noise. Oral history collections are almost the opposite. Narrators are frequently elderly, speak slowly with long pauses, and use regional vocabulary, place names, family nicknames, and references to businesses and people that disappeared decades ago. Interviews often happen in kitchens and living rooms with a clock ticking, a television in the next room, or a dog. Two people talk over each other, and the interviewer's voice is often closer to the microphone than the narrator's.

Accent and dialect matter a great deal. A widely cited 2020 study published in the Proceedings of the National Academy of Sciences, "Racial disparities in automated speech recognition," tested five commercial systems on sociolinguistic interviews and found an average word error rate of 0.35 for Black speakers compared with 0.19 for white speakers. The systems have improved since, but the underlying lesson holds for archives: error rates are not evenly distributed across narrators. A collection documenting a Black neighborhood, an Appalachian community, a Gullah Geechee family, or a rural Spanish-speaking parish will likely see worse first drafts than a collection of local business leaders, and that unevenness can quietly make some voices less findable than others.

Speech recognition models can also invent text. Researchers led by Allison Koenecke studied Whisper in a 2024 paper, "Careless Whisper: Speech-to-Text Hallucination Harms," and found that roughly 1% of transcriptions in their sample contained entire phrases or sentences that did not exist in the audio, with hallucinations more likely for speakers with longer pauses, such as people with aphasia. The authors noted that the specific behavior they measured was tied to the Whisper version they tested, and OpenAI has since updated its models. For oral historians, the takeaway is still important. Long silences, trailing sentences, and halting speech are characteristic of older narrators, and those are exactly the conditions where a model may fill a gap with plausible words the narrator never said.

Multilingual and code-switching interviews add another layer. An American Archive of Public Broadcasting assessment of Whisper on Tagalog-language recordings found that when no language was specified, the model more often translated Tagalog speech into English rather than transcribing it, and that it cannot be told to use more than one language in a single pass. That is a serious problem for a narrator who moves between Spanish and English, Tagalog and English, Yiddish and English, or an Indigenous language and English within a single sentence. A silent translation erases exactly the linguistic texture that makes the interview historically valuable. Our guide to reviewing AI translation quality describes how to involve fluent reviewers when more than one language is involved.

Finally, the audio itself may be degraded. Cassettes stretch, reels develop sticky-shed problems, and earlier digitization may have been done at low quality or with heavy noise reduction that smeared consonants. AI audio cleanup can help a speech model, but it can also introduce artifacts. Keep your preservation master untouched, and run any cleanup on a derivative copy used only for transcription.

Keyword Tagging and a Verification Workflow You Can Sustain

Keyword and subject tagging is where AI can either strengthen or quietly distort a collection. A language model asked to tag a segment will happily produce terms, but it will use its own vocabulary, which may be modern, generic, or inconsistent across interviews. One segment gets "racism," the next "discrimination," a third "civil rights movement," and none of them use the controlled vocabulary your catalog already relies on. Worse, a model may label a narrator's experience in words the narrator would reject, or apply outdated or offensive terminology that still circulates in its training data.

The fix is to constrain the model. Give it your thesaurus or a short list of approved subject terms, ask it to choose only from that list, and ask it to flag segments where none of the terms fit rather than inventing new ones. Ask it to extract proper nouns separately, as a list of candidate names and places for a human to confirm, rather than folding them into the synopsis. Many speech recognition tools also accept a list of expected names and terms before transcription, which can noticeably improve how local surnames and street names come out. Building that list from your existing catalog and from a quick listen to the first few minutes is time well spent.

Verification is the part organizations most often underestimate. A sustainable workflow does not require checking every word of every interview to the same standard. It requires being explicit about what level of review each output has had, and recording it. A practical approach uses tiers. Unreviewed machine text stays internal. A spot-checked transcript, where a reviewer listened to sampled passages and corrected names, has a documented accuracy level. A fully reviewed transcript has been corrected against the audio in full. An index is reviewed segment by segment while listening. Whatever is published carries a note telling users how it was produced.

Volunteers can be excellent reviewers, especially community members who knew the narrators or the neighborhood, and who will catch a misheard surname instantly. Give them a short style guide that covers how to mark inaudible passages, how to handle false starts and filler words, how to render dialect without caricature, and what to do when they hear something sensitive. That last point matters: reviewers are often the first people to notice a passage that raises an access question, so they need a simple way to flag it rather than quietly correcting and moving on. If you are building a team to sustain this kind of work, our guide to AI for museums and historical societies covers staffing and adoption more broadly.

Sensitive Content, Living Third Parties, and Restricted Segments

Oral histories are full of other people. A narrator recalling a factory closure names the foreman who was cruel and the coworker who drank. A woman describing her childhood names the uncle who abused her. A veteran describes a comrade's breakdown. A civil rights activist names the local officials who harassed her and the neighbors who informed on others. These people did not consent to anything. Some are still living, and many have children and grandchildren in the same town. In an analog reading room, the practical obscurity of an untranscribed tape protected them. Transcription and keyword search remove that obscurity, and a machine-generated index can put a name into a searchable synopsis that ranks in a web search.

This does not mean archives should scrub collections of difficult content. Much of the historical value of oral history lies precisely in testimony about injustice, conflict, and private experience that never made it into official records. It does mean that making an interview searchable deserves a deliberate review for a few specific categories: allegations of crimes or misconduct against identifiable living people, health information, sexual history, immigration status, information about minors, and statements that could expose the narrator or others to retaliation or embarrassment. Defamation risk is generally lower when the person discussed has died, but the reputational and relational harm to families can still be real.

AI can help with the triage, carefully. A language model running on a transcript you control can flag passages that mention named individuals alongside allegations, medical conditions, or other sensitive categories, so a human reviewer knows where to listen closely. It should never make the access decision itself, and it will miss things that depend on local knowledge, such as a nickname everyone in town recognizes. Use it as a highlighter, not a censor.

Restrictions themselves need careful handling in an AI workflow. If a narrator sealed a portion of an interview until 2030, the sealed passage should not be sent to a vendor for transcription, should not appear in a full-text index, and should not inform a public summary. That sounds obvious, but it is easy to break when a batch process runs on whole audio files. Segment the audio first, or process the full file only in an environment you control and then strip restricted segments from every derivative before anything is published. Make restrictions machine-readable in your collection system, with an end date and a reason, so that the next staff member or volunteer cannot accidentally undo them.

Can the Recordings Leave the Building? Cloud Vendors and Local Processing

Cloud transcription is convenient, fast, and often inexpensive, which makes it tempting to upload an entire collection at once. Before doing so, treat each upload as a disclosure. You are giving a copy of a narrator's voice and story to another company, under that company's terms. The questions to ask are the same ones you would ask of any vendor handling sensitive data: Does the service retain audio and transcripts, and for how long? Is customer content used to train or improve models, and can that be switched off? Where is data stored, and who at the vendor can access it? What happens to your data if the company is acquired or shuts down? Enterprise and API offerings frequently have stronger terms than consumer apps, but read the actual agreement rather than the marketing page. Our AI vendor evaluation checklist covers these questions in detail.

There is also a consent dimension. Even if a vendor's terms are excellent, a narrator may not have agreed to their interview being handled by outside companies, and some communities explicitly prohibit it. For open, low-sensitivity interviews with modern releases that mention service providers, cloud processing is usually reasonable. For sealed segments, community materials, and interviews with significant sensitive content, it often is not.

Local processing is a real alternative. Open-source speech recognition models, including Whisper itself, can run on an ordinary modern computer without sending audio anywhere. Several university libraries and archives have built local transcription workflows for exactly this reason, and free desktop applications now wrap these models in a simple interface. Local processing is slower on modest hardware and requires someone comfortable installing software, but it removes the vendor question entirely for the most sensitive material. Our guide to running AI locally for privacy explains the trade-offs and hardware considerations.

Many organizations end up with a split approach. Open collections with clear consent go to a reputable cloud service or a platform designed for archives. Sensitive, restricted, and community-controlled recordings are processed locally, or not at all, and the decision is recorded for each file. That approach lets you move quickly on most of the backlog while giving the hardest material the care it needs.

A Decision Framework, Interview by Interview

The question in this article's title, whether AI should index your oral history collection, rarely has a single answer for a whole collection. It has an answer for each interview, and sometimes for each segment. The most practical way to proceed is to sort the collection into groups and make a decision for each group, documenting your reasoning so it can be revisited.

Start with an inventory. For each recording, note the date, the narrator, whether a release exists and what it covers, whether the narrator is living, whether any restrictions were requested, whether the material involves a tribal nation or cultural community, the languages spoken, and a rough sense of sensitivity. AI cannot do this step for you, though it can help with spreadsheets and form letters. Then work through a short sequence of questions for each group.

First, does any community hold authority over this material, and have they been consulted? If not, pause. Second, does the release, read in light of what the narrator likely understood, support the access you are planning? If it is thin and the narrator or family can be reached, ask. Third, may the audio go to an outside service, or must processing stay local? Fourth, what level of output does this interview need: internal search text, an index, or a full transcript? Fifth, what review will that output get, and by whom? Sixth, which segments, if any, should be restricted, and are those restrictions enforced in every derivative? Only when those questions have answers should anything be published.

It is entirely reasonable for the outcome to be that some interviews are processed immediately, many are indexed and released over the next year as consent is refreshed, and a meaningful number stay offline. A collection that is mostly searchable with careful consent is better stewardship than one that is entirely searchable with surprises waiting in it. Write your decisions into a short collection policy approved by your board or collections committee, so the approach survives staff turnover. For a broader view of how this fits into organizational planning, see our guide to AI and nonprofit knowledge management.