Back to Articles
    AI Search Visibility

    llms.txt, Schema Markup, and AI Crawlers: A Technical Checklist for Nonprofit Websites

    More and more people learn about causes by asking an AI assistant rather than typing into a search box. When someone asks ChatGPT, Gemini, or Perplexity about a problem your nonprofit works on, whether your organization shows up in the answer depends on how well those systems can find, read, and trust your website. This is a plain-language technical checklist for making that possible.

    Published: July 20, 202613 min readAI Search Visibility
    A nonprofit website structured so AI crawlers and answer engines can read it clearly

    For twenty years, being found online meant ranking on a page of blue links. That world is changing quickly. A growing share of people now ask an AI assistant a question and read the answer it composes, often without ever clicking through to a website at all. For nonprofits, this shift is both an opportunity and a risk. The opportunity is that a well-structured site can become a trusted source that AI systems quote when someone asks about your issue area. The risk is that a site the machines cannot read cleanly simply disappears from these new answers, no matter how good the underlying work is.

    The good news is that most of what makes a website legible to AI is technical hygiene, not marketing spend. It comes down to giving automated readers clear signposts: a machine-readable summary of your site, structured data that spells out who you are, crawler rules that decide who gets in, and clean underlying code. None of this requires a large budget. It requires a comms person or a web contractor willing to spend a focused day or two, guided by a clear list of what to do and why.

    This article is that list. It explains, in language a non-technical leader can follow, how AI crawlers and answer engines actually work, what an llms.txt file is and how to create a basic one, which pieces of Schema.org structured data matter most for nonprofits, how to control which AI crawlers can read your site, and the foundational elements of clean code, performance, and consistency that tie everything together. It closes with a concrete step-by-step checklist you can hand to whoever maintains your site.

    If you want the strategic picture behind these tactics, our overview of generative engine optimization versus traditional SEO sets the context. This piece is the hands-on companion: the specific technical steps that make your organization readable to the systems now standing between your cause and the people who care about it.

    How AI Crawlers and Answer Engines Read Your Site

    To make your site legible to AI, it helps to understand the two very different jobs happening behind the scenes. The first is crawling, which is when automated programs, called crawlers or bots, visit web pages and copy their content so it can be used later. The second is answering, which is when a tool like ChatGPT or Perplexity assembles a response to a user's question, sometimes drawing on what was crawled earlier and sometimes fetching a live page in the moment.

    There are broadly three ways an AI system might encounter your content. Some crawlers gather large amounts of text to help train future models. Some fetch a page in real time because a user asked a question the assistant wants to answer with current information. And some answer engines lean heavily on traditional search indexes, layering an AI summary on top of results a conventional search engine already ranked. Each pathway rewards the same underlying quality: content that is easy for a machine to locate, parse, and trust.

    The critical thing to grasp is that these systems do not see your website the way a person does. They do not admire your photography or your color palette. They read the underlying code and text. If your key facts, your mission, your services, your contact details, live only inside an image, a video, or a script that loads content after the fact, a crawler may never register them. Everything in this checklist exists to move the information that matters out of the decorative layer and into a form machines can reliably read.

    Training Crawlers

    Bots that collect text to help build future AI models. What they gather shapes what an assistant later knows about your cause and your organization.

    Live Retrieval

    Bots that fetch a page at the moment a user asks a question, so the assistant can answer with current, specific information rather than memory alone.

    Index-Backed Answers

    Answer engines that build on a traditional search index, meaning your existing search visibility still feeds directly into AI-generated results.

    The llms.txt File: A Map You Write for AI

    One of the newer ideas in this space is a simple text file called llms.txt. The concept is straightforward. You place a plain-text file at the root of your website, at an address like yoursite.org/llms.txt, that acts as a curated guide to your most important content, written specifically for large language models. Think of it as a hand-drawn map that points AI systems to the pages you most want them to understand, along with a short description of what each one covers.

    It is important to be honest about the status of llms.txt. It is an emerging, voluntary proposal, not an official standard that every AI system obeys today. Some tools may look for it, others may ignore it. The reason to create one anyway is that the cost is close to zero and the discipline is valuable. Writing an llms.txt file forces you to decide what the most important, most trustworthy pages on your site actually are, and to describe them in clear language. That clarity helps human visitors too.

    The format is designed to be readable by both people and machines. It uses simple Markdown, which is a lightweight way of writing structured text with headings and links. A basic file for a nonprofit might look something like the example below. You do not need to be a programmer to write one; if you can edit a document, you can draft this.

    # Helping Hands Community Trust
    
    > A registered nonprofit providing emergency shelter, meals,
    > and case management to people experiencing homelessness.
    
    ## About
    - [Our Mission](https://example.org/about/): Who we are, what we do, and the communities we serve.
    - [Programs](https://example.org/programs/): Shelter, meals, and case management services in detail.
    
    ## Get Involved
    - [Donate](https://example.org/donate/): How your gift is used and where it goes.
    - [Volunteer](https://example.org/volunteer/): Current volunteer roles and how to apply.
    
    ## Trust and Contact
    - [Financials](https://example.org/financials/): Annual reports, EIN, and accountability information.
    - [Contact](https://example.org/contact/): Address, phone, and email for our team.

    Notice how the file leads with a one-line summary of the organization, then groups the most useful pages under clear headings with short descriptions. The goal is not to list every page. It is to point to the handful of pages that best answer the questions people ask about your work. Keep it focused, keep the links accurate, and update it whenever your key pages change. A stale map is worse than no map, because it sends readers to information that no longer exists.

    Schema Markup: Telling Machines Exactly What You Are

    Schema markup, also called structured data, is the single most powerful technical step most nonprofit sites are missing. Here is the idea in plain terms. When a person reads your homepage, they understand from context that Helping Hands is a charity, that a phone number is a phone number, and that a paragraph is describing your mission. A machine does not have that context automatically. Schema markup is a small, invisible layer of labeled information you add to your pages that spells out, in a vocabulary machines already understand, exactly what each piece of information means.

    The shared vocabulary comes from Schema.org, a project supported by the major search companies. It is usually added to a page in a format called JSON-LD, a block of labeled data that sits quietly in the page code without changing how the page looks to visitors. A web developer can add it in an afternoon, and many website platforms and plugins can generate it for you. The payoff is that AI systems and search engines can read your identity directly rather than guessing at it.

    A minimal example for a nonprofit's organization details is short and readable. The block below labels the organization as a nonprofit, states its name and mission, and provides its official identifier and contact point. This is the kind of information you want machines to have exactly right.

    <script type="application/ld+json">
    {
      "@context": "https://schema.org",
      "@type": "NGO",
      "name": "Helping Hands Community Trust",
      "url": "https://example.org/",
      "description": "Emergency shelter, meals, and case management for people experiencing homelessness.",
      "nonprofitStatus": "Nonprofit501c3",
      "taxID": "12-3456789",
      "email": "[email protected]",
      "telephone": "+1-555-123-4567"
    }
    </script>

    You do not need every schema type, only the ones that match the content you actually publish. The cards below cover the types most relevant to nonprofits. Add the ones that fit, and skip the rest. Getting a few correct is far better than sprinkling many that are inaccurate, because incorrect structured data erodes the trust these signals are meant to build.

    Organization and NGO

    The foundation. Use the NGO type, with NonprofitType status where it applies, to state your name, mission, website, tax identifier, and contact details. This is the block that tells AI who you are.

    Article

    For blog posts, guides, and news. Labels the headline, author, publish date, and description so answer engines can quote your content with proper attribution.

    FAQPage

    For question-and-answer sections. Because people phrase requests to AI as questions, clearly labeled question-answer pairs are especially easy for these systems to pick up and reuse.

    Event and DonateAction

    Event markup describes galas, drives, and volunteer days with dates and locations. DonateAction, where relevant, signals a clear giving pathway so an assistant can point supporters toward how to help.

    Structured data is also what gives you a fighting chance when an AI answers a question without sending anyone to your site. Clear markup makes it more likely your organization is named and credited in that answer, which is one small defense against the trend we explore in our piece on the zero-click donation problem. For a broader treatment of how structured content improves discoverability, see our guide to SEO for nonprofits in the age of AI.

    robots.txt: Deciding Which AI Crawlers Get In

    Your website already has, or should have, a file called robots.txt at its root. It is a long-standing convention that lets you tell automated visitors which parts of your site they may or may not access. Well-behaved crawlers read it and follow its instructions. In recent years, the major AI companies have introduced named crawlers that you can specifically allow or block in this file, which puts a real decision in your hands.

    A few names are worth knowing. GPTBot is OpenAI's crawler associated with ChatGPT. Google-Extended is the control Google offers for whether your content is used to improve its AI products, separate from normal search indexing. PerplexityBot is the crawler for the answer engine Perplexity. ClaudeBot is Anthropic's crawler. There are others, and the list changes, but these represent the main systems most nonprofits care about. The example below shows how you might explicitly welcome them.

    # robots.txt
    
    User-agent: GPTBot
    Allow: /
    
    User-agent: Google-Extended
    Allow: /
    
    User-agent: PerplexityBot
    Allow: /
    
    User-agent: ClaudeBot
    Allow: /
    
    Sitemap: https://example.org/sitemap.xml

    The real question is whether to allow these crawlers at all, and this is a genuine trade-off rather than a settled answer. Allowing them means your content can inform AI answers and, in some cases, future models, which increases the chance your organization is surfaced and cited when people ask about your cause. Blocking them protects your material from being absorbed into systems you do not control, which some organizations prefer for reasons of principle, licensing, or the sensitivity of their content.

    Reasons to Allow

    • Your work can be surfaced and cited when people ask AI about your issue
    • Accurate information about your mission reaches these systems directly
    • You stay visible as more people research causes through assistants

    Reasons to Block

    • You want to keep original research or licensed content out of training data
    • Your content covers sensitive communities you would rather not expose
    • Organizational principle favors control over reach for now

    Two cautions are worth stating plainly. First, robots.txt is a request, not a lock. Reputable crawlers honor it, but it is not a security control and will not stop bad actors, so never rely on it to protect confidential material, which should not be publicly accessible in the first place. Second, you can be selective. You might allow live-retrieval crawlers so your content can be cited in real time while declining the ones used for training, or open most of your site while keeping certain directories closed. Most nonprofits benefit from allowing the mainstream crawlers, which is why we discuss visibility gains at length in our look at how ChatGPT recommends charities.

    Clean Code, Speed, and Consistency: The Foundations

    The files above sit on top of a foundation, and the foundation matters more than any single trick. If the underlying page is messy, slow, or inconsistent, no amount of markup will fully compensate. Four foundational habits do most of the work, and none of them are exotic. They are simply the marks of a well-built website that happens to be exactly what machines need too.

    The first is clean, semantic HTML. Semantic simply means using the right structural element for the right job: a real heading marked as a heading, a real list marked as a list, one clear main heading per page followed by an orderly hierarchy of subheadings. When your page has a logical outline that a machine can follow, it can extract your key points reliably. When everything is a visually styled block with no structural meaning, machines have to guess, and they often guess wrong. Alongside this, put your most important facts, your mission, services, contact details, and tax identifier, into plain readable text rather than locking them inside images or videos.

    The second is speed and accessibility. Pages that load quickly and work on modest devices are easier for crawlers to process, and the same practices that make a site usable for people with disabilities, descriptive text for images, clear link labels, sensible reading order, also make it far more legible to machines. Accessibility and machine readability are two sides of one coin, which means the work you do for one audience quietly serves the other.

    The third is consistency across the wider web. AI systems build confidence in a fact when they see it stated the same way in several places. If your organization's name, address, phone number, and mission match across your own site, your listing on charity databases, your social profiles, and any directories you appear in, machines treat that information as more trustworthy. Contradictions, an old address here, a slightly different name there, create doubt that can push you out of an answer.

    Semantic Structure

    One clear main heading per page, an orderly hierarchy of subheadings, real lists, and key facts in plain text rather than buried in images or scripts.

    Speed and Accessibility

    Fast pages, descriptive image text, clear link labels, and a sensible reading order that serves both people using assistive technology and machines reading the code.

    Consistent Identity

    The same name, address, phone number, and mission everywhere you appear, from your own site to charity databases and social profiles, so machines trust the facts.

    Machine-Readable Trust

    Your EIN, financial reports, and accountability information stated in plain text and structured data, so an assistant can verify you are a legitimate organization.

    The Step-by-Step Checklist

    Here is everything above distilled into an ordered sequence you can hand to whoever maintains your website. Work through it roughly in order, because the early steps make the later ones easier. Most nonprofit sites can complete the whole list in a focused day or two of work, and the result is a site that both people and AI systems can read with confidence.

    Foundations First

    The groundwork that everything else builds on

    • Audit your key facts: confirm your mission, services, contact details, and tax identifier appear as plain readable text, not only inside images or videos.
    • Check heading structure: ensure each important page has one clear main heading followed by an orderly hierarchy of subheadings.
    • Test speed and accessibility: run your site through a free page-speed and accessibility checker and fix the highest-impact issues it reports.
    • Confirm a sitemap exists: make sure you have an XML sitemap listing your pages, and that it is referenced in your robots.txt file.

    Add the Machine-Readable Layer

    Structured data, crawler rules, and your llms.txt map

    • Add Organization or NGO schema: place structured data on your homepage stating name, mission, website, tax identifier, and contact point.
    • Mark up the content types you publish: add Article schema to posts, FAQPage schema to question sections, Event schema to events, and DonateAction where relevant.
    • Validate your structured data: run each marked-up page through a free structured-data testing tool and correct any errors before moving on.
    • Decide your crawler policy: discuss allowing or blocking GPTBot, Google-Extended, PerplexityBot, and ClaudeBot, then set your robots.txt to match that decision.
    • Create a basic llms.txt file: write a short, curated map of your most important pages and place it at the root of your site.

    Consistency and Maintenance

    Keeping everything accurate over time

    • Reconcile your details everywhere: make your name, address, phone, and mission identical across your site, charity databases, and social profiles.
    • Publish clear trust signals: put your EIN, annual reports, and accountability information on an easy-to-find page in plain text.
    • Set a review reminder: revisit your llms.txt, schema, and crawler rules whenever key pages change, and at least a couple of times a year.
    • Watch how you appear: periodically ask a few AI assistants about your issue area and note whether your organization is mentioned accurately.

    If your organization is early in its AI journey and this list feels daunting, start with the foundations and the Organization schema, then add the rest over time. Even the first few steps meaningfully improve how machines read your site. For a gentler on-ramp to the whole subject, our guide for nonprofit leaders getting started with AI puts this technical work in the context of a broader, manageable plan.

    Conclusion: Build for the Readers You Cannot See

    For most of the web's history, nonprofits built their websites for one kind of reader: a person with eyes, scrolling and clicking. That reader still matters enormously. But there is now a second audience reading your site constantly, quietly, and at scale, and it does not scroll or click. It parses code, follows structure, and decides whether your organization is clear and trustworthy enough to include when someone asks a question. The work in this checklist is simply the courtesy of writing for that second reader too.

    What makes this encouraging is that almost none of it competes with your existing priorities. Clean structure, fast pages, accessible design, consistent information, and honest trust signals are exactly what serve human visitors best. The llms.txt file, the schema markup, and the crawler decisions are a thin, low-cost layer on top of good fundamentals. You are not building a separate site for machines. You are making your one good site legible to everyone, including the systems that increasingly stand between your cause and the people who care about it.

    Start with the foundations, add the machine-readable layer, keep your information consistent, and revisit the whole thing a couple of times a year. Do that, and when the next supporter asks an assistant how they can help with the problem your nonprofit exists to solve, there is a real chance the answer includes your name, accurately, and points them toward the door you have worked so hard to keep open.

    Make Your Site Legible to AI

    We help nonprofits audit their websites, add structured data and llms.txt files, and set crawler policies that fit their goals, so the AI systems people now ask for advice can find, read, and trust your organization. If you want a clear plan tailored to your site, we can help.