An llms.txt file is a plain markdown file you drop at the root of your site to hand AI models a clean, curated map of what you publish. It is a small thing to build, and I put one on this site a few months ago. Here is what it is for and how to write one that actually earns its place.

The problem it solves is simple. More and more of the traffic reasoning about your site is not a person in a browser — it is a language model pulling pages to answer a question. That model does not want your navigation, your cookie banner, your analytics scripts, or three hundred kilobytes of markup wrapped around four paragraphs of text. It wants the content, and it wants to know which pages matter. Left to its own devices it either burns context untangling your HTML or misses the pages you would most want it to read.

llms.txt is the answer the community landed on. The proposal came from Jeremy Howard in late 2024, and the idea is deliberately modest: a single markdown file, at /llms.txt, that tells a model what your site is and points it at the pages worth reading. Think of it as a cross between robots.txt and a sitemap, written for something that reads rather than crawls.

Where it lives and how it is shaped

The file sits at the root of your domain, so yourdomain.com/llms.txt. That location is the whole convention — a model or tool that wants it knows exactly where to look.

The format is markdown, and it follows a loose but specific shape:

  • An H1 with the name of the site or project. This is the only required line.
  • A blockquote right after it with a one or two sentence summary of what the site is.
  • Some optional prose filling in the important context — no headings, just plain paragraphs.
  • A series of H2 sections, each containing a markdown list of links. Every link is a URL with a short description of what it is.

In raw markdown, each of those pieces is a single, familiar bit of syntax:

# Site or Project Name          ← H1, the only required line

> One or two sentences on what this is.   ← blockquote summary

A short paragraph of plain prose adding the context that matters most — no headings, just text.

## Section Name                 ← H2 grouping related links
- https://example.com/page — what this page is and why it matters
- **Bold Label** — wrap text in ** to bold it, handy for naming an item

## Optional                     ← reserved heading for skippable links
- https://example.com/deep-dive — nice to have, safe to skip when context is tight

That is the entire grammar. A # for the title, a > for the summary, plain lines for prose, ## for each section, and - list items pairing a URL with a description. The one bit of inline formatting worth knowing is **double asterisks**, which renders the wrapped text in bold — I use it in my Services section to make each service name stand out from its description. Nothing else is required.

Markdown is not an accident here. It is readable to a person, trivial for a model to parse, and it keeps the file honest — you cannot hide a mess inside clean markdown the way you can inside HTML. If you can write a decent README, you can write an llms.txt.

Here is the shape in practice. This is the file that sits at robbclarke.com/llms.txt, trimmed in the middle so you can see the structure without scrolling past every article I have written:

# Robb Clarke

> AI consultant and Head of AI at RB2B. 20+ years in web development. Helps SaaS and B2B teams build AI support agents that resolve tickets — not just answer questions. Based in Fredericton, NB, Canada.

Robb Clarke is a full-stack developer turned AI strategist. He leads AI at RB2B and MoltSets, where he built an AI support agent that handles 98.5% of support inquiries autonomously.

## Key Pages

- https://robbclarke.com — Home / overview
- https://robbclarke.com/about-me — Biography, career history, published books
- https://robbclarke.com/services — AI consulting services
- https://robbclarke.com/applied-ai — AI writing and case studies (full archive)
- https://robbclarke.com/contact — Contact / get in touch

## Services

- **AI Support Agent Audit** — Deep review of how an AI support agent behaves in production: conversation analysis, prompt structure, escalation logic
- **AI Agent Training & Prompt Design** — Hands-on rewriting of prompts and conversation patterns for better intent, tone, and decision-making
- ...

## Writing — Applied AI

Articles covering AI implementation, strategy, case studies, and tools.

- https://robbclarke.com/applied-ai/llms-txt-best-practices — Best practices for writing an llms.txt file that AI models actually use (2026-07-07)
- https://robbclarke.com/applied-ai/building-the-moltsets-skills-library — How hand-packaged Claude Skills became a community library of MoltSets workflows (2026-07-02)
- ... (35+ more, newest first)

Nothing clever is going on. An H1 with my name, a blockquote that says what the site is in one breath, a sentence of context, then sections that group the links by what they are. Everything after this point in the article is really just a closer look at the decisions inside that file.

Curate, do not dump

This is the practice that matters most, and it is the one people get wrong first. An llms.txt is a map, not a mirror. The temptation is to list every page you have, the same way a sitemap does. Resist it. A sitemap exists so crawlers can find everything; llms.txt exists so a model can find the right things quickly.

Pick the pages that actually explain what you do and what you know. On this site that is the home page, the about page, the services, and the writing archive — plus the individual articles, because those are the substance. I left off the thank-you pages, the privacy policy, and the rest of the plumbing. Nobody is asking a model to summarise my privacy policy, and every line I add that does not need to be there makes the useful lines harder to find.

The description is the part that does the work

A bare list of URLs is close to useless. The value is in the note next to each one. Compare these:

  • - https://robbclarke.com/applied-ai/when-ai-resolves-not-just-answers
  • - https://robbclarke.com/applied-ai/when-ai-resolves-not-just-answers — The difference between AI that answers and AI that resolves

The second one tells a model whether the page is worth opening before it spends the tokens to open it. Write the description the way you would write it for a colleague who is deciding whether to click. Say what the page is and why someone would read it. A slug is not a description, and the model should not have to guess from the URL.

Put the important things first

Order is signal. Whatever sits at the top of the file, and at the top of each section, reads as the thing you consider most important. I put the home and about pages first, then services, then the writing. Inside the writing section the articles run newest to oldest, which doubles as most-to-least current. You are not just listing pages, you are ranking them, so rank them deliberately.

Use the Optional section for what can be skipped

The spec includes one special heading: a section titled Optional. Anything you list under it is fair to skip when context is tight. This is a genuinely useful lever. Secondary references, deep-cut archive material, background reading — park it under Optional and you are telling the model, in effect, "read this if you have room, drop it if you don't." Everything above Optional is the core you want read every time.

Consider a full version, and clean markdown pages

Two companions are worth knowing about. The first is /llms-full.txt, which holds the actual content of your key pages inlined into one file rather than just links to them. It saves a model the round trips of fetching each page, at the cost of a much larger file. If your important content is small and stable, it is worth doing. If it changes constantly or runs long, the maintenance is not worth it.

The second is offering clean markdown versions of individual pages — the same URL with a .md extension, serving the content without the site chrome. If you already generate your pages from markdown, this is nearly free, and it means a model that follows a link from your llms.txt lands on the content rather than the wrapper.

It will rot, so keep it current

An llms.txt is a snapshot, and a stale one is worse than none — it points a model confidently at pages that have moved or descriptions that no longer match. Mine updates whenever I publish a new article, because a writing archive that stops at last quarter tells the wrong story about how active the site is. Whatever your update trigger is, tie the file to it so it does not drift. If you can generate it from the same data that drives your pages, do that; a file a human has to remember to edit is a file that goes out of date.

It complements the files you already have

Worth being clear on this: llms.txt does not replace robots.txt or your XML sitemap, and it is not a permissions system. robots.txt tells crawlers where they may go. The sitemap helps search engines find everything. llms.txt helps a reasoning model understand what matters. They serve different readers and they sit alongside each other. Adding one does not mean removing anything.

A note on how much this pays off today

I will be straight about the state of things. Adoption is early. Not every model or tool reads llms.txt yet, and the ones that do are still working out how much weight to give it. If you are looking for a guaranteed traffic lever this quarter, this is not it.

I built one anyway, and I would tell you to as well. The cost is an afternoon and a file you can generate from data you already have. The upside is that as more tools do adopt the convention — and the direction of travel is clearly towards models consuming the web, not away from it — you are already legible to them. It is the same logic as writing good meta descriptions before you can prove any single one moved a number. You do it because it is cheap, it is correct, and the world is moving that way. Curate the map, describe every link, keep it current, and you have done the job.