Sometime in the last year you probably read that your website needs an llms.txt file.
The pitch is appealing. One small text file at the root of your domain, listing your most important pages in plain markdown, so that AI assistants can understand your site properly and start recommending you. It takes an afternoon. Everyone seems to be adding one.
So we checked who actually had. We fetched /llms.txt from 67 well-known websites across software, retail, publishing and AI, and the result splits so cleanly down one line that it answers the question by itself.
Not a single publisher or retailer had one. More than eight in ten developer-tool companies did.
That gap turns out to be correct, and understanding why will save you an afternoon.
Results at a glance
llms.txt is not an SEO file. It is a documentation file, read by coding assistants when a developer points one at your docs.
If you sell to developers, it is genuinely useful. If you publish articles or sell products, it will do nothing for you, and Google has said so in writing.
We checked 67 websites
The test is one anyone can repeat: request /llms.txt and see whether a real markdown file comes back rather than a 404 page dressed up as one.
Sorted by what the company actually sells, the pattern is not subtle.
Vercel, Netlify, Cloudflare, Stripe, Supabase, GitHub, MongoDB, Next.js, React and Svelte all publish one. So do Notion, Slack, Zapier, HubSpot, Salesforce, Atlassian, Shopify and Linear.
The New York Times does not. Nor does The Verge, Wired, the BBC, Wikipedia, Amazon, eBay or Etsy.
The file that names the answer
Here is the detail that gives the whole thing away.
anthropic.com does not have an llms.txt file. docs.anthropic.com does. The same is true of Perplexity: nothing on the marketing site, a full file on the documentation subdomain.
An AI company that publicly recommends this file did not put one on its own homepage — because the homepage is not what it is for.
Look at what a good one actually contains:
A title, one sentence describing the company, then a short list of links to documentation, an API description and a knowledge base. Roughly a page and a half of text.
That is the file working as intended.
What Google actually says
Now the part that most articles about this leave out.
Google's own documentation for site owners is explicit: you do not need to create machine-readable files, AI text files or markup to appear in Google Search or its generative features, because "Google Search itself doesn't use them". It states plainly that doing so will neither help nor harm your visibility.
The same guidance lists writing unnecessary AI text files among the tactics you can safely ignore, alongside chunking your content and chasing inauthentic mentions.
So if your goal is to be quoted in an AI Overview, or to be recommended when somebody asks a chatbot for the best tool in your category, this file is not the lever. There is no public evidence that any major AI provider's answer engine reads it in production.
What it is really for
Which brings us to what the file genuinely does, and why the developer-tool companies were right to adopt it.
When a developer asks a coding assistant to "read the Stripe docs and write the integration", the assistant has to decide what to fetch. It cannot read an entire documentation site, and crawling around a JavaScript-heavy docs portal is slow and lossy.
An llms.txt file solves that specific problem. It hands the assistant a curated table of contents in plain markdown, so it can jump straight to the three pages it needs. Coding tools read it when a user points them at a domain, and there is now even a browser audit that checks whether your site provides one.
So the file is real, and it is useful. It is simply useful in a developer's editor, not in a search result.
That is exactly why 83% of developer-tool companies have one and no publisher does. The market already worked this out. The people it helps adopted it, and the people it does not quietly ignored it.
Some of them are a megabyte
One more thing worth knowing before you write yours, because it is the most common way to get it wrong.
The whole idea is a short, curated map. Most companies understood that: the typical file we found was about 12 KB, listing 75 links. Vercel's is under 2 KB and lists 7.
Then there is the tail. Twilio's file is 2.3 megabytes and contains 7,324 links. Salesforce's is 1 MB. Datadog's is over half a megabyte.
At that size it is not a curated map any more — it is a sitemap in markdown, and it defeats the point. The reason the file exists is that an assistant cannot read everything and needs to be told what matters. Handing it 7,324 links tells it nothing.
The specification has an answer for this, and Vercel's file shows it: keep llms.txt short, and link out to a separate llms-full.txt for anything that wants the whole dump.
Should you write one?
Four questions, in order. It takes a minute.
1. Do developers read your documentation? If yes, write one. This is the case it was designed for and it works. If no, skip to question four.
2. Are you doing it for AI search visibility? Then do not bother. Google has said in writing that it changes nothing, and no answer engine has committed to reading it.
3. If you are writing one, keep it small. A title, one sentence about what you do, and links to your genuinely important documentation. If it goes past a few dozen links, you have built a sitemap by accident.
4. Spend the afternoon on the thing that does move AI visibility instead: being mentioned, cited and quoted on pages that models actually read. That is the same work that earns links, which is why it was never a shortcut.
What we could not measure
We cannot prove a negative. Our evidence that answer engines ignore this file is Google's own published guidance and the absence of any commitment from other providers — not access to their systems. If that changes, this article ages.
67 sites is a sample, not a census. We picked recognisable names across six categories. A different 67 would move the percentages, though a 0% and an 83% are unlikely to swap places.
The visibility check is one brand, one day. We ran Zapier through our AI Visibility Checker in a normal browser and reported exactly what came back. A single brand is an illustration, not evidence of a rule, and the platforms move constantly.
A file existing is not a file being good. We checked that a real markdown file came back and measured its size and link count. We did not grade the writing inside it.
We tested it on a site that has one
Zapier publishes an llms.txt. So we ran Zapier through our own AI Visibility Checker, which puts a set of real questions to several assistants and records who gets named and who gets cited.
Zapier was named in 16 of 24 answers, across 4 of the 6 platforms tested. ChatGPT named it in 4 of 6, ChatGPT Search 4 of 6, Perplexity 5 of 6, Gemini 3 of 6.
Then the number that matters for this article:
Zero.
Sixteen mentions, no citations. A well-formed llms.txt sitting at the root of the domain the whole time.
That is one brand on one day and it proves nothing on its own — but it is exactly what you would expect if the file does nothing for answer engines, and exactly what Google's guidance already told us to expect.
It also shows what the useful question actually is. Not "do I have the file" but "does anything name me, and does anything link to me". Those are measurable. The file is not.
This is the question llms.txt cannot answer. Our AI Visibility Checker puts real questions to AI assistants and reports whether your brand gets named, which pages get cited, and who is being recommended instead of you.
- Which answers name your brand, and which name a competitor
- The pages and sources the AI actually cites
- 3 checks a day, no signup, no card
Frequently asked questions
What is an llms.txt file?
A plain markdown file at the root of a domain, at /llms.txt, proposed as a standard way to give AI systems a short curated map of a site. The format is a title, a one-line summary in a blockquote, then sections of annotated links. It is unrelated to robots.txt, which controls crawling; llms.txt controls nothing and simply offers a reading list.
How many of the sites you checked had one?
35 of 67, or 52%, on 22 August 2026. By category: developer tools 15 of 18, SaaS 13 of 18, SEO tools 4 of 8, AI labs 3 of 7, publishers 0 of 9 and retailers 0 of 7.
Does llms.txt help me rank or get cited in AI answers?
There is no evidence that it does. Google's documentation for site owners states that you do not need machine-readable or AI text files to appear in Search or its generative features, that Google Search does not use them, and that creating them will neither help nor harm your visibility. No other major provider has committed to reading the file in production either.
Then why do so many companies publish one?
Because it works for a different audience. Coding assistants read llms.txt when a developer points them at a domain, using it as a table of contents so they can fetch the few documentation pages they need rather than crawling a whole site. That is genuinely valuable if your customers are developers, which is precisely the group that adopted it.
Why does Anthropic not have one on its homepage?
It does have one, but on docs.anthropic.com rather than anthropic.com. Perplexity is the same: nothing on the marketing domain, a full file on the documentation subdomain. That placement is the clearest possible statement of what the file is for.
How big should the file be?
Small. Across the 35 files we found, the median was about 12 KB with 75 links, and the smallest was Vercel's at under 2 KB with 7 links. At the other extreme, Twilio's is 2.3 MB with 7,324 links and Salesforce's is 1 MB, which is a sitemap rather than a curated map. If you need to publish everything, the convention is to keep llms.txt short and point to a separate llms-full.txt.
Is llms.txt the same as blocking AI crawlers?
No, and they are easy to confuse. Blocking is done in robots.txt, by disallowing crawlers such as GPTBot or ClaudeBot. llms.txt has no enforcement of any kind — it is a suggestion, not a permission. We checked the sites publishing one and none of them were simultaneously blocking the major AI crawlers, so at least the two are being used consistently.
Can I check all of this myself?
Easily, and you should. Open any website with /llms.txt on the end and see what comes back — a real one starts with a heading and a short description, while a missing one usually returns a styled 404 page rather than an honest error. That is the entire method behind this article.
Decide what you are optimising for
Every case study in this series has been about a company that found something real to build: measurements, currencies, templates, hotel themes, sold houses, a small pile of calculators, 880,000 drawings.
This one is about the opposite — a file that is quick to add, easy to write about, and does nothing for most of the people being told to add it.
It is not a scam. For a company whose documentation gets read by coding assistants, it is a small, sensible courtesy. But it is not an SEO tactic, and the adoption pattern shows the industry already knows: the people it helps have one, and the people it does not have quietly moved on.
If AI visibility is what you are actually after, the work is unchanged and unglamorous: be worth citing, on pages that get read.
Start by finding out where you stand — check whether AI mentions you. It takes a minute, and unlike a text file, the answer is real.
Checked on 22 August 2026 by requesting /llms.txt from 67 domains and confirming a genuine markdown file was returned rather than an HTML error page. Category counts and file sizes are in the FAQ above.
Google's position is quoted from its published guidance for site owners on optimising for generative AI features in Search. Guidance changes; re-check before relying on it.