llms.txt SEO: How 32 Top Sites Use It

We checked 59 big websites for an llms.txt file, and 32 have one, but nearly all of them sell software, while news sites and shops skip it.

An llms.txt is a plain text file that sits at the root of a site. It sums the site up for AI tools like ChatGPT, with links to the pages that matter.

Update the article date to match the survey date

Here are 7 lessons you can copy, plus the part you can't.

See what your rivals do

Start with the big picture. We sorted the sites into 5 groups and checked each one for the file.

Software apps and developer tools lead by a mile. Slack, Notion, Shopify, Stripe and GitHub all have one.

Publishers score 0 of 9. Not The New York Times, not the BBC, not CNN.

Zapier, which we studied too, is one of the software apps with a file.

Among the big shops and marketplaces, only Etsy has one. Amazon, eBay and Airbnb don't.

Our guess is simple. Software companies want AI tools to explain their product and recommend it.

Many publishers sell their words, so they'd rather keep AI tools out.

So find out what your own field does before you write a file. Open a rival's homepage, click the seotools.pro icon in our free browser extension, and its AI panel tells you whether they have an llms.txt.

Put one where your docs live

Here's the twist. The file is most common on docs sites, the part of a site where a company explains how its product works.

7 of the 8 docs sites we checked have one. Across the main sites, it's only about half.

Look at Anthropic, the company behind Claude. Its main site has no llms.txt, but its developer docs do.

Stripe has a file on its main site and on its docs. The docs site is where coders go to learn how to build with Stripe.

Every link on that page answers a "how do I" question. People ask AI tools questions like that all day long.

That's why a guide for AI tools helps most where your answers live. Your homepage sells, but your docs explain.

So if you have docs, a help center or guides, give that part of your site its own file. Then ask our AI Visibility Checker the questions your buyers ask, and see whether AI tools mention you.

Keep the file short

An llms.txt is meant to be a summary. Most files we found are, but a few are huge.

The smallest real files are 2 to 3 KB, about a page of text. Squarespace, Netlify and Supabase keep theirs that tight.

Twilio's file is about 2.3 MB. That's more than 800 times the size of Netlify's, and it holds more than 9K links.

No AI tool needs a post about a 2012 conference to understand what Twilio sells. A file this big stops being a summary and turns into a dump.

Zendesk and Mailchimp come next, at about 740 KB and 530 KB.

Your full list of pages already has a home. It's the sitemap, the file that lists a site's pages for Google.

So keep your llms.txt to the pages that matter most, and let the sitemap list the rest. Our XML Sitemap Generator builds that file for you.

Copy a simple layout

So what does a good file look like? Vercel's is a clean example, at about 5 KB with 25 links.

It follows the simple shape most good files share. First the name, then one line that says what you do.

Then come short sections, like Documentation or Help. Each line is a link with a few words on what's behind it.

That's all there is to it. You don't need code, just a text editor.

The hard part is picking the links. The pages that already answer people's questions are the ones AI tools need from you.

So build yours in this shape, with your best pages first. Our Organic Traffic Checker shows your organic traffic, meaning the people who find you on Google without ads, and lists the pages that bring the most.

Audit your file like a crawler

Having a file isn't enough. It has to reach a crawler, a program that visits pages on its own, as plain text.

So we asked each site for its file the way a crawler would.

32 sites sent a real file, and 20 had none. That part is simple.

But 6 sites turned us away. 5 said "forbidden", and Investopedia asked for payment.

That's often a firewall that blocks visitors it doesn't know. If it blocks us, it may block AI tools too, and then your file does nothing.

wikiHow is the odd one. It answered "OK", the code for success, but sent back a web page, not a text file.

So a status code alone can lie. You have to look at what actually comes back.

Here's the quick way to check yours. Open your site, click the seotools.pro icon in our browser extension, and look at the AI search readiness box.

On Stripe's docs, it finds the file and shows every AI crawler as allowed. That's a file doing its job.

Then check the whole site. Our Website Audit crawls your pages and checks for llms.txt and AI crawler access in its Technical SEO tab, next to the rest of your site's health.

Here's that card for the NHS website, from an audit we ran the same week.

The NHS lets every AI crawler in, but it has no llms.txt yet.

So open yoursite.com/llms.txt in a private window every few months. Make sure you see plain text, not a web page or an error, and run the audit after any big change.

Make robots.txt agree with it

An llms.txt invites AI tools in. But the real rules live in robots.txt, a file that tells crawlers what they may read.

The New York Times shows what a closed door looks like.

It has no llms.txt, and its robots.txt blocks 9 of the 10 AI crawlers our extension lists. Only Amazonbot gets through.

That's a clear choice, and both signals agree. You know exactly where the Times stands.

Stripe's docs, as you saw above, open the door wide. MDN, the coding guide we studied too, lets every AI crawler in as well.

The mistake is mixing the signals up. Meetup, which we studied too, has an llms.txt but blocks GPTBot, the crawler from the company behind ChatGPT.

Not every AI crawler does the same job. Some collect pages to train AI models, and some fetch a page to answer a question right now.

That means you can block training and still let answer tools quote you. Our robots.txt study, which we ran too, goes deeper into who blocks which crawler.

So decide first, then make both files say the same thing. Our Robots.txt Generator writes the rules, with a preset for each choice.

Treat it as a cheap test

Here's the honest part. No AI company has confirmed that it reads llms.txt the way search engines read robots.txt.

Even the AI labs are split. Only 3 of the 7 we checked have one on their main site, though most have one on their docs.

And don't add it hoping for better Google rankings. It's a note for AI tools, not for Google.

But it's cheap. A good file is about a page of text, and you can write it in an afternoon.

If AI tools start reading it, you're ready. If they don't, you've lost very little.

When you check, look for links as well as mentions. Our AI citation study, which we ran too, caught AI tools naming brands again and again without linking to them.

So add one, note the date, and watch what changes. Our AI Visibility Checker shows whether AI tools mention you, so you can compare before and after.

What you cannot copy

First, the part you can't copy: docs that people already read every day.

Stripe, GitHub and Vercel point their files at pages that coders use all the time. A file can't make your pages worth reading, so the pages have to come first.

Now a few honest notes on the numbers.

We asked every site for its file on 4 October 2026. Files change often, so some may look different by the time you read this.

We left out SEO software companies, our own included, so nobody grades their own homework.

We counted a file only when it came back as plain text. 6 sites refused us, so we can't say whether they have one.

We checked the root of each main site, plus 8 docs sites. A company may keep a file somewhere we didn't look.

The screenshots of the Vercel, Twilio and Anthropic files are a few weeks older than the survey. Their files may have changed since.

And this is 59 well-known sites, not the whole web. It shows what big brands do, not what works best.

The one number to take away

32.

That's how many of the 59 sites we checked have an llms.txt. Nearly all of them sell software, and docs sites have one even more often.

The good ones are short, plain and full of links to pages that answer questions. The bad ones are dumps, or they never reach the reader as plain text.

Write yours like Vercel did, keep it next to your answers, and make sure your robots.txt agrees with it.

You can run every one of these checks on your own site, and on your rivals', and see the same numbers for yourself.

Frequently asked questions