Robots.txt Review Template With AI Crawler Rules

Standard Production Site

When to use: A normal live site that wants full crawling plus a sitemap pointer.

Use when: you run a healthy production site and want search engines to crawl everything freely while finding your sitemap fast.

Directives

• User-agent: *
• Disallow: (leave the value blank to allow all)
• Sitemap: [https://example.com/sitemap.xml]

An empty Disallow means nothing is blocked. You do not need an Allow line for this; allow-by-default is the standard behavior.

Optional hygiene

• Disallow: [/wp-admin/] (block a real admin path if you have one)
• Allow: [/wp-admin/admin-ajax.php]

Note: robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and add a noindex meta tag, or protect it behind a login. Place the file at the domain root: [https://example.com/robots.txt].

Staging, Dev or Pre-Launch Site

When to use: A non-public environment you must keep entirely out of search.

Use when: the site is a staging server, development copy, or pre-launch build that no one should find in search.

Directives

• User-agent: *
• Disallow: /

A single Disallow: / tells compliant crawlers to skip every URL on the host. This is intentional here and dangerous everywhere else.

Critical warning: never let this line reach production. The most common deindexing disaster is a Disallow: / copied from staging to the live site during deployment.

Stronger protection

• Robots.txt is a request, not a lock. Rogue crawlers ignore it.
• Add HTTP authentication (a username and password) so the environment is truly private.
• If you add a site-wide noindex as a backup, don't also block the page with Disallow: /: a blocked page never reveals its noindex tag.

Note: if a staging URL was already crawled, do not rely on Disallow: / to remove it. Blocking the URL also blocks Google from seeing a noindex tag. Allow crawling temporarily with noindex, or use the removal tools in Search Console.

E-commerce Crawl Control

When to use: A store that needs to steer crawl budget away from low-value URLs.

Use when: a store generates endless faceted, parameter, and account URLs that waste crawl budget and dilute signals.

Typical blocks

• User-agent: *
• Disallow: [/cart]
• Disallow: [/checkout]
• Disallow: [/account/]
• Disallow: [/search] (internal site search results)
• Disallow: [/*?*sort=] (a sort parameter anywhere in the query string)
• Disallow: [/*?*filter=] (faceted filters)
• Disallow: [/*.pdf$] (block any URL ending in .pdf: $ anchors the end)

Use * to match any sequence and $ to anchor the end of a URL. Paths are case-sensitive, so [/Search] and [/search] are different rules.

Caution: only block parameters that create true duplicates or thin pages. Blocking a parameter that changes the main product content (or your canonical URLs) can hide pages you want ranked. Prefer canonical tags for duplicate content and reserve robots.txt for crawl waste you never want fetched.

Allow Search Engines, Block Scrapers

When to use: Welcome major search bots while turning away aggressive or unwanted crawlers.

Use when: you want Google, Bing, and other major engines to crawl freely but want to discourage known scrapers and bandwidth-heavy bots.

Allow the engines you want

• User-agent: Googlebot
• Disallow: (blank)
• User-agent: Bingbot
• Disallow: (blank)

Discourage a specific crawler

• User-agent: [BadBot]
• Disallow: /

Each User-agent block is matched independently, and a bot follows the most specific group that names it. Keep a final User-agent: * group so unnamed bots still get clear instructions.

AI crawlers need their own decision: some feed AI search answers, others collect training data. See the AI Crawler Decisions variant before you block any of them.

Reality check: robots.txt is voluntary. Well-behaved engines obey it; malicious scrapers can ignore it and can even use it to find paths you named. Major AI vendors document that their crawlers follow it, though user-initiated fetchers may not. For real enforcement, block by user-agent or IP at the server or firewall, or use rate limiting. Treat robots.txt as guidance, not security.

Sitemap & Multiple-Sitemap Declaration

When to use: Point crawlers to one or several sitemaps, including a sitemap index.

Use when: you want every crawler to discover your sitemaps, especially a large site split across multiple files.

Single sitemap

• Sitemap: [https://example.com/sitemap.xml]

Multiple sitemaps

• Sitemap: [https://example.com/sitemap-posts.xml]
• Sitemap: [https://example.com/sitemap-products.xml]
• Sitemap: [https://example.com/sitemap-images.xml]

Or one sitemap index

• Sitemap: [https://example.com/sitemap_index.xml]

Rules to follow:

• List each sitemap on its own line. There's no cap on the number of Sitemap lines, but each individual sitemap file is limited to 50,000 URLs and 50MB uncompressed: split larger sites across multiple files or a sitemap index.
• Always use a full absolute URL with the correct protocol (https) and host.
• The Sitemap directive is independent of User-agent groups, so place it anywhere in the file.
• Submitting sitemaps in Search Console and Bing Webmaster Tools is still recommended as a backup.

Pre-Publish QA Checklist

When to use: Catch the mistakes that quietly deindex a site before you ship the file.

Use when: you are about to publish or replace a robots.txt file and want to avoid the classic, costly errors.

Check each item

1. Confirm the file is at the root: [https://example.com/robots.txt]. A file in a subfolder is ignored.
2. Make sure there is no stray Disallow: / left over from staging.
3. Verify casing on every path; robots.txt is case-sensitive.
4. Check that CSS, JS, and image folders are not blocked, so Google can render pages.
5. Confirm each Sitemap line uses a full https URL that returns a 200.
6. Watch for Disallow: rules that are broader than intended (a trailing slash matters).
7. Remember that robots.txt does not deindex; pair blocks with noindex or removal tools where needed.
8. Re-test after any CMS or plugin update, which can overwrite the file.

Final test: fetch [https://example.com/robots.txt] in a browser to confirm it loads, then check Search Console's robots.txt report after launch to confirm the version Google fetched.

Directive Review Sheet (fill-in)

When to use: One row per rule: path, user agent, intended access, reason, test URL, test result and the rollback line.

Robots.txt directive review

File: [https://www.example.com/robots.txt] · Reviewed by: [name] · Date: [date]

Review rules

• Every rule needs a reason. If nobody can say why a rule exists, test what it blocks before deciding to keep it.
• Test real URLs, not patterns. Google applies the most specific (longest) matching rule, and if Allow and Disallow tie, the less restrictive wins.
• Specific groups don't inherit. A crawler follows only the most specific user-agent group that names it. Rules under * don't apply to a bot with its own group.
• Keep the last working file. Save a dated copy before every change so rollback is a paste, not a memory exercise.

Google facts worth knowing (checked October 2026)

• Google reads up to 500 KiB of the file and usually caches it for up to 24 hours.
• If robots.txt returns 4xx, Google treats it as if there were no restrictions. If it returns 5xx, Google stops crawling the site for 12 hours, then may use the last good copy.
• Google ignores crawl-delay. It supports user-agent, allow, disallow and sitemap.
• Disallow stops crawling, not indexing. A blocked URL can still appear in results without a snippet.

AI Crawler Decisions

When to use: Decide separately on AI search, AI training and user-requested fetches, using each vendor's documented user-agent tokens.

AI crawler decisions

Tokens and purposes as documented by each vendor, checked 20 October 2026. Vendors add bots often, so re-check before each review.

How to decide

• Search and training are different choices. Blocking a training bot like GPTBot or ClaudeBot doesn't, per the vendors, block their search bots. Blocking OAI-SearchBot means your pages won't appear in ChatGPT search answers.
• Google-Extended doesn't affect Google Search. Google says it isn't a ranking signal and doesn't change inclusion in Search.
• Write the business reason. A publisher protecting paid content and a SaaS company wanting to be recommended in AI answers should make opposite calls.

Example group

• User-agent: GPTBot
• Disallow: /
• User-agent: OAI-SearchBot
• Allow: /

This blocks OpenAI's training crawler but keeps ChatGPT search access. OpenAI notes changes can take about 24 hours to take effect.

Worked Example (fictional B2B SaaS)

When to use: A filled-in review of an invented SaaS company's robots.txt, including a staging rule that escaped to production and how the rollback column saved the day.

Worked example: Ledgerline

Example only. The company, people, URLs and figures are invented to show what a usable, filled-in version looks like.

What the review found

Two rules did the opposite of what the team wanted. The /docs/ block came from a staging file copied during a redesign. It hid 400 help articles from search. The OAI-SearchBot block was added next to GPTBot by someone who assumed they were the same bot. Both were fixed in one release, with the old file saved for rollback.

Next step

Rebuild a clean file in the Robots.txt Generator, and make sure your Sitemap line points to a working file from the XML Sitemap Generator. After changing AI crawler rules, use the AI Visibility Checker to track whether your brand shows up in AI answers over time. Need a full technical review? See our SEO & GEO services.