devprix.io

robots.txt Generator & Tester Online

Build a robots.txt file from user-agent groups, Allow and Disallow rules, sitemaps and an optional crawl delay, or start from a preset: allow everything, block everything, block common private paths, or block AI training crawlers. Then test any URL against it for Googlebot, Bingbot, GPTBot, ClaudeBot or any other crawler.

The tester tells you whether the URL is allowed and which rule on which line decided it, using RFC 9309 matching as Google applies it. It also checks the file for mistakes such as paths without a leading slash, relative sitemap URLs and Noindex lines. Everything runs in your browser and nothing is fetched or uploaded.

Groups

Paths start with /. * matches anything and $ marks the end. Crawl-delay is in seconds: Bing honours it, Google ignores it.

Block AI crawlers

Collect content for AI models
AI search: blocking keeps you out of its answers

Sitemaps

robots.txtUpload to the root of your site
User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Checks

No problems found.

robots.txt ready1 groupNo problems74 BProcessed locally
Send output toText Diff Checker

How to use it

  1. Build the file in Generate: add user-agent groups and rules, tick AI crawlers to block and add your sitemap, or start from a preset.
  2. Copy or download robots.txt, or press Test this file (⌘↵) to check URLs against it; you can also paste an existing file in Test a URL.
  3. Enter a URL or path and pick a crawler to see Allowed or Disallowed, the deciding rule and line, and any problems in the file.

Related tools

Meta Tag GeneratorWeb & APIOG Preview / DebuggerWeb & APIURL ParserWeb & APIRegex TesterFormat

Next: set noindex and other robots meta tags with the Meta Tag Generator, or compare your new robots.txt with the live one in the Text Diff Checker.

What this tool does

A robots.txt file sits at the root of a host (https://example.com/robots.txt) and asks crawlers to stay out of some URLs. It is made of groups: one or more User-agent lines naming crawlers by product token, followed by Allow and Disallow rules. A crawler obeys only the group that names it most specifically, falling back to the * group, and groups that name the same crawler are combined. Within that group the rule with the longest path wins, an Allow wins a tie, * matches any characters and $ marks the end of the URL; paths are case-sensitive and the query string counts. The tester follows RFC 9309 and the details of Google’s open-source parser: User-agent: Googlebot/2.1 is read as Googlebot, common misspellings such as Disalow are accepted, Allow: /dir/index.html also allows /dir/, and /robots.txt itself is always allowed. Some crawlers fall back to another group: Googlebot-Image to Googlebot, Bingbot to msnbot, and Applebot to Googlebot. URLs are percent-encoded as a browser sends them before matching, and only the first 500 KiB of the file is read, as Google does. Every AI crawler token in the generator is taken from its operator’s own documentation, which is linked next to it.

Example input and output

A crawler with its own group ignores the * group
Input
User-agent: *
Disallow: /admin/

User-agent: Googlebot
Disallow: /drafts/

URL: https://example.com/admin/settings
Output
Allowed for Googlebot
No rule in the group matches /admin/settings, so it may be crawled.
Group: The group for Googlebot (lines 4–5). Rules in the * group don’t apply to it.

Disallowed for bingbot
Line 2, Disallow: /admin/, is the most specific matching rule (7 characters).
Group: No group names bingbot or msnbot, so it obeys the * group (lines 1–2).

This is the most common robots.txt surprise. To block /admin/ for Googlebot as well, repeat Disallow: /admin/ in its group.

The longest rule wins, even against a wildcard
Input
User-agent: *
Disallow: /shop/
Allow: /shop/sale/
Disallow: /*.pdf$

URL: https://example.com/shop/sale/catalogue.pdf
Output
Allowed for Googlebot
Line 3, Allow: /shop/sale/, is the most specific matching rule (11 characters).

All matching rules, most specific first
Line 3  Allow: /shop/sale/   11
Line 4  Disallow: /*.pdf$     7
Line 2  Disallow: /shop/      6

Rule order in the file does not matter. To block PDFs everywhere, make the PDF rule longer than the Allow, for example Disallow: /shop/sale/*.pdf$.

Preset: block AI training crawlers
Input
Block AI training crawlers, with https://example.com/sitemap.xml
Output
User-agent: *
Disallow:

# AI crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Search crawlers are untouched. AI search bots such as OAI-SearchBot and PerplexityBot are listed separately, because blocking them keeps your pages out of those products’ answers.

Common errors and fixes

“private/” doesn’t start with /, so it never matches
Rules are compared with the URL path from its first character, and every path starts with /. Write Disallow: /private/. A rule may also start with *, such as Disallow: *.gif$.
Rules take a path, not a full URL, so “https://example.com/private/” never matches
A robots.txt applies only to its own host, so rules hold just the path: Disallow: /private/. Only Sitemap lines take full URLs.
No rule comes between “User-agent: Googlebot” and “User-agent: *”, so they form one group
A blank line does not end a group: consecutive User-agent lines share the rules that follow, so Googlebot gets Disallow: / too. If Googlebot should be allowed everywhere, give it its own empty Disallow: line. The generator does this for you.
Noindex isn’t a robots.txt rule: Google stopped reading it on 1 September 2019
Use a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex response header, and don’t disallow the page: a crawler that can’t fetch it never sees the noindex.
Sitemap must be a full URL with https:// and the host
Write Sitemap: https://example.com/sitemap.xml, not Sitemap: /sitemap.xml. A sitemap can be on another host, and you can list several.
Google ignores Crawl-delay; Bing honours it
Crawl-delay sets seconds between requests for crawlers that support it, such as Bingbot. Google ignores it and sets Googlebot’s rate itself; to slow it down urgently for a short time, Google suggests answering with 500, 503 or 429 status codes.
The page is blocked but still shows up in Google
Disallow stops crawling, not indexing: Google can list a blocked URL it finds through links, without a snippet. Allow crawling and add noindex instead, or remove the page.

Common use cases

  • Writing a robots.txt for a new site, with sitemap links
  • Checking why Googlebot can or can’t crawl a URL before deploying a change
  • Opting out of AI training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot
  • Keeping crawlers out of admin, cart, checkout and internal search pages
  • Reviewing an inherited robots.txt for mistakes that block more, or less, than intended

Privacy and security notes

  • Your robots.txt, URLs and settings are processed by JavaScript in this tab. Nothing is fetched from your site and nothing is sent to DevPrix.
  • Share copies a link with the robots.txt after the # in the URL. Browsers don’t send that part to servers, but anyone with the link can read it.
  • Remember that robots.txt is public. Listing a secret path in it tells everyone where to look; protect such paths with authentication instead.

Frequently asked questions

Which group does a crawler follow?

Only one: the group whose User-agent line names its product token most specifically, compared without regard to case. If no group names it, it follows the * group; if there is none, nothing is restricted. Groups for the same crawler are combined, and the order of groups in the file doesn’t matter. Some crawlers document a fallback: Googlebot-Image follows a Googlebot group, Bingbot an msnbot group and Applebot a Googlebot group when no group names them.

How are conflicting Allow and Disallow rules resolved?

The rule whose path is longest wins, counting wildcards, so Allow: /shop/sale/ (11 characters) beats Disallow: /shop/ (6). If an Allow and a Disallow are the same length, the Allow wins. The tester shows every matching rule with its length.

Does blocking GPTBot or Google-Extended remove my site from search?

No. GPTBot only collects content for OpenAI’s model training, and Google-Extended only controls Gemini training and grounding; neither affects Google Search or ChatGPT search. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot does keep your pages out of those AI search answers. Fetchers that act for a user, such as ChatGPT-User and Perplexity-User, may not follow robots.txt at all.

Is robots.txt enough to keep content private?

No. It is a request that well-behaved crawlers honour, not access control, and RFC 9309 says so. The file is public, so listing a path in it advertises it. Use authentication for private content and noindex for pages that shouldn’t appear in search.

Where does robots.txt go, and does it cover subdomains?

At the root of each host, protocol and port: https://example.com/robots.txt covers only https://example.com. https://shop.example.com and http://example.com each need their own file. Rules in /folder/robots.txt are never read.