Utilore

robots.txt Generator and Tester

Two things in one page: assemble the file from rules, and test the file you already have — the answer says not only «allowed» or «blocked» but which line made the call.

Template

Google ignores Crawl-delay; Yandex and Bing still read it.

How to use

  1. Pick a template or list the paths to close, one per line.
  2. Add exceptions under Allow and the address of your sitemap.
  3. Copy the result or download it and put it at the root of the site.
  4. Switch to Test, paste your file and check a specific address for a specific crawler.

Good to know

The longest rule wins, not the first one

Both Google and Yandex choose the rule by the length of the matched path, not by the order in the file. That is why the usual pair «Disallow: /wp-admin/» plus «Allow: /wp-admin/admin-ajax.php» works: the second path is longer, so it wins for that one file while the rest of the folder stays closed. A tester that reads the file top to bottom would give the opposite answer.

A rule above the first User-agent belongs to nobody

A Disallow written before any User-agent line is ignored by every crawler — the file is read in groups, and a rule outside a group has no owner. It looks perfectly fine and closes nothing, which is why this is the single most expensive mistake in robots.txt. The tester reports it separately, with the line number.

Frequently asked questions

Does robots.txt remove a page from search?

No — it only stops the crawl. A closed address can still appear in results without a description if other sites link to it. To keep a page out of the index, leave it open to the crawler and put a noindex meta tag on it: a robot has to fetch the page to see that tag.

Where should the file go?

Only at the root of the domain: https://example.com/robots.txt. A file in a subfolder is not read at all, and a subdomain needs its own — for shop.example.com, the file at example.com does not apply.

How do I block AI crawlers?

Add a group per bot: GPTBot, ClaudeBot, PerplexityBot, CCBot are in the list of crawlers here. They honour robots.txt, but the ban is per name — a wildcard group does not cover them if they also have their own group in the file.

Is Crawl-delay worth setting?

Google ignores it entirely; Yandex and Bing still read it. Set it only if the server is genuinely struggling under crawl load — otherwise it just slows down how quickly your new pages get discovered.

Related tools