robots.txt Generator

What robots.txt can and cannot do

It is a request, not a lock. Well-behaved crawlers read /robots.txt and respect it; anything malicious ignores it entirely. So it is the right tool for keeping a checkout page out of search results, and the wrong tool for keeping anything private. Private means behind a password.

There is a second catch worth knowing. Disallow stops a page being crawled, not indexed. A blocked page that other sites link to can still turn up in results, listed without a description. To genuinely keep something out, let it be crawled and give it a noindex meta tag instead — a crawler has to read the page to see the instruction.

Blocking AI crawlers

There is no single switch for this. Each company runs its own crawler with its own name, and a rule only applies to the user-agent it names, so the list has to be written out — which is what the second option does. It covers OpenAI, Anthropic, Perplexity, Google's and Apple's separate AI agents, ByteDance and Common Crawl.

Note that Google-Extended controls AI training only. It does not affect ordinary Google Search crawling, so blocking it costs you nothing in rankings.

Where it goes

At the root of the domain, exactly: https://example.com/robots.txt. Not in a subfolder. Each subdomain needs its own — a file on example.com says nothing about shop.example.com.