robots.txt generator
Choose a default, the paths to keep crawlers out of, and how AI crawlers should be treated. The generator writes the file, then reads it back the way crawlers do, so you can see what it allows.
Fine-tune each crawler
User-agent: * Disallow:
Runs in your browser. Nothing you type is sent to Notable.
How it works
- Put the file at the root of your site, as yoursite.com/robots.txt. Crawlers look only there, and each subdomain needs its own.
- A crawler that has its own section ignores the * section, so the generator repeats your blocked paths and crawl delay in every section that allows crawlers. The summary reads the finished file with the same rules as our AI crawler checker.
- Search crawlers such as Googlebot and Bingbot follow the default unless you change them one by one, because blocking them takes you out of search as well as AI answers. Google ignores Crawl-delay; Bing follows it.
The background
robots.txt is a request, not a lock
Well-behaved crawlers follow it, but nothing enforces it, and some agents that fetch a page because a person asked about it say they may not follow it at all. To keep a page private, put it behind a login. To stop a crawler for certain, block it at your server or CDN.
Blocking training does not take you out of AI answers
Blocking training crawlers such as GPTBot, ClaudeBot and Google-Extended keeps your pages out of future model training. AI answers built on search, such as Google's AI Overviews or ChatGPT search, rely on Googlebot and OAI-SearchBot instead, and the block-training preset keeps those allowed.
Blocked is not the same as hidden from search
A page blocked in robots.txt can still appear in search results, without a description, if other sites link to it. To keep a page out of results, let it be crawled and give it a noindex tag instead.
Want these numbers for your brand, every day?
Notable asks AI assistants your customers' questions daily and shows how often you are recommended, with the margin of error beside every number.