Skip to content
Bold PilotBold Pilot
Get started

robots.txt Generator

A correct robots.txt in a minute, AI crawlers included. Free, no sign-up.

Your site

For the Sitemap line — or import the file the site serves today and start from that.

Start from

Crawl it all, sitemap declared. The right file for most sites.

Keep crawlers out of

Paths no crawler should fetch. A trailing slash covers the whole folder.

But allow

Exceptions inside a blocked folder. The longer rule wins, so these beat the block above.

AI answer engines

These fetch a page to answer a question and cite it — ChatGPT search, Claude, Perplexity, Siri. Allowing them is how a site gets cited.

OAI-SearchBot · ChatGPT-User · Claude-SearchBot · Claude-User · PerplexityBot · Perplexity-User · Applebot · meta-externalfetcher · Amazonbot · DuckAssistBot · MistralAI-User · YouBot

AI training crawlers

These collect pages to train the next model and cite nothing. Blocking them does not affect answers or citations.

GPTBot · ClaudeBot · Google-Extended · Applebot-Extended · meta-externalagent · CCBot · Bytespider · cohere-ai

Sitemaps

Leave empty for /sitemap.xml on your site. Absolute URLs or paths.

Crawl-delay

Seconds between requests. Bing and Yandex honour it; Google ignores it. Usually leave empty.

robots.txt

Download
User-agent: *
Disallow:

# AI answer engines — fetch pages to answer questions and cite them
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: YouBot
Disallow:

Upload it to the root of your site so it answers at /robots.txt, then confirm with the robots.txt & sitemap checker.

No Sitemap: line. robots.txt is the first place Google looks for a sitemap. Without the line, discovery depends on a manual Search Console submission.

Crawlers can reach the site. 2 groups, 0 sitemaps declared.

Test a URL against this file

Crawlable is the easy half.

Bold Pilot checks every keyword against its live results page, writes the article, and publishes it — week after week.

Try Bold Pilot

Technical SEO guide

How to write a robots.txt that does what you meant — including what to tell the AI crawlers

robots.txt is a short text file at the root of a site that tells crawlers which paths they may request. It is the oldest convention on the web and still one of the easiest to get wrong, because it looks like configuration and behaves like law: a single wrong line removes a site from search, and nothing in the file tells you.

What it does, and what it does not

It controls crawling — which URLs a bot is allowed to fetch. It does not control indexing. A page blocked in robots.txt can still appear in results if other pages link to it, shown with no description because Google was never allowed to read it. To keep a page out of the index, let it be crawled and put noindex on the page. The two tools answer different questions.

The syntax, all of it

User-agent: *
Disallow: /admin/
Allow: /admin/public/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
  • User-agent opens a group and names who it is for. * is everyone not named elsewhere. Several User-agent lines in a row share one group.
  • Disallow and Allow are path prefixes. /admin/ covers everything under it; /admin also covers /administrator. An empty Disallow: means “nothing is blocked”.
  • Wildcards: * matches any run of characters and $ anchors the end. Disallow: /*.pdf$ blocks PDFs and nothing else.
  • Sitemap takes a full URL and lives outside any group. Several lines are fine.
  • Crawl-delay is honoured by Bing and Yandex and ignored by Google.

How rules are resolved

Not top to bottom, and not “Disallow wins”. Google picks the most specific matching rule — the longest one — and on a tie, Allow wins. That is why Allow: /admin/public/ beats Disallow: /admin/ whichever comes first. And a group for a named crawler replaces the * group for that crawler rather than adding to it: if GPTBot has its own group, it never reads the rules under *. The generator restates your exclusions inside every named group for exactly this reason.

The AI crawlers, and why there are two lists

Every AI company runs more than one bot, and they do different jobs. Some fetch a page because a person just asked a question — ChatGPT search, Claude, Perplexity, Siri — and those answers cite their sources. Others sweep the web to build a training set, and cite nothing. The generator keeps them apart:

  • Answer engines: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, meta-externalfetcher and the rest. Allowing them is how a site gets cited in AI answers.
  • Training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider and others. Blocking them changes what future models learn from; it does not affect today’s answers or citations.

Two things people get wrong here. Google-Extended is not Googlebot: blocking it opts out of Gemini training and has no effect on search or on AI Overviews, which use the normal index. And a bot has to be named to be governed — a file that says nothing about GPTBot has allowed it, because the * group applies. “Don’t mention” in the generator is that default, made explicit.

The five mistakes the linter catches

  1. Disallow: / on a live site. Usually a staging file that shipped. The whole index goes within weeks.
  2. Blocking CSS or JavaScript. Google renders pages before ranking them; a page it cannot style or run looks broken to it.
  3. Noindex: in robots.txt. Google dropped support in 2019. It does nothing now.
  4. No Sitemap line. Discovery then depends on a manual Search Console submission, which is one migration away from being forgotten.
  5. Rules above the first User-agent. They belong to no group, so they apply to nobody.

Where the file goes, per platform

  • Static hosting, Vercel, Netlify: put robots.txt in the public folder so it answers at /robots.txt. Next.js can also generate it from app/robots.ts.
  • WordPress: upload to the site root over SFTP, or set it in Yoast or Rank Math under Tools → File editor. WordPress serves a virtual one until a real file exists.
  • Shopify: edit the robots.txt.liquid theme template; the file itself cannot be uploaded.
  • Webflow, Squarespace, Wix: a settings field under SEO; paste the text.

Then confirm the live file with the robots.txt & sitemap checker, which also cross-checks it against your sitemap, and see what AI engines make of it with the AI visibility checker.

Frequently asked

Do I need a robots.txt at all?

A missing file means “crawl everything”, which is fine for many sites. The one thing you lose is the Sitemap line, which is the cheapest discovery signal there is. An open file with a sitemap is the sensible minimum.

Is the file case-sensitive?

Paths are: /Admin/ and /admin/ are different rules. Directive names are not.

Can I block a single page?

Disallow: /page$ blocks exactly that path. Without the $ the rule also covers /page-two and /page/child. If the goal is keeping it out of search results, noindex on the page is the right tool.

How long until a change takes effect?

Google caches robots.txt for up to a day. Search Console’s robots.txt report shows the version it holds and lets you ask for a recrawl.