Free tool

A robots.txt generator for AI

Choose who gets into your site — the assistants that cite you, the bots that train models, the search engines — and take away a robots.txt ready to publish. From scratch or from yours, without leaving this page: nothing you type is sent anywhere.

Preset

Group by group

The same groups as the bot table in the diagnostic. Open a group to choose bot by bot. Presets do not touch advertising, social networks or SEO tools: they stay as you leave them. Starting from your file, presets never reopen a bot in “Other search engines and assistants” that your file blocks.

  • Answer right now

    They fetch your page when someone asks an assistant a question. Shut them out and you drop out of that assistant's answers.

    All let in
    Show the 6 bots

  • Feed a search index

    They build the index assistants draw their answers from. Googlebot, Bingbot and Applebot are here: blocking them also takes you out of classic search.

    All let in
    Show the 8 bots

  • Other search engines and assistants

    They crawl for other search engines and assistants that the score does not measure. DuckDuckBot is DuckDuckGo's search crawler.

    All let in
    Show the 9 bots

  • Collect for training

    They collect text to train models. Blocking them takes you out of neither answers nor the score.

    All blocked
    Show the 7 bots

  • Opt-out signals

    These are not crawlers. Blocking Applebot-Extended only keeps you out of Apple's training. Blocking Google-Extended also keeps you out of Gemini answers grounded in Search: that is why it stays open in “Answers yes, training no”.

    1 of 2 blocked
    Show the 2 bots

  • Advertising

    Their operators document them for advertising. No preset touches them: they stay as you leave them.

    All let in
    Show the 2 bots

  • Social networks

    They read your page when someone shares a link. Block them and the link loses its title and image.

    All let in
    Show the 3 bots

  • SEO tools

    Agencies and competitors use these to audit sites. Blocking them is your call and does not affect the score.

    All let in
    Show the 5 bots

Your robots.txt

Download robots.txt

robots.txt

# robots.txt generated at parsigo.com — 2026-11-01
# Answers yes, training no

User-agent: *
Allow: /

# Collect for training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: AI2Bot
User-agent: MistralAI-Training
Disallow: /

# Opt-out signals
User-agent: Applebot-Extended
Disallow: /

Put it at the root of your site, so it answers at /robots.txt.

On WordPress with Yoast or Rank Math, paste it into the plugin's file editor. On Shopify, edit the theme's robots.txt.liquid. On Wix, it is in the SEO tools. Squarespace does not let you edit the file: it only has an option to block AI crawlers in its settings.

Before you publish

You reach 9 of 9 assistants.

Every assistant we measure can still read you.

Once it is live, check it with the diagnostic

What each preset decides

All three come from our registry of 42 crawlers, and every name has its operator's official documentation. A bot that joins the registry lands in the right preset on its own, by what it does: answering, feeding an index or training. Presets only decide the AI bots: social networks and SEO tools are not AI, and stay as you set them with the switches.

  • Answers yes, training no

    Crawlers that answer questions or feed an assistant's search get in; the ones that collect text for training stay out, and so does Applebot-Extended. Google-Extended is not blocked: blocking it also keeps you out of Gemini's answers. It is the one we recommend: blocking training is a legitimate editorial choice and does not take you out of answers.

  • Out of AI

    Blocks every AI bot, including the ones that cite you. No preset blocks the search engines you already rely on (Googlebot, Bingbot, Applebot and DuckDuckBot): from scratch they stay open, and from your file they stay as they are.

  • Everything open

    No AI bot is blocked. From scratch, with social networks and SEO open, the file keeps the User-agent: * group and nothing else: the same as having no rules for AI at all.

Why “Out of AI” still lets Googlebot in

Google's AI Overviews, Microsoft Copilot and Apple's Siri use the same crawler as classic search: Googlebot, Bingbot and Applebot. There is no way to opt out of just the AI answers without leaving search. That is why no preset blocks them: from scratch they stay open, from your file they stay as they are, and “Out of AI” says so under the file. If you really want them out, open the group and switch them off one by one.

Google-Extended and Applebot-Extended

These are not crawlers: they are names Google and Apple read in robots.txt to decide what to do with what they have already crawled. Blocking Applebot-Extended keeps your content out of Apple's model training, and your pages stay in search. Blocking Google-Extended also keeps it out of grounding, the Search content the Gemini apps use to answer; that is why the recommended preset does not block it. From your file, no preset touches it: if you blocked it, it stays blocked until you change it by hand.

The Content-Signal

It is a line Cloudflare proposed to state, in the User-agent: * group, what your content may be used for: search=yes (appearing in search results), ai-input=yes (being a source for an answer) and ai-train=no (not training models). It is a convention, not a rule: not every bot reads it, and none is bound to honour it. That is why it starts switched off.

Starting from your robots.txt

Paste your file and the switches show what it does today, bot by bot. We only touch the bots whose decision changes. A bot your file already treats the way you chose stays exactly where it is, with its rules and its Crawl-delay; everyone else's rules — a Disallow: /wp-admin/, the Sitemap, bots outside the registry — stay too. A bot you block moves out of the group it is in and into a new group at the end. A bot you let in leaves the group that blocked it and follows your User-agent: * rules; if it needs a group of its own, it takes the * path blocks with it. We list under the file who is now let in, and warn you if your * closes the whole site.

From your file, presets do not touch the classic search engines or Google-Extended: they stay as they are. If the file has the block managed by Cloudflare (# BEGIN Cloudflare Managed content), that block comes out of the new file: Cloudflare adds it by itself when it serves the file, and the bots it decides are changed in its dashboard. Names the operator has retired, such as anthropic-ai, stay too, and we tell you which name is current.

The generator is tested with the same robots.txt reader the diagnostic uses: every registry bot ends up with the decision you chose at /, and the user-agents your file names that you do not change keep the result they had.

What this generator does not do

  • It does not fetch your robots.txt

    You paste the file yourself: this page makes no request to your site, nor to our servers.

  • It does not write server rules

    Nginx, Apache and Cloudflare can block a bot before it reads robots.txt, and that is out of scope. If your firewall stops AI bots, the diagnostic shows it in the live test.

  • It does not make up names

    We only use names the operator documents. A made-up name blocks nothing, and gives you an answer about a bot that does not exist.

  • It decides at the home page

    We read and write each decision at /. A file with path exceptions (an Allow: /blog/ in a blocked group) can show a group as blocked while some paths stay open, and the other way round.