Your robots.txt is the first thing an AI crawler checks before reading you. Many templates and CDNs block GPTBot, ClaudeBot or PerplexityBot by default without anyone remembering: the result is that when an assistant searches live, your content is off-limits to it. Here is what you need to decide your policy on purpose.
It is a text file at your domain root (/robots.txt) declaring which user-agents may access which paths. It is not a technical wall — it is a good-faith convention. The documented AI crawlers from OpenAI, Anthropic, Perplexity and Google claim to honor it, so blocking there does stop those specific bots.
The main ones: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Gemini training), CCBot (Common Crawl, a dataset used by many models) and Applebot-Extended (Apple). Each has its own name and its own User-agent line.
Not all bots do the same thing. GPTBot or ClaudeBot gather data to train and index; ChatGPT-User and Perplexity-User visit your page on behalf of a specific user. Blocking training does not stop an assistant citing you when someone asks about you — and blocking live browsing does keep you out of those answers.
Allowing maximizes your presence: if you want AIs to read, understand and cite you, let them in. Blocking makes sense if you do not want your content feeding training runs. That is a legitimate business call; what you do not want is to block by accident and lose citations you never chose to give up.
Add a User-agent block per bot with its Allow or Disallow. Being explicit avoids surprises: a clear rule for GPTBot beats relying on the User-agent: * wildcard. Declare what you actually want, bot by bot.
Add the line Sitemap: https://yourdomain.com/sitemap.xml at the end of robots.txt. It helps crawlers and agents discover your pages instead of guessing, and it costs nothing.
Many generators, "privacy" plugins and CDN settings add a Disallow for AI bots without telling you. Check your real robots.txt — the one your domain serves right now — and confirm it reflects your decision, not a template author's.
After editing, check the file served at https://yourdomain.com/robots.txt and validate which bots end up allowed or blocked. Our free checker shows you, bot by bot, whether your policy is what you think it is.
Do not guess whether you block yourself by accident: check your robots.txt for free and see, AI bot by AI bot, who can read you today — and get your full AI Visibility Score while you are at it.
Not entirely. Blocking GPTBot stops data collection for training and indexing, but live browsing (ChatGPT-User) uses a different user-agent. Even so, blocking the crawler reduces what the assistant knows about you and can cut your citations. Decide it on purpose, not by neglect.
No. It is a good-faith convention: bots that respect it obey it, but it does not technically stop access by anyone who ignores it. It signals your intent, not a lock.
With no robots.txt, bots enter by default: you do not block them, but you also declare no policy and no sitemap. An explicit file is better than nothing.
No. AI crawlers (GPTBot, ClaudeBot…) are distinct from Googlebot, which governs traditional SEO. Blocking the AI ones does not change your Google ranking; it only changes your visibility to AI assistants.