The AI crawlers reading your website, one by one

15 crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Meta and Common Crawl: what each one does, what blocking it costs you, and which ones count in your AI readiness score.

Check which ones can read your site Start with the ones that count

Three kinds of crawler

A crawler is a program that visits your website and reads it, the way a browser does for a person. Every AI tool runs at least one, and they are not all doing the same job.

Search index
Reads your pages in advance so the tool can show your site, with a link, when someone searches.
Counts in the AI readiness score
On request
Fetches a page only when a user asks the tool about it or shares the link. Nothing is kept for training.
Counts in the AI readiness score
Training
Gathers material to train AI models. It is never used to find or quote your site when someone asks.
Not in the AI readiness score

Our Website Super Scanner checks your site’s crawler rules against all 15. Keeping a training crawler out is a fair business choice, so it never costs a point. Keeping out a search or on-request crawler means that tool cannot show or read your site when someone asks, so those do.

01 · 9 crawlers

The ones that count: search and on request

These are how AI tools find your site and read it when a person asks. Block one and that tool goes quiet about you.

Search index

ChatGPT search

Run by OpenAI. Known to websites as OAI-SearchBot.

This is what ChatGPT search uses, in the way Google Search uses Googlebot: it reads your pages in advance and builds the index, so your site can be shown with a link when someone searches inside ChatGPT. It follows a site's crawler rules.

If you block it: Your site cannot appear in ChatGPT search results. It does not affect model training either way.

How it differs: Unlike GPTBot, nothing it reads is used for training. Unlike ChatGPT on request, it reads ahead of time rather than on request.

On request

ChatGPT on request

Run by OpenAI. Known to websites as ChatGPT-User.

This is what ChatGPT uses when a person asks it about your site, pastes your link, or uses a tool that calls your site: it fetches the page there and then. One visit, one answer.

If you block it: ChatGPT cannot read your page when a user asks it to, so it answers from whatever else it can find.

How it differs: It only ever acts on a person's request. It neither builds an index like ChatGPT search nor trains models like GPTBot.

Search index

Claude search

Run by Anthropic. Known to websites as Claude-SearchBot.

This is what Claude's web search uses, as Google Search uses Googlebot: it reads pages in advance to build the index behind Claude's search answers, so your site can be cited with a link.

If you block it: Your site cannot appear in Claude's search answers.

How it differs: Reads ahead of time for search, like ChatGPT search; nothing it reads is used for training.

On request

Claude on request

Run by Anthropic. Known to websites as Claude-User.

This is what Claude uses when a person asks it about your site or shares your link: it fetches the page there and then. One visit, one answer.

If you block it: Claude cannot read your page when a user asks it to.

How it differs: Acts only on a person's request, the same role as ChatGPT on request, and Perplexity on request.

Search index

Perplexity search

Run by Perplexity. Known to websites as PerplexityBot.

This is what Perplexity uses to build its index, as Google Search uses Googlebot. Perplexity's answers always cite their sources with links, and this is how your site gets to be one. It follows a site's crawler rules.

If you block it: Your site cannot be one of the sources Perplexity cites.

How it differs: Perplexity has no separate training crawler: this one is about being found and cited. Perplexity on request, is the on-request one.

On request

Perplexity on request

Run by Perplexity. Known to websites as Perplexity-User.

This is what Perplexity uses when a person asks it about your site or shares your link: it fetches the page there and then. Because a person asked, Perplexity says it may fetch the page even where crawler rules say no.

If you block it: Rules may not stop it; it is acting for a person, much as their browser would.

How it differs: The one crawler here whose publisher says it does not always follow crawler rules, on the grounds that it acts on a user's request.

Search index

Google and AI Overviews

Run by Google. Known to websites as Googlebot.

This is what Google Search uses - the crawler every website has met. The same visits now also feed the AI Overviews at the top of Google's results and Google's AI Mode.

If you block it: Your site drops out of Google altogether, AI Overviews included. There is no way to stay in Google Search but out of AI Overviews by blocking a crawler.

How it differs: Google Gemini (training) is a separate switch for training. Blocking that one leaves Search and AI Overviews untouched; blocking Googlebot removes everything.

Search index

Bing and Copilot

Run by Microsoft. Known to websites as Bingbot.

This is what Bing uses - Microsoft's search engine, the one built into Windows and Edge. The same visits feed Microsoft Copilot's answers and the search engines that buy Bing's index, DuckDuckGo among them.

If you block it: Your site drops out of Bing, Copilot and DuckDuckGo together.

How it differs: Like Googlebot, one crawler serves both the classic search results and the AI answers; there is no separate training switch to block.

Search index

Amazon Alexa

Run by Amazon. Known to websites as Amazonbot.

This is what Amazon uses so Alexa can answer questions with information from the web, as Google Search uses Googlebot. It follows a site's crawler rules.

If you block it: Alexa cannot draw on your site when answering.

How it differs: An answers crawler for a voice assistant rather than a search engine; Amazon also says it may use what it reads to improve its AI.

02 · 6 crawlers

Your choice: the training crawlers

These gather material to train AI models. Blocking them keeps your words out of future models and changes nothing about being found today, so the AI readiness score leaves them out.

Training

ChatGPT (training)

Run by OpenAI. Known to websites as GPTBot.

This is the one OpenAI uses to collect pages for training ChatGPT's models. It has no part in ChatGPT searching or answering; that is the next two. It follows a site's crawler rules.

If you block it: Nothing changes in ChatGPT search or when a user asks ChatGPT about your site. Your words are simply not used to train future models.

How it differs: OpenAI runs three crawlers. This is the training one; ChatGPT search and ChatGPT on request, are the two that put your site in front of people.

Training

Claude (training)

Run by Anthropic. Known to websites as ClaudeBot.

This is the one Anthropic uses to collect pages for training Claude. It has no part in Claude searching or answering; that is the next two. It follows a site's crawler rules.

If you block it: Claude search and Claude on request, carry on as before. Your words are not used to train future Claude models.

How it differs: Anthropic's training crawler, as GPTBot is OpenAI's. Claude search and Claude on request, are the two that matter for being found.

Training

Google Gemini (training)

Run by Google. Known to websites as Google-Extended.

This is the switch Google reads to know whether the pages Googlebot already collected may be used to train Gemini and to ground its answers. It is not a separate crawler and makes no visits of its own.

If you block it: Google Search, AI Overviews and AI Mode are unaffected. Your pages are not used to train Gemini.

How it differs: Shares Googlebot's visits rather than making its own. It controls training use only, not whether you are found.

Training

Apple AI (training)

Run by Apple. Known to websites as Applebot-Extended.

This is the switch Apple reads to know whether the pages its Applebot already collected for Siri and Spotlight may also train Apple's AI models. It is not a separate crawler.

If you block it: Siri and Spotlight suggestions keep working. Your pages are not used to train Apple Intelligence.

How it differs: The same arrangement as Google Gemini (training): a training switch riding on the main crawler's visits.

Training

Meta AI (training)

Run by Meta. Known to websites as Meta-ExternalAgent.

This is what Meta uses to collect pages for training its AI models and improving its products, including Meta AI inside Facebook, Instagram and WhatsApp. It follows a site's crawler rules.

If you block it: Your pages are not used to train Meta's models. It has no effect on how Facebook or Instagram show links to your site.

How it differs: Purely a training crawler, like GPTBot and ClaudeBot. Meta has no search product that would show your site with a link.

Training

Common Crawl (training)

Run by Common Crawl, a non-profit. Known to websites as CCBot.

This is what Common Crawl, a non-profit, uses to build a free public copy of much of the web. Researchers and many AI companies train their models from that copy. It follows a site's crawler rules.

If you block it: Your pages stay out of that public copy, and so out of every model trained from it. Nobody uses Common Crawl to find or quote a site live.

How it differs: Not run by any AI company. Blocking it keeps your words out of many models at once, but changes nothing about being found.

Which of them can read your website?

Type your website address and the Website Super Scanner shows which crawlers are allowed in, which are kept out, and everything else that decides whether AI tools can find and describe your business.