ClaudeBot, GPTBot, ChatGPT-User: which AI crawler does what

Open your server log and you find names like ClaudeBot, GPTBot, ChatGPT-User, OAI-SearchBot and meta-externalagent. They are AI crawlers, and they are not all doing the same thing. Most AI companies send up to three of them, each for a different job.

One collects pages to train future models. One keeps the company’s search index up to date. And one reads a single page right now, because a person just asked ChatGPT or Claude a question that page answers. Which one you are looking at decides what blocking it does, whether it listens to your robots.txt, and whether your analytics will ever see it. Below is what each company documents, and what actually came to our own site in five days.

Three jobs, three crawlers

Collecting training data. GPTBot, ClaudeBot, meta-externalagent and their kind copy public pages into the data future models learn from. They crawl on their company’s schedule, not because anyone asked anything. Blocking them keeps your pages out of future training sets. It does not remove anything already collected, and it does not stop the same company from finding and quoting your pages.

Reading for its search. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the search index the AI service answers from, much like Googlebot does for Google. A page that is not in that index cannot turn up when someone asks. Blocking a search crawler is how you disappear from that service’s answers, which is usually not what a site owner wants.

Reading for an answer. ChatGPT-User, Claude-User and Perplexity-User request one page at the moment a person needs it: someone pasted your link, or the service decided your page answers their question. Each request is one person’s question. This is the closest thing to a visit an AI service makes, and it is the one your analytics never shows (more on that below).

Each job is its own name in robots.txt. OpenAI and Perplexity say plainly that every setting is independent, and Anthropic describes what blocking each of its three crawlers does on its own. Blocking GPTBot keeps ChatGPT’s search and answers exactly as they were.

Every AI crawler, by company

What each company documents about its crawlers, in its own words where it matters. Each company’s page is linked; that is where to look when a name changes.

OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User

GPTBot collects training data: disallowing it “indicates a site’s content should not be used in training generative AI foundation models.” OAI-SearchBot builds ChatGPT’s search, and “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” ChatGPT-User reads a page when someone asks ChatGPT or a custom GPT something, and “because these actions are initiated by a user, robots.txt rules may not apply.” OpenAI adds that if you allow both GPTBot and OAI-SearchBot, it may use one crawl for both purposes. Changes to robots.txt take about 24 hours to reach its search. OpenAI’s crawler page

Anthropic: ClaudeBot, Claude-SearchBot and Claude-User

ClaudeBot collects training data. Blocking it “signals that the site’s future materials should be excluded from our AI model training datasets.” It also honours Crawl-delay, if you want it to come less often. Claude-SearchBot builds Claude’s search, and Claude-User reads a page when someone asks Claude a question. Anthropic says all three follow robots.txt, and that blocking Claude-User “prevents our system from retrieving your content in response to a user query.” That makes it the only company of the big ones whose answer crawler listens to robots.txt, and our own test agrees: Claude asked for robots.txt one second before it read the page. ClaudeBot is also quick: on our test site it arrived within minutes of the TLS certificate being issued, before anyone had the link. Anthropic’s crawler page

Google: Googlebot, Google-Extended and the Gemini crawlers

Google has no separate AI training crawler. Google-Extended is “a standalone product token”: a name you use in robots.txt to say whether pages Googlebot already crawled may train Gemini and ground its answers. No request ever carries it, and it “does not impact a site’s inclusion in Google Search.” Google’s page does not mention AI Overviews; they come from the ordinary search index, which Googlebot builds. For reading on a person’s behalf, Google lists Google-Agent (agents that “browse and act on user request”) and Google-GeminiNotebook (pages someone added as sources), among other user-triggered fetchers that “generally ignore robots.txt rules.” The Gemini app itself is not on that list: in our test it read the page as a bare Google. Google’s common crawlers, user-triggered fetchers

Perplexity: PerplexityBot and Perplexity-User

PerplexityBot builds Perplexity’s search and “is not used to crawl content for AI foundation models”; Perplexity documents no training crawler. Perplexity-User reads a page for a person’s question and “generally ignores robots.txt rules.” In our test, though, Perplexity answered a person’s question with PerplexityBot, so you can see the search crawler’s name on a request that was really for an answer. Perplexity’s crawler page

Meta: meta-externalagent, meta-webindexer and meta-externalfetcher

meta-externalagent crawls to “train foundation AI models or to improve products by indexing content directly”. meta-webindexer improves search results in Meta AI, and allowing it “helps us cite and link to your content.” meta-externalfetcher reads individual links at a user’s request, and “may bypass robots.txt rules.” As with Perplexity, our test saw Meta AI answer a question with meta-webindexer. The link previews in Facebook, Instagram and WhatsApp come from a fourth name, facebookexternalhit. Meta’s crawler page

Mistral, Amazon, Apple and Microsoft

Mistral has the clearest names: MistralAI-Training, MistralAI-Index and MistralAI-User. It doesn’t say whether MistralAI-User follows robots.txt; in our test it read robots.txt once and then every page twice in the same second. Mistral’s page

Amazon sends Amazonbot, which “may be used to train Amazon AI models”, Amzn-SearchBot for search, and Amzn-User for questions such as Alexa’s, which “may not follow all robots.txt directives.” Amazonbot does more than training, so blocking it costs more than a training opt-out. Amazon’s page

Apple crawls with Applebot for Siri, Spotlight and Safari. Applebot-Extended is a robots.txt name like Google-Extended: it “does not crawl webpages”, and disallowing it keeps your pages out of Apple’s model training while they stay in search. Apple’s page

Microsoft documents no separate AI crawler. Copilot answers from Bing’s index, which bingbot builds, so blocking bingbot removes you from Bing and Copilot at once.

The ones that don’t say

Some AI services document nothing we could check. Bytespider is ByteDance’s crawler; reports disagree on whether it respects robots.txt. xAI documents no crawler for Grok, and in our test Grok’s browsing agent looked exactly like a person’s Chrome, from the person’s own connection. DeepSeek read our page in a full browser with an ordinary Firefox user agent. Nothing in a log line names either of them.

What actually came to our site in five days

From 29 October to 2 November 2026 our server reported every crawler request to privatrak.com and its subdomains: 3,785 requests. 1,661 of them, 44%, carried the name of an AI crawler.

Training came first. GPTBot requested 384 pages in five days, more than any other crawler, search engines included: Bingbot requested 297 and Googlebot 133.

Answers were the smaller part, and the most valuable. ChatGPT-User requested 62 pages and Claude-User 27, each one a person whose question our page answered. 46 of ChatGPT’s were for the home page.

At least 635 of the 1,661 AI-named requests, 38%, were not AI crawlers at all. They asked for .env files, /proc/self/environ, private keys and docker-compose.yml: files no website shows its visitors, which scanners try in the hope of finding a leaked password. Under the names of answer crawlers there were more of these (146) than page requests (97). Perplexity-User asked for a secret file 36 times out of 40, DuckAssistBot 38 out of 42, Claude-SearchBot 47 out of 52. Nobody borrowed Meta’s names, the one large company whose crawlers can’t be checked against a published IP list.

llms.txt was hardly read. Across every crawler, it was requested three times in five days: once each by GPTBot, Googlebot and GoogleOther. ClaudeBot, on the other hand, asked for robots.txt 30 times while requesting 19 pages.

One site, five days. The 38% is a lower bound: it counts only requests for files that are never a page, so a scanner asking for /admin under ChatGPT’s name counts as a page request.

What robots.txt can and can’t do

robots.txt is a request, not a lock. The training and search crawlers of the companies above say they follow it. The crawlers that read for an answer mostly say they may not, on the grounds that a person asked: OpenAI, Google, Perplexity, Meta and Amazon all say so in one form or another. Anthropic is the exception.

So the choice robots.txt gives you is mostly about training. If you want to stay out of training sets but keep turning up in AI answers, block the training names and leave the rest alone:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

CCBot is Common Crawl, whose public archive many models are trained on. Amazonbot is there because Amazon says it may train on what it collects, but it also feeds other Amazon products, so leave it out if that matters to you. Expect a day before a change takes effect: OpenAI and Perplexity name about 24 hours, Meta caches robots.txt for up to 24 hours and Amazon for up to 30 days.

Why your analytics shows none of this

Analytics tools such as Google Analytics, Plausible or Privatrak count a visit when the page’s JavaScript runs in a browser. A crawler that reads the page as text never runs it. When we sent AI services to a test site, ChatGPT, Claude, Gemini, Perplexity, Mistral and Meta AI all read the page without running a single script. None of them reached our analytics, Google Analytics included.

Our own site agrees. In the five days above, ChatGPT-User, Claude-User and the other answer crawlers requested 97 pages and left not one page view in our analytics. Search crawlers sometimes do run the script: OAI-SearchBot showed up 4 times and meta-webindexer 10 times.

That is fine for training and search crawlers, which you don’t want in your visitor numbers anyway. It is a real gap for the answer crawlers. Each of their requests is a person asking an AI service a question your page answers, and that is a number worth knowing. It shows whether AI services read you at all, which pages they read, and whether that turns into visitors clicking through.

Only your server sees those requests. Your server log has them; so does a log at your CDN. Privatrak can count them if your server reports the requests it answers: a setting in the WordPress plugin, a Cloudflare Worker, or a few lines in Next.js or Nuxt. Your server only reports requests from known crawlers, so nothing about your visitors leaves it. The AI services page then shows how often ChatGPT, Claude and the others read your pages for an answer, next to the visitors they sent you.

Is it really ClaudeBot? Check the IP address

A user agent is a claim, and anyone can make it. Google says as much: user agent strings “can be spoofed.” On our own site, at least 38% of the requests carrying an AI crawler’s name were scanners (see above), borrowing a name that sites tend to let through.

The one reliable check is the address the request came from. Most companies publish the IP ranges their crawlers use, and a request from outside them is not that crawler, whatever it says it is:

Every list was reachable on 3 November 2026. Mistral publishes none for MistralAI-Training. Google and Microsoft also document a reverse DNS check, and Bing has a Verify Bingbot tool.

Meta is the one large company that publishes no list, so a request calling itself meta-externalagent can’t be checked this way. The lists also change, so a firewall rule built from them needs refreshing. Anthropic warns that blocking by IP “may not work correctly or persistently guarantee an opt-out”: the lists are for checking who a request is from, robots.txt is for saying what you want.

The short version

• One AI company, up to three crawlers: one for training, one for its search, one that reads for a person’s answer. Each is a separate name in robots.txt.
• Blocking a training crawler keeps you out of future training data and nothing else. Blocking a search crawler takes you out of that service’s answers.
• Crawlers that read for an answer mostly don’t follow robots.txt. Anthropic’s does.
• None of the AI services we tested ran the page’s JavaScript, so only your server sees them.
• On our site, at least 38% of the requests under AI crawler names were scanners looking for passwords, and llms.txt was requested three times in five days.
• A name is a claim. The published IP lists are how you check it.