Each checked crawler gets its own block. robots.txt is a request, not enforcement — well-behaved crawlers honour it, others ignore it entirely.

OpenAI

GPTBot ChatGPT-User Fetches only when a user asks for your page — blocking it can drop you from cited answers. OAI-SearchBot Builds that product's search index — blocking removes you from its results.

Anthropic

ClaudeBot Claude-User Fetches only when a user asks for your page — blocking it can drop you from cited answers. Claude-SearchBot Builds that product's search index — blocking removes you from its results. anthropic-ai Older token some tools still honour; harmless alongside the current one.

Google

Google-Extended Training opt-out only — does not affect this vendor's search crawling or indexing.

Apple

Applebot-Extended Training opt-out only — does not affect this vendor's search crawling or indexing.

Common Crawl

CCBot

Perplexity

PerplexityBot Builds that product's search index — blocking removes you from its results. Perplexity-User Fetches only when a user asks for your page — blocking it can drop you from cited answers.

ByteDance

Bytespider