AI training crawlers

AI Training Crawlers

These bots crawl websites to collect training data for AI models. Many respect robots.txt - giving you control over whether your content is used for AI training.

bots profiled
26
respect robots.txt
11
publish IP verification
4

Bots in this category

Crawl activity, 25 Oct - 21 Nov 2026 · Activity trend: Cloudflare Radar

About this category

AI training crawlers download web pages so their operators can use the content to train large language models and other AI systems. Understanding which bots are active on your site - and whether they respect your access rules - is the first step in controlling how your content is used.

Block every AI training crawlers in one go

Rules for the 13 bots in this category that document a robots.txt token. Bots without a token can't be targeted this way.

robots.txt
# AI training crawlers
User-agent: Amazonbot
User-agent: GPTBot
User-agent: ICC-Crawler
User-agent: meta-externalagent
User-agent: PetalBot
User-agent: SemrushBot-OCOB
User-agent: AwarioSmartBot
User-agent: magpie-crawler
User-agent: TerraCotta
User-agent: ClaudeBot
User-agent: Cotoyogi
User-agent: LinerBot
User-agent: WARDBot
Disallow: /

Waitlist open

Know which of these bots visit you

Can AI See It shows which bots from this catalogue hit your site, which are fake and what each platform sends back.

No spam. Launch updates only.