AI training crawlers
AI Training Crawlers
These bots crawl websites to collect training data for AI models. Many respect robots.txt - giving you control over whether your content is used for AI training.
- bots profiled
- 26
- respect robots.txt
- 11
- publish IP verification
- 4
Bots in this category
- Amazonbot Amazon Respects robots.txt
- Echobot Bot Echobox No robots.txt commitment
- Factset_spyderbot Factset No robots.txt commitment
- Google-CloudVertexBot Google No robots.txt commitment
- GoogleOther Google No robots.txt commitment
- GPTBot OpenAI Respects robots.txt
- ICC Crawler NICT Respects robots.txt
- Meta-ExternalAgent Meta Respects robots.txt
- PetalBot Huawei Respects robots.txt
- QualifiedBot Qualified.com, Inc. No robots.txt commitment
- SemrushBot-OCOB Semrush No robots.txt commitment
- ShapBot Parallel No robots.txt commitment
- Amazon Kendra Amazon No activity data No robots.txt commitment
- Anchor Browser Anchor No activity data No robots.txt commitment
- AwarioSmartBot Awario No activity data Respects robots.txt
- Big Sur AI Big Sur AI No activity data No robots.txt commitment
- Brandwatch Brandwatch No activity data Respects robots.txt
- Ceramic TerraCotta Ceramic No activity data Respects robots.txt
- ClaudeBot Anthropic No activity data Respects robots.txt
- Cotoyogi Research Organization of Information and Systems No activity data Respects robots.txt
- Google NotebookLM Google No activity data No robots.txt commitment
- LINER Bot Liner Bot No activity data No robots.txt commitment
- netEstate Imprint Crawler netEstate No activity data No robots.txt commitment
- Novellum AI Crawl Novellum No activity data No robots.txt commitment
- SemrushBotSwa SEMrush No activity data No robots.txt commitment
- WARDBot WEBSPARK No activity data Respects robots.txt
Crawl activity, 25 Oct - 21 Nov 2026 · Activity trend: Cloudflare Radar
About this category
AI training crawlers download web pages so their operators can use the content to train large language models and other AI systems. Understanding which bots are active on your site - and whether they respect your access rules - is the first step in controlling how your content is used.
Block every AI training crawlers in one go
Rules for the 13 bots in this category that document a robots.txt token. Bots without a token can't be targeted this way.
# AI training crawlers
User-agent: Amazonbot
User-agent: GPTBot
User-agent: ICC-Crawler
User-agent: meta-externalagent
User-agent: PetalBot
User-agent: SemrushBot-OCOB
User-agent: AwarioSmartBot
User-agent: magpie-crawler
User-agent: TerraCotta
User-agent: ClaudeBot
User-agent: Cotoyogi
User-agent: LinerBot
User-agent: WARDBot
Disallow: / Waitlist open
Know which of these bots visit you
Can AI See It shows which bots from this catalogue hit your site, which are fake and what each platform sends back.