AI search crawlers
AI Search Crawlers
These bots crawl websites to power AI-generated search results. They fetch your content so AI can cite it in answers.
- bots profiled
- 5
- respect robots.txt
- 4
- publish IP verification
- 2
Bots in this category
- Applebot Apple Respects robots.txt
- Bravebot Brave Software, Inc. No robots.txt commitment
- OAI-SearchBot OpenAI Respects robots.txt
- AutoRAG Cloudflare No activity data Respects robots.txt
- PerplexityBot Perplexity AI No activity data Respects robots.txt
Crawl activity, 25 Oct - 21 Nov 2026 · Activity trend: Cloudflare Radar
About this category
AI search crawlers fetch web pages in real time to build AI-powered search results. When these bots access your content, it may appear as a source in AI-generated answers - sometimes with a link back to your site, sometimes not.
Block every AI search crawlers in one go
Rules for the 4 bots in this category that document a robots.txt token. Bots without a token can't be targeted this way.
# AI search crawlers
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: Cloudflare-AutoRAG
User-agent: PerplexityBot
Disallow: / Waitlist open
Know which of these bots visit you
Can AI See It shows which bots from this catalogue hit your site, which are fake and what each platform sends back.