AI search crawlers

AI Search Crawlers

These bots crawl websites to power AI-generated search results. They fetch your content so AI can cite it in answers.

bots profiled
5
respect robots.txt
4
publish IP verification
2

Bots in this category

Crawl activity, 25 Oct - 21 Nov 2026 · Activity trend: Cloudflare Radar

About this category

AI search crawlers fetch web pages in real time to build AI-powered search results. When these bots access your content, it may appear as a source in AI-generated answers - sometimes with a link back to your site, sometimes not.

Block every AI search crawlers in one go

Rules for the 4 bots in this category that document a robots.txt token. Bots without a token can't be targeted this way.

robots.txt
# AI search crawlers
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: Cloudflare-AutoRAG
User-agent: PerplexityBot
Disallow: /

Waitlist open

Know which of these bots visit you

Can AI See It shows which bots from this catalogue hit your site, which are fake and what each platform sends back.

No spam. Launch updates only.