Search engines

Search Engine Crawlers

These bots crawl and index web pages for search engines. They determine which pages appear in search results.

bots profiled
47
respect robots.txt
15
publish IP verification
5

Bots in this category

Crawl activity, 25 Oct - 21 Nov 2026 · Activity trend: Cloudflare Radar

About this category

Search engine crawlers are the foundation of web search. They systematically visit websites, analyze content, and build the search index that powers organic results. Many now also feed AI features like AI Overviews and Copilot answers.

Block every Search engines in one go

Rules for the 14 bots in this category that document a robots.txt token. Bots without a token can't be targeted this way.

robots.txt
# Search engines
User-agent: Baiduspider
User-agent: bingbot
User-agent: DuckDuckBot
User-agent: Googlebot-Image
User-agent: Googlebot
User-agent: MojeekBot
User-agent: Qwantbot
User-agent: Slurp
User-agent: YandexBot
User-agent: jobswithgptcom-bot
User-agent: Linespider
User-agent: MRGbot
User-agent: Notabot
User-agent: SMTnetPMBot
Disallow: /

Waitlist open

Know which of these bots visit you

Can AI See It shows which bots from this catalogue hit your site, which are fake and what each platform sends back.

No spam. Launch updates only.