Search engines
Search Engine Crawlers
These bots crawl and index web pages for search engines. They determine which pages appear in search results.
- bots profiled
- 47
- respect robots.txt
- 15
- publish IP verification
- 5
Bots in this category
- Algolia Algolia No robots.txt commitment
- Baiduspider Baidu Respects robots.txt
- Bingbot Microsoft Respects robots.txt
- Cốc Cốc Coccoc No robots.txt commitment
- Coveo Bot Coveo No robots.txt commitment
- Dataprovider.com Dataprovider.com No robots.txt commitment
- DuckDuckBot DuckDuckGo Respects robots.txt
- Funnelback Squiz - FunnelBack No robots.txt commitment
- Google Images Google Respects robots.txt
- Google Videos Google No robots.txt commitment
- Googlebot Google Respects robots.txt
- Marginalia Search Marginalia No robots.txt commitment
- Mojeek Mojeek Respects robots.txt
- MSN Microsoft No robots.txt commitment
- Qwantbot Qwant Respects robots.txt
- Seekport SISTRIX No robots.txt commitment
- Seznam Seznam No robots.txt commitment
- Toutiao Bytedance No robots.txt commitment
- Yahoo Slurp Yahoo Respects robots.txt
- YandexBot Yandex Respects robots.txt
- Yeti Naver No robots.txt commitment
- ZoomInfo ZoomInfo No robots.txt commitment
- All Africa Crawler AllAfrica Global Media No activity data No robots.txt commitment
- Baidu ADS Server Proxy Baidu No activity data No robots.txt commitment
- Crawlson Crawlson No activity data Respects robots.txt
- Daum Daum No activity data No robots.txt commitment
- Freespoke Freespoke No activity data No robots.txt commitment
- Gigabot Gigablast No activity data No robots.txt commitment
- Google Scholar Google No activity data No robots.txt commitment
- Hype Machine Hype Machine No activity data No robots.txt commitment
- IbouBot Babbar No activity data No robots.txt commitment
- jobswithgptcom-bot jobswithgptcom No activity data No robots.txt commitment
- Kagi Bot Kagi No activity data No robots.txt commitment
- Level9SearchBot Level 9 No activity data No robots.txt commitment
- Linespider LINE Corporation No activity data Respects robots.txt
- MagiBot Peak Labs No activity data No robots.txt commitment
- Mars Finder Mars Flag No activity data Respects robots.txt
- MRGbot MRG Web Services srl No activity data Respects robots.txt
- Neevabot Neeva No activity data No robots.txt commitment
- NewsNow NewsNowUK No activity data No robots.txt commitment
- Notabot Helpfeel No activity data No robots.txt commitment
- Potions Potions No activity data No robots.txt commitment
- SiteSearch360 SEMKNOX GmbH No activity data No robots.txt commitment
- SMTnet PM Bot SMTnet No activity data Respects robots.txt
- Sogou Sogou No activity data No robots.txt commitment
- Yahoo! JAPAN Yahoo! JAPAN No activity data Respects robots.txt
- ZumBot Zum Internet Corp No activity data No robots.txt commitment
Crawl activity, 25 Oct - 21 Nov 2026 · Activity trend: Cloudflare Radar
About this category
Search engine crawlers are the foundation of web search. They systematically visit websites, analyze content, and build the search index that powers organic results. Many now also feed AI features like AI Overviews and Copilot answers.
Block every Search engines in one go
Rules for the 14 bots in this category that document a robots.txt token. Bots without a token can't be targeted this way.
# Search engines
User-agent: Baiduspider
User-agent: bingbot
User-agent: DuckDuckBot
User-agent: Googlebot-Image
User-agent: Googlebot
User-agent: MojeekBot
User-agent: Qwantbot
User-agent: Slurp
User-agent: YandexBot
User-agent: jobswithgptcom-bot
User-agent: Linespider
User-agent: MRGbot
User-agent: Notabot
User-agent: SMTnetPMBot
Disallow: / Waitlist open
Know which of these bots visit you
Can AI See It shows which bots from this catalogue hit your site, which are fake and what each platform sends back.