Open bot catalogue

AI bot & web crawler directory

487 bots from 392 operators, with user-agent strings, robots.txt rules and verification steps. Free to use, no signup.

bots profiled
487
operators
392
AI bots
40
respect robots.txt
70

AI training crawlers 26

AI model training

Category guide

AI assistants 9

AI assistant content fetching

Category guide

Search engines 47

Search indexing

Category guide

SEO tools 58

SEO analysis

Social media bots 35

Link preview generation

Monitoring bots 107

Website monitoring

Web archivers 3

Web archiving

Advertising bots 38

Ad verification

Security scanners 31

Security scanning

Other bots 128

Web crawling

How this catalogue is built

Profiles are prepared with AI assistance from what operators publish: documentation pages, help articles and bot info pages. Where a source exists, the profile links to it, and anything the operator doesn't document is marked as unknown.

Waitlist open

Know which of these bots visit you

Can AI See It shows which bots from this catalogue hit your site, which are fake and what each platform sends back.

No spam. Launch updates only.