Open bot catalogue
AI bot & web crawler directory
487 bots from 392 operators, with user-agent strings, robots.txt rules and verification steps. Free to use, no signup.
- bots profiled
- 487
- operators
- 392
- AI bots
- 40
- respect robots.txt
- 70
AI crawlers to know
- GPTBot OpenAI AI training Respects robots.txt
- OAI-SearchBot OpenAI AI search Respects robots.txt
- ChatGPT-User OpenAI AI assistant robots.txt not documented
- Meta-ExternalAgent Meta AI training Respects robots.txt
- Applebot Apple AI search Respects robots.txt
- Googlebot Google Search Respects robots.txt
Crawl activity, 25 Oct - 21 Nov 2026 · Activity trend: Cloudflare Radar
AI training crawlers 26
AI model training
- Amazon Kendra Amazon Amazon Kendra is an intelligent search service operated by Amazon, using natural language processing for accurate search results. robots.txt: No
- Amazonbot Amazon Amazonbot is Amazon's web crawler used to improve products and services, such as training AI models. robots.txt: Yes
- Anchor Browser Anchor The Anchor Browser is a web browser for AI agents, operated by Anchor, used for automating workflows and web interactions. robots.txt: No
- AwarioSmartBot Awario AwarioSmartBot is a web crawler operated by Awario, used for collecting new and updated web data for Internet marketers. robots.txt: Yes
- Big Sur AI Big Sur AI Big Sur AI Crawler, operated by Big Sur AI, crawls user websites for AI-infused experiences. robots.txt: No
- Brandwatch Brandwatch Brandwatch's Magpie Crawler indexes web pages for social media monitoring and analysis. robots.txt: Yes
- Ceramic TerraCotta Ceramic Ceramic TerraCotta is a web crawler operated by Ceramic, an AI training infrastructure company. It crawls websites to support Ceramic's AI model training optimization platform. robots.txt: Yes
- ClaudeBot Anthropic ClaudeBot is Anthropic's web crawler that collects web content for training Claude AI models. It respects robots.txt directives and supports Crawl-delay. robots.txt: Yes
- Cotoyogi Research Organization of Information and Systems Cotoyogi is a bot operated by the Research Organization of Information and Systems for AI training purposes. robots.txt: Yes
- Echobot Bot Echobox Echobot is a web scraping bot operated by Echobox for AI training purposes, specifically for automating content distribution for digital publishers. robots.txt: No
- Factset_spyderbot Factset Factset Spyderbot is a web scraping bot operated by Factset for delivering reliable financial data. robots.txt: No
- Google NotebookLM Google Google NotebookLM bot operated by Google for AI training. robots.txt: No
- Google-CloudVertexBot Google Google-CloudVertexBot is a crawler operated by Google for targeted AI training of site owners' own sites. robots.txt: No
- GoogleOther Google GoogleOther is a generic crawler operated by Google for fetching publicly accessible content from sites, used for internal research and development. robots.txt: No
- GPTBot OpenAI GPTBot is used to crawl content that may be used in training OpenAI's generative AI foundation models. robots.txt: Yes
- ICC Crawler NICT ICC Crawler is a web crawler operated by NICT, collecting web pages for AI training. robots.txt: Yes
- LINER Bot Liner Bot LINER Bot is a web crawler operated by Liner Bot, used for AI training by collecting data from the internet. robots.txt: No
- Meta-ExternalAgent Meta Meta-ExternalAgent is a bot operated by Meta for AI training purposes, specifically for training AI models or improving products by indexing content directly. robots.txt: Yes
- netEstate Imprint Crawler netEstate The NetEstate Imprint crawler crawls websites for public contact information. robots.txt: No
- Novellum AI Crawl Novellum Novellum.ai is building out tools for building agents. This MCP tool will be used by agents to crawl sites. robots.txt: No
- PetalBot Huawei PetalBot is a search engine crawler operated by Huawei, used for indexing websites and providing content recommendations in Petal Search engine, Huawei Assistant, and AI Search services. robots.txt: Yes
- QualifiedBot Qualified.com, Inc. QualifiedBot is a crawler operated by Qualified.com, Inc. to power AI products by crawling customer websites. robots.txt: No
- SemrushBot-OCOB Semrush SemrushBot-OCOB is a bot operated by Semrush for ai-training, specifically for Content Toolkit. robots.txt: No
- SemrushBotSwa SEMrush SEMrushBotSwa is a bot operated by SEMrush for ai-training purposes, collecting data for the SEO Writing Assistant tool. robots.txt: No
- ShapBot Parallel Not officially documented robots.txt: No
- WARDBot WEBSPARK WARDBot is a monitoring bot operated by WEBSPARK that tracks URL status codes for users. robots.txt: Yes
AI search crawlers 5
AI-powered search
- Applebot Apple Applebot data is used to power various features, such as the search technology that is integrated into many user experiences in Apple’s ecosystem including Spotlight, Siri, and Safari. robots.txt: Yes
- AutoRAG Cloudflare AutoRAG is Cloudflare's AI search solution for indexing and querying data. robots.txt: Yes
- Bravebot Brave Software, Inc. Bravebot is a search crawler operated by Brave Software, Inc., used for discovering new pages and indexing their content. robots.txt: No
- OAI-SearchBot OpenAI OAI-SearchBot is used to link to and surface websites in search results in OpenAI's SearchGPT and ChatGPT search features. robots.txt: Yes
- PerplexityBot Perplexity AI PerplexityBot is Perplexity AI's search crawler, used to surface and link websites in Perplexity search results. It is explicitly not used to crawl content for AI foundation models. robots.txt: Yes
AI assistants 9
AI assistant content fetching
- Amazon Buy For Me Amazon Buy For Me agent places orders on e-commerce websites at the direction of customers. robots.txt: No
- ChatGPT Agent OpenAI ChatGPT Agent is OpenAI's agent mode that uses a visual browser to navigate websites and perform tasks on behalf of users. robots.txt: No
- ChatGPT-User OpenAI ChatGPT-User is for user actions in ChatGPT and Custom GPTs. When users ask ChatGPT or a Custom GPT a question, it may visit a web page to help answer. robots.txt: Not documented
- Devin Devin AI Devin is an autonomous AI software engineer operated by Devin AI, designed to assist engineering teams with coding tasks. robots.txt: No
- DuckAssistbot DuckDuckGo DuckAssistBot is a web crawler operated by DuckDuckGo for AI-assisted search answers. robots.txt: Yes
- Meta-ExternalFetcher Meta Meta-ExternalFetcher is a crawler operated by Meta, receiving individual links at user's initiative to support product features. robots.txt: No
- MistralAI-User Mistral AI MistralAI-User bot, operated by Mistral AI, for user actions in le Chat. robots.txt: No
- Perplexity-User Perplexity AI Perplexity-User is a user-initiated fetcher that visits web pages in real time when a Perplexity user asks a question that requires reading a specific page to produce an accurate answer. robots.txt: No
- TwinAgent Twin TwinAgent is a bot operated by Twin that automates complex operations end-to-end using AI. robots.txt: No
Search engines 47
Search indexing
- Algolia Algolia Indexes content for Algolia search engine robots.txt: No
- All Africa Crawler AllAfrica Global Media All Africa Crawler, operated by AllAfrica Global Media, aggregates news from Africa. robots.txt: No
- Baidu ADS Server Proxy Baidu Baidu's DDoS scrubbing proxy. robots.txt: No
- Baiduspider Baidu Baiduspider is the search engine crawler for the search engine Baidu. robots.txt: Yes
- Bingbot Microsoft Bingbot is a crawler from Microsoft that powers Bing search. robots.txt: Yes
- Cốc Cốc Coccoc Cốc Cốc bot, operated by Coccoc, is a search engine bot that scrapes websites requested by the Vietnamese search engine Coc Coc. robots.txt: No
- Coveo Bot Coveo Coveo Bot is a search engine bot operated by Coveo, providing relevant experiences to end users by crawling websites. robots.txt: No
- Crawlson Crawlson Crawlson is a search engine crawler for the crawlson.com search engine. robots.txt: Yes
- Dataprovider.com Dataprovider.com Dataprovider.com indexes the web and structures the data. robots.txt: No
- Daum Daum Daum is a Korean search engine crawler operated by Daum. robots.txt: No
- DuckDuckBot DuckDuckGo DuckDuckBot is a web crawler for DuckDuckGo. DuckDuckBot's job is to constantly crawl the web and find web pages to add to the DuckDuckGo search index. robots.txt: Yes
- Freespoke Freespoke Freespoke is a search engine that believes in free speech and shows you all viewpoints. robots.txt: No
- Funnelback Squiz - FunnelBack Funnelback is an enterprise search platform operated by Squiz, used for indexing content from an organization's websites and data repositories to power internal search functions. robots.txt: No
- Gigabot Gigablast Gigablast is the only non-Big Tech search engine in the U.S. that uses its own search index and algorithms. robots.txt: No
- Google Images Google Google Images bot is a search engine crawler operated by Google for Google Images Search. robots.txt: Yes
- Google Scholar Google Google Scholar bot crawls and indexes scholarly literature from academic publishers, repositories, and university websites. robots.txt: No
- Google Videos Google The Google Videos bot is a search engine crawler operated by Google for Google Video Search. robots.txt: No
- Googlebot Google Google uses crawlers and fetchers to perform actions for its products, either automatically or triggered by user request. robots.txt: Yes
- Hype Machine Hype Machine Hype Machine is a search engine bot operated by Hype Machine that monitors music publications/blogs for new artists. robots.txt: No
- IbouBot Babbar IbouBot is a search engine crawler operated by Babbar, used to fuel and update the graph representation of the World Wide Web. robots.txt: No
- jobswithgptcom-bot jobswithgptcom jobswithgptcom-bot is a crawler operated by jobswithgptcom, focusing on job postings for jobswithgpt.com. robots.txt: No
- Kagi Bot Kagi Kagi Bot is a web crawler for the Kagi search engine, operated by Kagi, used to build its own search index for its ad-free search product. robots.txt: No
- Level9SearchBot Level 9 Level9SearchBot is a search engine bot operated by Level 9. robots.txt: No
- Linespider LINE Corporation Linespider is a web crawler operated by LINE Corporation, providing search results for LINE services. robots.txt: Yes
- MagiBot Peak Labs MagiBot is owned by Peak Labs which focuses on the research and development of information extraction and retrieval technology to transform knowledge in natural language into immeasurable value. robots.txt: No
- Marginalia Search Marginalia Marginalia Search is a noncommercial niche search engine focusing on old websites, personal websites, and blogs. robots.txt: No
- Mars Finder Mars Flag Mars Finder is a website search service operated by Mars Flag, designed to utilize the maximum potential of a website. robots.txt: Yes
- Mojeek Mojeek MojeekBot is a web crawler for the Mojeek search engine, operated by Mojeek. robots.txt: Yes
- MRGbot MRG Web Services srl MRGbot is a search engine bot operated by MRG Web Services srl, aimed at generating a corpus of data. robots.txt: Yes
- MSN Microsoft MSNBot is a web crawler operated by Microsoft for indexing web pages for MSN Search. robots.txt: No
- Neevabot Neeva Neevabot is the web crawler for the search engine neeva.com. robots.txt: No
- NewsNow NewsNowUK NewsNow bot is a web crawler for the news aggregator service NewsNow, operated by NewsNowUK. robots.txt: No
- Notabot Helpfeel Notabot is a crawler operated by Helpfeel for integrating external search engine. robots.txt: No
- Potions Potions The Potions bot, operated by Potions, fetches product feeds and crawls e-commerce data. robots.txt: No
- Qwantbot Qwant Qwantbot is a web crawler operated by Qwant, a European search engine that prioritizes user privacy, used for search indexing and web content discovery. robots.txt: Yes
- Seekport SISTRIX Seekport is a search engine bot operated by SISTRIX, providing a public, free, and independent alternative to Google. robots.txt: No
- Seznam Seznam SeznamBot is a search engine crawler operated by Seznam for search indexing. robots.txt: No
- SiteSearch360 SEMKNOX GmbH SiteSearch360 is a search engine bot operated by SEMKNOX GmbH, used for indexing content on customer sites for search. robots.txt: No
- SMTnet PM Bot SMTnet SMTnet PM Bot is a search engine crawler operated by SMTnet, crawling partner company websites for on-site search engine inclusion. robots.txt: Yes
- Sogou Sogou Sogou spider is a search engine crawler operated by Sogou, used for indexing web content. robots.txt: No
- Toutiao Bytedance Toutiao is ByteDance's automated news aggregation bot collecting content across web platforms. robots.txt: No
- Yahoo Slurp Yahoo Yahoo Slurp is a search engine crawler operated by Yahoo, used for indexing web pages and collecting content. robots.txt: Yes
- Yahoo! JAPAN Yahoo! JAPAN Yahoo! JAPAN manages and operates a system that accesses web pages published on the Internet for the purpose of providing services, research, development, maintenance, etc. robots.txt: Yes
- YandexBot Yandex The main indexing robot for Yandex search. robots.txt: Yes
- Yeti Naver Yeti is a web crawler operated by Naver, a South Korean search engine, used for indexing websites to provide search results. robots.txt: No
- ZoomInfo ZoomInfo Zoominfobot is an indexing robot for a web search engine, similar to Google. Created by Zoom Information Inc.(www.zoominfo.com), Zoominfobot’s patented technology continually scans millions of corporate websites, press releases, electronic news services, SEC filings and other online sources. Usin... robots.txt: No
- ZumBot Zum Internet Corp ZumBot is a web crawler operated by Zum Internet Corp for indexing webpages for Zum Open Internet Search. robots.txt: No
SEO tools 58
SEO analysis
- AddSearchBot Addsearch AddSearchBot is a search indexing bot operated by Addsearch, used to add content from sites to their search results. robots.txt: No
- AddThis AddThis The AddThis bot, operated by AddThis, crawls websites to gather and update content for its website marketing tools. robots.txt: No
- Agency Analytics Crawler Agency Analytics Agency Analytics Crawler is a web crawler operated by Agency Analytics for SEO site checking. robots.txt: No
- Ahrefs Site Audit Ahrefs Ahrefs Site Audit bot, operated by Ahrefs, used for SEO site auditing. robots.txt: No
- AhrefsBot Ahrefs AhrefsBot is a web crawler operated by Ahrefs for SEO toolset, crawling the web to fill its 12 trillion link database. robots.txt: No
- Attracta Attracta The Attracta bot analyzes website content for SEO services, operated by Attracta. robots.txt: No
- Barkrowler Babbar Barkrowler is an SEO web crawler operated by Babbar, used to identify web page popularity and thematic. robots.txt: Yes
- BLEXBot WebMeUp BLexBot is a backlink analysis tool operated by WebMeUp, providing comprehensive backlink data for SEO purposes. robots.txt: No
- Botify Botify Botify is an SEO tool operated by Botify, used for website crawling and analytics. robots.txt: No
- CaliberBot Conductor CaliberBot is a web crawler operated by Conductor for SEO tool purposes, extracting HTML features for Content Analytics. robots.txt: No
- Chrome-Lighthouse Google Chrome-Lighthouse is an open-source tool by Google for auditing web page quality, not a traditional web crawler. robots.txt: No
- ClearscopeBot Mushi Labs ClearscopeBot is a web scraper operated by Mushi Labs for SEO content optimization, retrieving publicly available content from websites. robots.txt: Yes
- Cloudflare Custom Error Page Crawler Cloudflare Cloulflare internal service that crawls customer error pages in order to serve them directly from our edge network. robots.txt: No
- Cloudflare Digicert DCV Cloudflare Cloudflare Digicert DCV service operated by Cloudflare for domain control validation. robots.txt: No
- Cloudflare Healthchecks Cloudflare Cloudflare Healthchecks service robots.txt: No
- Cloudflare Prefetch Cloudflare Cloudflare Prefetch is a bot operated by Cloudflare for URL prefetching to improve cache hit rates and user experience. robots.txt: No
- Cloudflare Purge Cloudflare Cloudflare Purge service. robots.txt: No
- Cloudflare SpeedTest Cloudflare Cloudflare SpeedTest service operated by Cloudflare for SEO tool purposes. robots.txt: No
- Cloudflare SSLDetector Cloudflare Cloudflare SSLDetector service. robots.txt: No
- Cloudflare Stream Webhook Cloudflare Cloudflare Stream Webhook service. robots.txt: No
- Cloudflare-Traffic-Manager Cloudflare Cloudflare-Traffic-Manager service. robots.txt: No
- cognitiveSEO Crawler cognitive SEO Internet Marketing Tools. CognitiveSEO is an SEO toolset that crawls the web and analyzes links. robots.txt: No
- Cxense Cxense Cxensebot is a bot operated by Cxense for SEO monitoring and analysis. robots.txt: No
- DataForSEO DataForSEO DataForSEO bot operated by DataForSEO for SEO and digital marketing data solutions. robots.txt: No
- DataForSEO Bot DataForSEO DataForSEO Bot is an SEO tool operated by DataForSEO, used for backlink analysis and providing website owners, webmasters, and SEO professionals with backlink analytics. robots.txt: No
- DigiCert DCV DigiCert DigiCert DCV bot operated by DigiCert for SEO tool purposes. robots.txt: No
- EzoicBot Ezoic Inc EzoicBot is a web crawler by Ezoic Inc, designed to extract information about internet, search engines, and websites to help publishers understand site performance and search engine ranking. robots.txt: Yes
- Google Digital Asset Links Google The Digital Asset Links bot verifies statements lists made by website operators. robots.txt: No
- Google StoreBot Google Google StoreBot is a search-engine-based program that automatically crawls web pages to gather and analyze data, operated by Google. robots.txt: No
- Google-AdWords-Express Google Google-AdWords-Express is a Google bot for AdWords product aimed at small businesses, crawling advertiser websites for ad creation and site verification. robots.txt: No
- Google-Adwords-Instant Google Google-Adwords-Instant is a Google bot that visits advertiser landing pages for verification and quality checks. robots.txt: No
- Huckabuy Bot Huckabuy Huckabuy Bot is a web crawler operated by Huckabuy for SEO tool purposes, crawling and indexing websites for optimization. robots.txt: No
- LinkCheck SiteImprove Siteimprove's LinkCheck crawler analyzes websites for quality assurance, SEO, and accessibility purposes. robots.txt: No
- LinkChecker Bot LinkChecker LinkChecker Bot is a web crawler operated by LinkChecker that monitors backlinks profiles. robots.txt: Yes
- marketgoo marketgoo marketgoo is an SEO tool operated by marketgoo, providing white label SEO tools. robots.txt: Yes
- MJ12Bot Majestic MJ12bot is a web crawler operated by Majestic for SEO backlink analysis and search engine data mapping. robots.txt: No
- Moz dotbot Moz Moz Dotbot is a web crawler operated by Moz for SEO backlink analysis and site auditing. robots.txt: No
- Moz rogerbot Moz Moz's site audit crawler for Moz Pro Campaigns, operated by Moz. robots.txt: Yes
- NitroBot NitroPack Ltd NitroBot is a site speed optimization bot operated by NitroPack Ltd, making requests to clients' sites to fetch content for optimization. robots.txt: No
- OnCrawl Oncrawl OnCrawl is an enterprise SEO platform powered by an industry-leading SEO Crawler and Log Analyzer, operated by Oncrawl. robots.txt: No
- Online Webceo Bot Online Webceo Online Webceo Bot is a bot from Online Webceo that scans websites for SEO analysis and technical audits. robots.txt: No
- prerender Prerender, LLC It's HTML pre rendering service for SPA(Single Page Application) Website SEO. robots.txt: No
- Revvim Revvim Our bot crawls our customers' websites to identify SEO opportunities robots.txt: No
- SearchAtlas Bot Search Atlas SearchAtlas Bot is an SEO tool operated by Search Atlas, used to evaluate customer websites and provide SEO optimization strategies. robots.txt: No
- SemrushBot Semrush SemrushBot is the search bot software that Semrush sends out to discover and collect data from the web for SEO analysis and competitive research. robots.txt: Yes
- SemrushBotBacklinkAudit SEMrush SEMrushBotBacklinkAudit is a bot operated by SEMrush for SEO backlink analysis and site auditing. robots.txt: No
- SemrushBotBacklinks Semrush SemrushBotBacklinks is a bot operated by Semrush for collecting data for Semrush's Backlink Analytics tool. robots.txt: No
- SemrushBotLinkBuilding SEMrush SEMrushBotLinkBuilding is a bot operated by SEMrush for SEO backlink analysis. robots.txt: No
- SemrushBotSI Semrush SEMrushBotSI is a bot operated by Semrush for SEO tool purposes, specifically for data collection. robots.txt: No
- SEO Audit Check Bot SEO Audit Check SEO Audit Check Bot is an automated tool by SEO Audit Check that performs comprehensive SEO audits on websites. robots.txt: No
- seo4ajax Capsule Code The seo4ajax bot, operated by Capsule Code, helps make single-page applications (SPAs) crawlable by search engines. robots.txt: No
- SeobilityBot Seobility The Seobility Bot is an SEO tool operated by Seobility that crawls websites to gather SEO information. robots.txt: No
- SerpstatBot Serpstat SerpstatBot is a bot operated by Serpstat, used for SEO backlink analysis and site auditing by collecting data for Serpstat's Backlink Analysis tool. robots.txt: No
- SiteAuditBot Semrush SiteAuditBot is a SEO tool operated by Semrush that checks for common website issues and provides reports on crawlability, markups, internal linking, speed, HTTPS, and international SEO. robots.txt: No
- Siteimprove Crawl Siteimprove Siteimprove Crawl bot, operated by Siteimprove, for SEO and content suite analysis. robots.txt: No
- Trellis-Services Mediavine Trellis-Services bot, operated by Mediavine, generates critical CSS for website optimization. robots.txt: No
- W3 Validator Services World Wide Web Consortium (W3C) W3C Validator Services is a bot operated by World Wide Web Consortium (W3C) that checks website conformance to open standards. robots.txt: Yes
- Yahoo Japan SEO Crawler Yahoo SEO crawler for Yahoo Japan. robots.txt: No
Monitoring bots 107
Website monitoring
- 360Monitoring 360Monitoring 360Monitoring bot checks website availability and performance from selected locations. robots.txt: No
- AccessStatus AccessStatus AccessStatus bot checks HTTP status codes of web pages to ensure link functionality. robots.txt: No
- ActiveComply Bot ActiveComply ActiveComply Bot is a crawler for social media monitoring and regulatory compliance, operated by ActiveComply. robots.txt: No
- Adagio Bot Adagio Adagio Bot, operated by Adagio, for monitoring and demand optimization solutions. robots.txt: No
- Alertsite by Smartbear Smartbear Alertsite by Smartbear is an HTTP monitoring probe operated by Smartbear that monitors websites for availability and performance anomalies. robots.txt: No
- Area 360 Area360 Area 360 is a property search and analytics bot operated by Area360, used for monitoring property insights. robots.txt: No
- Audisto Crawler Audisto Audisto Crawler is a website monitoring bot operated by Audisto, fetching all accessible URLs of a website. robots.txt: No
- Automaton Automaton Automaton is an end-to-end campaign and integration testing tool operated by Automaton, used for monitoring and optimizing marketing, advertising, and sales technology stacks. robots.txt: No
- Better Uptime Better Stack Better Uptime is a monitoring bot operated by Better Stack, used for monitoring infrastructure platforms. robots.txt: No
- BinaryCanary Binary Canary BinaryCanary is a website monitoring bot operated by Binary Canary, tracking availability and performance issues. robots.txt: No
- BlogVault BlogVault BlogVault is a monitoring bot operated by BlogVault, providing WordPress services like backup, security, and monitoring. robots.txt: No
- BugsNag Bugs Nag BugsNag is a monitoring bot operated by Bugs Nag for error and performance monitoring of web applications. robots.txt: No
- catchpoint Catchpoint Catchpoint is a monitoring bot operated by Catchpoint, focused on internet performance monitoring from the end-user perspective. robots.txt: No
- Checkly Bot Checkly Checkly Bot is a monitoring bot operated by Checkly, used for proactive monitoring and testing of websites and APIs. robots.txt: No
- Cloudflare Diagnostics Cloudflare Cloudflare diagnostic and testing services. robots.txt: No
- Cloudtrellis Cloudtrellis Cloudtrellis bot scans websites for broken links, accessibility issues, and SEO improvements, operated by Cloudtrellis. robots.txt: Yes
- Cludo Cludo Cludo is a monitoring bot operated by Cludo, facilitating search and analytics solutions. robots.txt: Yes
- ContentKing ContentKing ContentKing is a cloud-based service that monitors websites for technical issues and content changes. robots.txt: No
- Crazy Egg Bot CrazyEgg Crazy Egg bot that takes screenshots of pages, collects assets, tests script installation. robots.txt: No
- deadlinkchecker DLC Websites Dead Link Checker bot, operated by DLC Websites, crawls websites to identify broken links. robots.txt: No
- Deepcrawl Lumar Deepcrawl bot, operated by Lumar, crawls websites for performance analytics and SEO optimizations. robots.txt: No
- Doctom Monitor Dotcom-Monitor The Doctom Monitor is a website monitoring bot operated by Dotcom-Monitor, checking websites for uptime and performance issues. robots.txt: No
- Dr. Link Check Dr. Link Check Dr. Link Check is a monitoring bot operated by Dr. Link Check that crawls websites to identify and fix broken links. robots.txt: No
- Easydns EasyDNS Easydns is a monitoring bot operated by EasyDNS, used for uptime monitoring. robots.txt: No
- elmah.io Uptime Monitoring elmah.io elmah.io Uptime Monitoring bot monitors website availability for elmah.io users. robots.txt: No
- eRepublik.tools Sebastian Foth - Software Solutions eRepublik.tools is a monitoring bot operated by Sebastian Foth - Software Solutions, providing gaming statistics and tools for the browser game 'eRepublik'. robots.txt: No
- EvoUptimeBot Evo Agency Ltd. EvoUptimeBot is a monitoring bot operated by Evo Agency Ltd. for uptime monitoring of the EvoCommerce platform. robots.txt: No
- FDL Stats Bots FTW Entertainment LLC FDL Stats Bot, operated by FTW Entertainment LLC, generates analytical data on rocket league player information. robots.txt: No
- Foregenix ThreatView/WebScan Foregenix Limited Foregenix ThreatView/WebScan is a security and risk scanning bot operated by Foregenix Limited, monitoring websites for malware and vulnerabilities. robots.txt: No
- FullStory Full Story FullStory is your digital experience analytics platform for on-the-fly funnels, pixel-perfect replay, custom events, heat maps, advanced search, Dev Tools, and more. FullStoryBot’s fetches and stores assets required to rebuild sites when viewing recorded sessions. robots.txt: No
- Ghost Inspector Ghost Inspector Ghost Inspector is a monitoring bot operated by Ghost Inspector for automated browser testing. robots.txt: No
- Google Schema Markup Testing Tool Google Google Schema Markup Testing Tool bot, operated by Google, validates structured data on web pages. robots.txt: No
- Grafana's Synthetic Monitoring Grafana Labs Grafana's Synthetic Monitoring agent provides probe functionality and executes network checks for monitoring remote targets. robots.txt: No
- GTmetrix GTmetrix GTmetrix is a free tool that analyzes a page's speed performance. Using PageSpeed and YSlow, GTmetrix generates scores for pages and offers actionable recommendations on how to fix them. robots.txt: No
- GuestpostsBot Guest Posts GuestpostsBot is a Web Crawler that has several functions to facilitate the website owner who has registered his site on the guestposts.com.br platform to monitor his site. The bot constantly tracks the sites registered on the platform in order to check if the partnerships made on the guestpost p... robots.txt: No
- HetrixTools HetrixTools Inc HetrixTools Uptime Monitoring Bot checks website and server uptime, operated by HetrixTools Inc. robots.txt: No
- honeybadgerbot Honeybadger Industries HoneybadgerBot is a monitoring bot operated by Honeybadger Industries for error and uptime monitoring services. robots.txt: No
- HostTracker HostTracker HostTracker is a monitoring bot operated by HostTracker that tracks website availability and performance. robots.txt: No
- Hotjar Hotjar Hotjar is a user analytics and feedback tool operated by Hotjar, now part of Contentsquare. robots.txt: No
- InternetArchiveBot Internet Archive InternetArchiveBot is a monitoring bot operated by Internet Archive that checks URL references on Wikipedia for broken links. robots.txt: No
- IsDownBot IsDownBot IsDownBot monitors endpoints to ensure they are up and running, operated by IsDownBot. robots.txt: No
- Jagged Pixel UptimeBot Jagged Pixel Inc. UptimeBot by Jagged Pixel Inc. monitors Shopify store uptime and alerts for outages. robots.txt: No
- Jetpack Automattic Jetpack is a monitoring bot operated by Automattic, tracking uptime for WordPress.com and Jetpack users. robots.txt: No
- KargoBot-Artemis Kargo KargoBot-Artemis is Kargo's autonomous content verification bot. It's a simulation of a user on an iOS device. The bot is used to scan sites for content that may be unsuitable for customers on the Kargo ad network. robots.txt: No
- Kinsta Kinsta Kinsta bot for uptime monitoring and domain pointing verification robots.txt: No
- Link Tiger LinkTiger, Inc. LinkTiger is a monitoring bot operated by LinkTiger, Inc. that crawls customer sites and reports broken links. robots.txt: No
- logicmonitor LogicMonitor LogicMonitor bot performs synthetic monitoring checks on websites and online services. robots.txt: No
- Macrobondbot Macrobondbot Macrobondbot is a custom web crawler operated by Macrobondbot to fetch macroeconomic data from official sources. robots.txt: No
- Mavifinds Bot Mavifinds The Mavifinds Bot is part of a security service that monitors websites. It can automatically activate security measures, such as an under attack mode, in response to threats. robots.txt: No
- Medialogia Bot Medialogia Medialogia Bot is a web crawler operated by Medialogia, a Russian media monitoring company, collecting data from online news and social media. robots.txt: No
- Monitage Monitage Monitage is a monitoring bot operated by Monitage, used for uptime monitoring of websites and services. robots.txt: No
- Monitis Monitis Monitis bot for premium monitoring service, operated by Monitis. robots.txt: No
- Monsidobot Acquia Monsidobot is a monitoring bot operated by Acquia, used for scanning and improving content and web accessibility. robots.txt: No
- MonSpark MonSpark MonSpark is a website monitoring service operated by MonSpark, checking website availability, network conditions, and TLS certificate validity. robots.txt: No
- MontasticMonitor Montastic MontasticMonitor is a website availability monitoring bot operated by Montastic. robots.txt: No
- Netumo Netumo Netumo is a website uptime monitor service operated by Netumo, providing monitoring and verification of online infrastructure. robots.txt: No
- NETVIGIE Netvigie NETVIGIE is a monitoring bot operated by Netvigie, checking website and mobile app availability and performance. robots.txt: No
- New Relic New Relic New Relic bot, operated by New Relic, monitors customer applications for availability and performance issues. robots.txt: No
- NixStatsMonitoringBot NixStats NixStatsMonitoringBot is a monitoring bot operated by NixStats to track website availability and performance. robots.txt: No
- Nodeping Nodeping Nodeping is a monitoring bot operated by Nodeping that checks websites for uptime. robots.txt: No
- Oh Dear Oh Dear Oh Dear is a monitoring bot operated by Oh Dear, used for checking website availability and performance. robots.txt: No
- Onetrust CMP Scanner OneTrust, LLC. The Onetrust CMP Scanner is a monitoring bot operated by OneTrust, LLC, that identifies and categorizes cookies and tracking tech on customer sites. robots.txt: No
- OnlineOrNot Monitor OnlineOrNot OnlineOrNot Monitor is a website monitoring bot operated by OnlineOrNot, providing uptime checks and page speed tests. robots.txt: No
- OutsellURLValidator Outsell Corporation OutsellURLValidator bot by Outsell Corporation monitors client website URLs for hosting/provider changes. robots.txt: No
- PaesslerCloudBot Paessler PaesslerCloudBot is a monitoring bot operated by Paessler, used for website availability and performance monitoring. robots.txt: No
- Panopta Panopta Panopta is an uptime monitoring service acquired by Fortinet. robots.txt: No
- Parse.ly Parse.ly Parse.ly is a content analytics tool operated by Parse.ly, used for monitoring content performance. robots.txt: Yes
- Pingdom Pingdom Pingdom bot is an HTTP monitoring probe for Pingdom's website monitoring service, operated by Pingdom. robots.txt: No
- PingPing PingPing PingPing is a website monitoring service operated by PingPing that checks website uptime and TLS certificate validity. robots.txt: No
- PingPing Bot PingPing.io PingPing Bot is a monitoring bot operated by PingPing.io to check website uptime and SSL validity. robots.txt: No
- Pulsetic Pulsetic Pulsetic is a website monitoring service operated by Pulsetic, providing uptime monitoring and down alerts. robots.txt: No
- Qualys SSL Scanner Qualys Qualys SSL Scanner is a monitoring bot operated by Qualys, used to test and monitor SSL ratings. robots.txt: No
- Rackspace Rackspace The Rackspace HTTP Monitor monitors customer websites for uptime and other issues. robots.txt: No
- RetroListeCOM Niclas Papst RetroListeCOM is a monitoring bot operated by Niclas Papst, tracking user counts on gaming-related websites. robots.txt: No
- SE Ranking Bot SE Ranking Crawler for analyzing SE Ranking clients websites for potential issues. robots.txt: No
- Sentry Uptime Monitoring Sentry Uptime Bot Sentry Uptime Monitoring bot checks websites and APIs for availability, operated by Sentry Uptime Bot. robots.txt: No
- SequelWP SequelWP SequelWP monitoring bot checks website uptime every 5 minutes for verified customers. robots.txt: No
- Server Density Stackpath Server Density bot by Stackpath for proactive infrastructure monitoring robots.txt: No
- Site24x7 Site24x7 Site24x7's global website monitoring probe operated by Site24x7 for monitoring purposes. robots.txt: No
- SiteUpTimeBot SiteUpTime SiteUpTimeBot is the HTTP monitoring probe for SiteUpTime.com. robots.txt: No
- SolarWinds Observability SolarWinds SolarWinds Observability is a monitoring bot operated by SolarWinds, used for synthetic monitoring of websites and services. robots.txt: No
- Spectate Spectate Spectate is an uptime monitoring bot operated by Spectate, used for healthchecks. robots.txt: No
- Splunk Splunk Splunk is a monitoring platform operated by Splunk, used by Enterprise organizations for synthetics monitoring. robots.txt: No
- Splunk Synthetics Splunk Synthetic monitoring tool with global agents running in AWS robots.txt: No
- Stape Scanner Stape Inc Stape Scanner is a monitoring bot operated by Stape Inc, used for evaluating tracking tag configurations. robots.txt: No
- StatusCake StatusCake StatusCake is a monitoring bot operated by StatusCake, used to monitor webpage availability and performance. robots.txt: No
- Swifteq Link Checker Swfiteq Ltd Swifteq Link Checker bot by Swifteq Ltd for monitoring and detecting broken links in Zendesk help centers. robots.txt: No
- Synthetic Bot DataDog Datadog Synthetic Bot, operated by DataDog, for monitoring applications and API endpoints via simulated user requests. robots.txt: No
- TermlyBot Termly TermlyBot is a web crawler operated by Termly for detecting and categorizing website cookies. robots.txt: No
- Thousand Eyes Cloud Agent Thousand Eyes ThousandEyes monitors network infrastructure, troubleshoots application delivery and maps Internet performance, all from a SaaS-based platform. robots.txt: No
- Trendiction Bot Trendiction S.A. Trendiction Bot is a monitoring bot operated by Trendiction S.A. robots.txt: No
- UpDownBot Updown UpDownBot is the HTTP monitoring probe for updown.io, operated by Updown, for monitoring website uptime and performance. robots.txt: No
- Uptime LLC Uptime Uptime LLC bot, operated by Uptime, for website availability and performance monitoring. robots.txt: No
- UptimeBot UptimeBot UptimeBot is an actionable website monitoring tool that works great with Slack. robots.txt: No
- uptimedoctor uptimedoctor UptimeDoctor is a website uptime monitoring service. robots.txt: No
- UptimeRobot UptimeRobot UptimeRobot is a free website uptime monitoring service that checks if your website is up or down at configurable intervals. robots.txt: No
- Uptimia Uptimia Uptimia is a website monitoring service operated by Uptimia, monitoring website performance and availability. robots.txt: No
- UptrendsBot Uptrends GmbH UptrendsBot is a monitoring bot operated by Uptrends GmbH for synthetic monitoring and load testing. robots.txt: No
- Watchbot WatchBot Watchbot is a monitoring bot operated by WatchBot, tracking website availability and SSL certificate expiration. robots.txt: No
- Watchful Watchful LLC Watchful is a monitoring system operated by Watchful LLC to check uptime on client websites. robots.txt: No
- webpagetest WebPageTest WebPageTest is a web performance testing tool operated by Catchpoint, used for measuring webpage performance from various locations and browsers. robots.txt: No
- WebsitePulseBot WebsitePulse WebsitePulseBot is an HTTP monitoring probe operated by WebsitePulse for their monitoring service. robots.txt: No
- WkndHealthCheckBot Wunderkind WkndHealthCheckBot is a monitoring bot operated by Wunderkind, used for performing health checks on clients' domains. robots.txt: No
- WormlyBot Wormly WormlyBot is an HTTP monitoring probe operated by Wormly for uptime monitoring. robots.txt: No
- WP Umbrella WP Umbrella WP Umbrella bot fetches data from WordPress sites for Umbrella plugin users, operated by WP Umbrella. robots.txt: No
- WPMU DEV Broken Link Checker WPMUDEV WPMU DEV Broken Link Checker is a bot operated by WPMUDEV that scans sites for broken links. robots.txt: No
- WPMUDEV Uptime Monitor 5.0 WPMUDEV WPMUDEV Uptime Monitor 5.0 is a website monitoring bot operated by WPMUDEV, checking site uptime and performance. robots.txt: No
Web archivers 3
Web archiving
- Arquivo Web Crawler Arquivo Arquivo Web Crawler is a web archiving bot operated by Arquivo, capturing the Portuguese web. robots.txt: Yes
- Internet Archive - Archive-It Archive-It Internet Archive's Archive-It bot preserves web pages for historical records. robots.txt: No
- Internet Archive Bot Internet Archive The Internet Archive bot crawls and preserves publicly accessible web pages for the Internet Archive's Wayback Machine historical record. robots.txt: Yes
Advertising bots 38
Ad verification
- AdsBot-Google Google Google AdsBot is a special-case crawler that monitors websites where Google Ads are served, checking landing page quality and ad compliance. robots.txt: No
- AdsTxtCrawler OneTag Limited AdsTxtCrawler by OneTag Limited crawls ads.txt files to verify onetag.com line presence. robots.txt: No
- Amazon AdBot Amazon Amazon AdBot is a crawler used by Amazon's advertising services to ensure advertiser landing pages comply with advertising policies. robots.txt: Yes
- Amazon Contxbot Amazon Amazon Contxbot is a crawler bot operated by Amazon for advertising purposes, visiting sites to determine content for relevant ads. robots.txt: No
- BigUpData Bot BigUpData BigUpData Bot, operated by BigUpData, automates product deals pipeline to advertisement. robots.txt: No
- Bing Ads Microsoft Bing Ads bot operated by Microsoft for ad quality control. robots.txt: No
- BrightEdge Bot BrightEdge Autopilot is an SEO marketing automation tool that includes features for internal linking and image optimization. We crawl customer sites so that we can determine the best links to use on the site and to find images that need to be optimized. robots.txt: No
- Channable Channable Channable is a tool used by some Cloudflare customers. The tool downloads data from the customers' systems for marketing automation and e-commerce automation. robots.txt: No
- Clickagy Clickagy The Clickagy Intelligence Bot is an ad verification bot for Clickagy. robots.txt: No
- ContextualBot Outcomes ContextualBot is a web indexer operated by Outcomes that analyzes webpage content to create contextual segments for targeting within advertising platforms. robots.txt: No
- CriteoBot Criteo CriteoBot is a web crawler operated by Criteo for serving relevant ads. robots.txt: No
- Google Adsense: Mediapartners-Google Google The Mediapartners-Google bot, operated by Google, monitors website content for AdSense. robots.txt: Yes
- Google Firebase Google Google Firebase is a crawler operated by Google for advertising purposes. robots.txt: No
- Google-Display-Ads-Bot Google Google-Display-Ads-Bot is a crawler operated by Google to verify sites when added to AdSense. robots.txt: Yes
- Grapeshot Grapeshot The Grapeshot bot, part of Oracle Advertising, crawles web pages for content analysis and classification. robots.txt: No
- Groovinads Groovinads Groovinads bot gathers e-commerce data for ad services, operated by Groovinads. robots.txt: No
- IAS crawler Integral Ad Science IAS crawler is a bot operated by Integral Ad Science for digital ad verification and content rating. robots.txt: No
- klaviyo Klaviyo Klaviyo is a B2C CRM bot operated by Klaviyo, used for advertising and marketing automation. robots.txt: No
- LeikiBot DoubleVerify LeikiBot, operated by DoubleVerify, crawls webpages for advertisers. robots.txt: No
- Mediavine Medatada Parser Mediavine Our App is called Grow and we allow publishers to enable bookmarking, social sharing, and searching on their sites. robots.txt: No
- Meta-ExternalAds Meta Meta-ExternalAds is a web crawler operated by Meta, used for improving advertising and business-related products and services. robots.txt: Yes
- MgidBot MGID MgidBot is a bot operated by MGID for detecting context categories of content for advertising recommendations. robots.txt: Yes
- Missinglettr Bot Missinglettr Missinglettr Bot crawls blog posts to turn them into social media campaigns. robots.txt: No
- Noorobot Noor Digital Agency AB Noorobot is an SEO tool operated by Noor Digital Agency AB that crawls websites to provide recommendations and identify SEO-impacting problems. robots.txt: No
- NostoCrawlerBot Nosto NostoCrawlerBot is a bot operated by Nosto for advertising purposes, used for crawling and accessing web pages. robots.txt: No
- Outbrain Outbrain Outbrain bot, operated by Outbrain, analyzes content for ad serving. robots.txt: No
- Proximic ComScore Proximic bot, operated by ComScore, crawls websites to determine ad suitability. robots.txt: Yes
- Quantcastbot Quantcast Quantcastbot is a web crawler used by Quantcast for advertisement quality assurance and to understand page content for Interest-Based Audiences. robots.txt: No
- Reelevant Reelevant Reelevant is an advertising bot operated by Reelevant, allowing customers to dynamically update content inside their emails by fetching images of their product at runtime. robots.txt: No
- Salesforce Salseforce Salesforce bot operated by Salesforce for advertising purposes with CRM and marketing capabilities. robots.txt: No
- SalesViewerBot SalesViewer GmbH SalesViewerBot is a web crawler operated by SalesViewer GmbH for advertising and sales intelligence purposes. robots.txt: No
- Slickstream Slickstream Slickstream is a SaaS bot operated by Slickstream, providing engagement features like site search and content recommendations. robots.txt: No
- SmartologyBot Smartology SmartologyBot is a bot operated by Smartology, used for generating semantic vectors from domain pages to serve semantically-relevant ads. robots.txt: No
- Taboola Taboola Taboola bot, operated by Taboola, visits websites to audit content and gather metadata for advertiser campaigns. robots.txt: No
- TTD Content The Trade Desk The Trade Desk Content bot, operated by The Trade Desk, classifies webpage content for ad placement. robots.txt: No
- Yahoo Ad Monitoring Yahoo Yahoo Ad Monitoring bot, operated by Yahoo, monitors webpages for Yahoo ads. robots.txt: No
- Yahoo Japan Yahoo Japan Corporation Yahoo Japan Advertising Bot operated by Yahoo Japan Corporation for advertising purposes. robots.txt: No
- Yext Inc Yext The Yext Crawler is a bot operated by Yext, used for retrieving data from websites for Yext customers. robots.txt: No
Security scanners 31
Security scanning
- Black Duck Fast Dynamic Black Duck Black Duck Fast Dynamic is a security scanning bot operated by Black Duck Software, performing Fast Dynamic DAST security scanning. robots.txt: No
- Blockaid Blockaid Scanning the internet to find malicious sites that scam crypto users into draining their wallets robots.txt: No
- Bushbaby Cloudflare Bushbaby is a security bot operated by Cloudflare to manage and renew SSL certificates. robots.txt: No
- Cert Chief Chief Tools Cert Chief is a security bot operated by Chief Tools that monitors HTTPS domains for certificate issues and configuration problems. robots.txt: Yes
- Cloudflare Security Scanner Cloudflare Cloudflare Radar URL Scanner scans and analyzes URLs for security threats, phishing, and malware on behalf of Cloudflare's security services. robots.txt: No
- Cookie Hub CookieHub Cookie Hub bot detects cookies set by websites for CookieHub clients and verifies user consent. robots.txt: No
- cookie maestro Cookie Maestro Within the GDPR legislation it is mandatory to ask a visitor for permission before placing so-called marketing or tracking cookies. Many websites contain a cookie notice, but what is not clear to everyone is that those cookies may not be placed before the visitor has given explicit permission. Co... robots.txt: Yes
- Cookiebot Cybot A/S Cookiebot scans websites for cookies and trackers, operated by Cybot A/S for security purposes. robots.txt: No
- CookieYesBot CookieYes CookieYesbot is a security and compliance scanner operated by CookieYes to verify cookie consent banner implementations and compliance configurations on websites. robots.txt: No
- Detectify Detectify Detectify is a security bot operated by Detectify that analyzes web application security levels. robots.txt: No
- Drata Autopilot Drata The Drata Autopilot bot continuously monitors the security posture of customer domains. robots.txt: No
- Google Inspection Tool Google Google Inspection Tool is a crawler used by Google for Search testing tools, operated by Google. robots.txt: No
- Google Safety Google Google Safety bot crawls for abuse-specific purposes like malware discovery on Google properties. robots.txt: No
- Google Site Verification Google Google Site Verification bot verifies site ownership for Google Search Console, operated by Google for security purposes. robots.txt: No
- Google Trust Services (DCV Check) Google Google Trust Services bot for DCV Check, operated by Google, for certificate validation. robots.txt: No
- Jumio Coinbase Digital ID Verification and Scanning Tool used by Coinbase team. robots.txt: No
- LegalMonster Legal Monster ApS Monitor customers' sites for compliance with cookie- and other privacy-related legislation robots.txt: No
- Let's Encrypt Let's Encrypt Let's Encrypt bot for domain control validation, operated by Let's Encrypt. robots.txt: No
- Netcraft Netcraft Netcraft bot analyzes web server technology stacks for their Web Server Survey. robots.txt: No
- Project Honeypot Unspam Technologies, Inc We use Project Honeypot for IP info. robots.txt: No
- ProjectShield Url Check Google ProjectShield Url Check bot operated by Google for security purposes, providing DDoS attack protection. robots.txt: No
- Qualys Qualys Qualys bot, operated by Qualys, crawls and tests web applications for vulnerabilities. robots.txt: No
- Reward Gateway Reward Gateway With the iboss Cloud Platform, each customer gets dedicated source cloud IP Addresses which are associated with the organization. Because of this, any data traversing the global cloud containerized gateways in the Platform will have a uniquely associated IP Address that can be mapped to the organ... robots.txt: No
- Sansec Security Monitor Sansec Security Monitor Sansec Security Monitor is a web crawler that enhances e-commerce security by monitoring stores for malicious code, preventing data breaches and digital skimming. robots.txt: Yes
- SecurityHeaders Security Headers Provides a free security scanning service at https://securityheaders.com robots.txt: No
- Sentry Sentry Sentry is a security bot operated by Sentry that monitors webpages for availability and performance issues. robots.txt: No
- SiteLock SiteLock SiteLock is a security bot operated by SiteLock that scans websites for malware and malicious code. robots.txt: No
- SmarshBot https://central.smarsh.com/s/ SmarshBot is a security bot operated by Smarsh, providing archiving solutions for compliance purposes. robots.txt: No
- Splunk Attack Analyzer Splunk Splunk Attack Analyzer is a security bot operated by Splunk, analyzing URLs for malicious content using a headless Chrome browser. robots.txt: No
- Sucuri Sucuri The Sucuri bot is a security bot operated by Sucuri, crawling websites to scan for malware, security risks, and blacklisting status. robots.txt: No
- WebTotemBot WebTotem WebTotemBot is a security monitoring and defense bot operated by WebTotem, protecting web applications and their data. robots.txt: No
Other bots 128
Web crawling
- 2checkout 2checkout 2Checkout Payment Platform's webhooks integration bot, operated by Verifone, for notifying websites about payment events. robots.txt: No
- Accessible Web Bot Accessible Web Accessible Web Bot crawls websites to discover pages and monitor for accessibility violations. robots.txt: No
- Adyen Adyen Adyen bot operated by Adyen, sends HTTP requests for payment-related events. robots.txt: No
- AfterShip Magento 2 Connector AfterShip AfterShip Magento 2 Connector bot operated by AfterShip for order tracking and delivery updates. robots.txt: No
- Akismet Akismet Akismet is a spam filtering service operated by Akismet, protecting websites from comment, form, and text spam. robots.txt: No
- Apple App Site Association Apple The Apple App Site Association bot, operated by Apple, supports Universal Links in native iOS apps by requesting specific paths for given hostnames. robots.txt: No
- Artemis Web Reader capjamesg Artemis is a calm, independently-run, free web reader. The reader often runs into Cloudflare access blocks which makes it hard to follow many sites. robots.txt: No
- Authory Authory The Authory bot visits websites to back up articles on behalf of journalists and other writers who use the service. robots.txt: No
- Bazqux Bazqux Bazqux Fetcher is a user-initiated bot operated by Bazqux that collects and refreshes RSS/Atom feeds and comments for users subscribed to BazQux Reader. robots.txt: No
- BestChange Bot BestChange BestChange Bot operated by BestChange, downloads exchange rate information from 600 websites every 5 seconds. robots.txt: No
- Bibliotheque Nacional de France Crawler Bibliothèque nationale de France The Bibliotheque Nacional de France Crawler is a bot operated by the Bibliothèque nationale de France, tasked with capturing and preserving the national documentary heritage. robots.txt: No
- Bitbucket Atlassian Bitbucket bot operated by Atlassian for CI/CD and webhooks. robots.txt: No
- Bling ERP Bling Bling ERP is an online ERP system operated by Bling that integrates with ecommerce platforms. robots.txt: No
- Blogtrottr Blogtrottr Blogtrottr is a service operated by Blogtrottr that delivers updates from news, feeds, and blogs to users' email inboxes. robots.txt: No
- BoardGamePrices Bot KP Software Consult ApS BoardGamePrices Bot is a price comparison bot operated by KP Software Consult ApS, crawling store pages for board game prices. robots.txt: No
- Buttondown RSS-Feed-Parser Buttondown Buttondown RSS-Feed-Parser bot fetches RSS feeds for Buttondown newsletters. robots.txt: No
- Capital One Bot Capital One Capital One Bot crawls dealer websites for usage information. robots.txt: No
- Channel3Bot Channel3 Channel3Bot is a web crawler operated by Channel3 that indexes product pages to drive traffic back to those websites. robots.txt: Yes
- ChargeBee ChargeBee ChargeBee bot, operated by ChargeBee, for payment event notifications via webhooks. robots.txt: No
- Cloudflare CSUP Cloudflare Cloudflare CSUP is a bot used by Cloudflare's customer support for diagnostic purposes. It is not a general web crawler and is used to investigate technical issues with customer websites. robots.txt: No
- Cloudflare Validator Cloudflare Cloudflare Validator is a bot operated by Cloudflare that verifies IPs for Cloudflare Bots Directory. robots.txt: No
- Coinbase Webhooks Coinbase Coinbase Webhooks are automated messages sent from the Coinbase platform to a user's server, used for notifying users about events such as receiving crypto payments. robots.txt: No
- ConvermaxBot Convermax Corp. ConvermaxBot is a site search indexer operated by Convermax Corp. robots.txt: No
- Critical CSS Bot Critical CSS The Critical CSS Bot is operated by Critical CSS, a service for website optimizations that generates critical CSS to improve website performance. robots.txt: No
- cron-job.org cron-job.org cron-job.org is a service for scheduled execution of websites and scripts, operated by cron-job.org. robots.txt: No
- EasyBill Import Manager easybill.de EasyBill Import Manager is a tool operated by easybill.de that synchronizes order data from external systems to EasyBill. robots.txt: No
- EasyCron EasyCron EasyCron is an online cron job service operated by EasyCron, allowing users to schedule HTTP requests. robots.txt: No
- eMoney Advisor eMoney Advisor eMoney Advisor bot collects financial data for planning and analysis, operated by eMoney Advisor. robots.txt: No
- Epivoz Crawler Techmeme Epivoz Crawler is a news aggregator bot operated by Techmeme, used to crawl news and blog articles to generate short summaries for page previews of attributed links. robots.txt: No
- Exodus Exodus Exodus bot, operated by Exodus, manages cryptocurrencies like Bitcoin, Ethereum, and Ripple. robots.txt: No
- FedReporter Bot for FFIEC Fed Reporter, Inc. FedReporter Bot for FFIEC, operated by Fed Reporter, Inc., downloads public banking institution data. robots.txt: Yes
- Feedbin Feedbin Feedbin is a web-based RSS reader operated by Feedbin. robots.txt: No
- Feeder Really Simple AB Feeder is a cloud-based RSS reader operated by Really Simple AB, used for tracking online sources and bundling content into an easy-to-digest reading experience. robots.txt: Yes
- Feedly Feedly Feedly RSS fetcher service operated by Feedly, collecting and refreshing user-initiated feeds. robots.txt: No
- FeedWind Crawler Mikle FeedWind Crawler by Mikle accesses feed sources to update the feed widget. robots.txt: No
- FlipboardRSS Flipboard FlipboardRSS bot operated by Flipboard, uses visitor's RSS feed to discover articles and generate summaries. robots.txt: No
- Freshping freshping Freshping is a website monitoring bot operated by freshping that checks if a website is online and sends alerts when it's down. robots.txt: No
- Google API Google Google API bot for delivering push notification messages. robots.txt: No
- Google Feed Fetcher Google Google Feed Fetcher is a bot operated by Google for fetching RSS feeds. robots.txt: No
- Google Publisher Center Google Google Publisher Center bot operated by Google, fetches and processes publisher-supplied feeds for Google News landing pages. robots.txt: No
- Google Read Aloud Google Google Read Aloud is a service by Google that enables reading web pages using text-to-speech (TTS). robots.txt: No
- GoPay GoPay.cz GoPay payment gateway http notification service operated by GoPay.cz robots.txt: No
- Hatena Hatena Hatena's service bot for automatic web page visits to gather information. robots.txt: Yes
- HelloWork HelloWork Group HelloWork bot aggregates job listings for the HelloWork platform, operated by HelloWork Group. robots.txt: No
- HIFIBot HIFI HIFIBot is a bot operated by HIFI, a financial services company for musicians and professional creators, used for automating retrieval and processing of royalty earnings statements. robots.txt: No
- HSTS preload bot Chromium The HSTS preload bot, operated by Chromium, checks that links can be preloaded for Chromium. robots.txt: No
- HubSpot Crawler Hubspot HubSpot Crawler is a bot operated by HubSpot for marketing, sales, and customer service platform. robots.txt: No
- HubSpot Feed Fetcher HubSpot HubSpot Feed Fetcher bot operated by HubSpot for pulling images when posting to LinkedIn. robots.txt: No
- IFTTT RSS Feed Service IFTTT RSS Feed Servuce IFTTT RSS Feed Service fetches RSS feeds to power user-configured automations. robots.txt: No
- IndeedJobBot Indeed Indeed's job crawling bot that crawls job and job related information robots.txt: Yes
- Innguma Fetcher Innguma Innguma Fetcher is a user-controlled feed retriever operated by Innguma, collecting and periodically refreshing user-initiated RSS or Atom feeds. robots.txt: No
- Innologica Innologica Inoreader is an RSS feed reader operated by Innologica, used for content aggregation and discovery. robots.txt: No
- Instapaper Instant Paper Instapaper is an app operated by Instant Paper that lets people save articles to read later. robots.txt: No
- Integromat Make Over 50 active Cloudflare users are using Integromat to automate their workflow. robots.txt: No
- kb.dk_bot Netarkivet The kb.dk_bot is operated by Netarkivet, a part of the Royal Danish Library, to collect and preserve the Danish Internet for research purposes. robots.txt: No
- Library Of Congress Web Archiving United States Library of Congress The Library of Congress Web Archive is a bot operated by the United States Library of Congress that manages, preserves, and provides access to archived web content. robots.txt: No
- MagnetmeBot Magnet.me MagnetmeBot is a respectful web crawler operated by Magnet.me to discover and update career-related content. robots.txt: Yes
- MainWP Direct Support / Website Managed MainWP bot for managing websites through the MainWP Control Dashboard. robots.txt: No
- Make Platform Make Make is a no-code platform to help with task and workflow automation. robots.txt: No
- ManageWP ManageWP ManageWP is a web management tool operated by ManageWP, used to monitor and maintain multiple WordPress websites from a single dashboard. robots.txt: No
- MediaMonitoringBot MediaMonitoringBot MediaMonitoringBot is a web crawler operated by MediaMonitoringBot that indexes news and media publishers' websites. robots.txt: Yes
- Mediatoolkitbot Determ Mediatoolkitbot is a media monitoring tool operated by Determ that crawls the internet for user-specified phrases. robots.txt: Yes
- Metorik Metorik Metorik is an analytics and email automation service bot operated by Metorik, used by eCommerce businesses to sync data from customer sites via API requests. robots.txt: No
- MirrorWebCrawler MirrorWeb Ltd MirrorWebCrawler is a web archiving bot operated by MirrorWeb Ltd, providing archival solutions for financial and public sectors. robots.txt: No
- Modular DS Uniqoders Technologies SL Modular DS is a service operated by Uniqoders Technologies SL to manage WordPress websites via the WP JSON API. robots.txt: No
- Mollie Bot Mollie B.V. Mollie Bot operated by Mollie B.V., a payment service provider, uses webhooks to notify merchants about payment updates. robots.txt: No
- MonitoRSS MonitoRSS MonitoRSS is a bot operated by MonitoRSS that polls RSS feeds of Cloudflare-protected sites to deliver news articles to Discord servers. robots.txt: Yes
- MotoMinerBot Motominer MotoMinerBot is a web crawling bot operated by Motominer, used for indexing vehicle detail pages. robots.txt: No
- Muck Rack Scraper Muck Rack Muck Rack uses many approaches for source discovery such as RSS feeds, sitemaps, and other structured formats robots.txt: No
- New York Times Newsgathering The New York Times New York Times Newsgathering bot collects public, non-copyright data for newsroom use. robots.txt: No
- NewsBlur NewsBlur NewsBlur RSS News fetcher operated by NewsBlur robots.txt: No
- Noibu JS Beautifier Noibu Noibu JS Beautifier is a microservice operated by Noibu that downloads and formats JavaScript files for error solving. robots.txt: No
- Nooshub Nooshub Nooshub is an RSS Reader bot operated by Nooshub that fetches RSS/Atom feeds. robots.txt: No
- Omnisend Omnisend Omnisend bot checks integration in WooCommerce shops and Shopify, operated by Omnisend. robots.txt: No
- OpenRSS OpenRSS OpenRSS is an RSS feed provider and fetcher operated by OpenRSS, offering a healthier alternative to algorithmic feeds. robots.txt: No
- OurFamilyWizard OurFamilyWizard OurFamilyWizard bot syncs external iCalendars into Cozi calendars. robots.txt: No
- Overcast Overcast Radio Overcast is a podcast player application operated by Overcast Radio, fetching RSS feeds and audio files from podcast hosting servers. robots.txt: No
- PayPal PayPal The PayPal webhooks is part of Paypal's Instant Payment Notification message service, automatically notifying merchants of events related to Paypal transactions. robots.txt: No
- PinterestBot Pinterest PinterestBot is Pinterest's web crawler that indexes content and scrapes Pin details to drive traffic and ensure accuracy. robots.txt: Yes
- Pocket Casts Feed Parser Pocket Casts Fetches podcast feeds, for playback in the Pocket Casts apps robots.txt: No
- Polar Polar Polar is a webhook service operated by Polar that sends HTTP requests to inform web servers about billing events. robots.txt: No
- PrintFriendly PrintFriendly.com Extract Content to Show Print Friendly version. Publishers typically embed our button - https://www.printfriendly.com/button - so that their visitors can view a Print Friendly Page and/or create a PDF robots.txt: No
- Pro Sitemaps PRO Sitemaps Pro Sitemaps is an online sitemap generator service operated by PRO Sitemaps, used for creating and maintaining sitemaps for websites. robots.txt: No
- Protopage Protopage Ltd Protopage is an RSS reader bot operated by Protopage Ltd, indexing RSS news headlines. robots.txt: No
- PWABuilder Microsoft PWABuilder is a free, open-source tool from Microsoft that helps developers build progressive web apps. robots.txt: Yes
- Rakuten Image extraction bot Rakuten Rakuten Image extraction bot, operated by Rakuten, retrieves product images from partner websites. robots.txt: No
- Razorpay Webhook Razorpay Razorpay Webhook bot operated by Razorpay, sends webhooks to ecommerce sites. robots.txt: No
- Readable Added Bytes Ltd Readable is a text analysis tool operated by Added Bytes Ltd, focused on clarity and plain language. robots.txt: No
- Retool Retool Retool platform user agent robots.txt: No
- RSS API RSS API (by Tibush GmbH) RSS API is a bot operated by Tibush GmbH that periodically requests and parses RSS feeds for customers to monitor changes. robots.txt: No
- rss2tg_bot Yellow Rubber Duck Consulting rss2tg_bot is a notification RSS/Atom bot for Telegram operated by Yellow Rubber Duck Consulting. robots.txt: No
- Scour RSS Bot Scour Scour RSS Bot is a personalized RSS feed reader operated by Scour, using AI/ML to surface content matching user interests. robots.txt: No
- SendGrid SendGrid SendGrid bot operated by SendGrid, sends email event data to customer APIs. robots.txt: No
- ServerHunterSpider Server Hunter ServerHunterSpider is a bot operated by Server Hunter that indexes hosting plans' price, specifications, and stock. robots.txt: Yes
- Shopify-Captain-Hook Shopify Shopify-Captain-Hook is a system used by Shopify to deliver webhooks. It sends automated messages to a user's server when specific events occur within their Shopify store. robots.txt: No
- SilktideBot Silktide Website accessibility, SEO, and content quality scanner robots.txt: No
- Skroutz ImageBot Skroutz S.A. Skroutz ImageBot to fetch the individual product images. robots.txt: No
- SkroutzBot Skroutz S.A. Skroutz uses SkroutzBot web crawler to download XML feeds. robots.txt: No
- Slackbot Salesforce Slackbot fetches URLs shared in Slack channels to generate link previews and unfurls for team communication. robots.txt: No
- Snipcart Snipcart Snipcart is an e-commerce solution for developers operated by Snipcart. robots.txt: No
- Spark Shipping Spark Shipping Spark Shipping is an eCommerce automation software for retailers running WooCommerce, automating product data integration, inventory synchronization, order sending, and tracking updates. robots.txt: No
- SparkpostBot Bird SparkpostBot is a bot operated by Bird for automating email transactions on web server events. robots.txt: No
- Statabot StataCorp LLC Statabot is a bot operated by StataCorp LLC that searches for stata.toc files and indexes their contents. robots.txt: No
- StatistikAustria Statistik Bot StatistikAustria bot operated by Statistik Bot to collect product prices for Austria's official consumer price index. robots.txt: No
- StatsDroneBot StatsDrone StatsDroneBot is a statistics scraping and aggregating tool for iGaming affiliates, operated by StatsDrone. robots.txt: No
- Stripe Stripe Stripe bot, operated by Stripe, pushes real-time event data to customers' webhook endpoints. robots.txt: No
- Stripebot Stripe Stripebot is a web crawler operated by Stripe, collecting data from user websites for services and financial regulation compliance. robots.txt: No
- Svix Webhooks Svix Inc. Svix Webhooks is a scalable webhook platform operated by Svix Inc., featuring automatic retries, signature verification, and deep observability. robots.txt: No
- Termly Termly Termly bot scanners for site compliance robots.txt: No
- Trustly Trustly Group AB Trustly is a payment bot operated by Trustly Group AB, sending notifications to merchants about payment statuses. robots.txt: No
- TurnitinBot Turnitin Turnitin.com offers various services to the educational community. Most prominently, we provide a widely used and effective plagiarism detection service. Part of the plagiarism prevention service relies on comparing student papers to content found on the Internet. Since we do not know ahead of ti... robots.txt: No
- Twilio Proxy Twilio, Inc. Twilio Proxy bot operated by Twilio, Inc. for sending and receiving SMSes, making calls, and managing customer engagement. robots.txt: No
- upday upday GmbH & Co. KG upday is a news aggregator app bot operated by upday GmbH & Co. KG, crawling news sources to collect and index articles for user recommendations. robots.txt: Yes
- VaultPress Automattic VaultPress is a WordPress backup service by Automattic, offering daily and real-time backups. robots.txt: No
- videootv Bot Digital Green Crawler to extract the newest articles in the publisher's website (via feed or parsing html) to make a carrousel with images, links and text for our native ads module in order to improve recirculation in the publisher's web. Only crawls our publisher's webpages. robots.txt: Yes
- WebSpiderMount AspenTechLabs Inc Job wrapping data processor handling jobs distribution from employer websites to multiple endpoints, like job boards, advertisement platforms, job alerts etc. robots.txt: No
- WMF Citoid Wikimedia Foundation WMF Citoid is a Wikimedia Foundation service that generates citations from URLs, DOIs, and ISBNs. robots.txt: No
- WMF Zotero Translation Server Wikimedia Foundation WMF Zotero Translation Server, operated by Wikimedia Foundation, extracts citation data using Zotero translators. robots.txt: No
- Wordpress Namecheap namecheap This is the major egress IP for our containerised WordPress platform so it is likely to be many flavours of WordPress and the potential to be any domain. robots.txt: No
- Worldline Bot Worldline The Worldline Bot, operated by Worldline, handles notifications and callbacks related to payment processing. robots.txt: No
- WorldPay WorldPay Payment confirmation callbacks to ecommerce backends robots.txt: No
- WOVN Crawler Wovn Technologies, Inc. We offer WOVN.io, a service for localization websites. We run a crawler to get source language of our clients' websites robots.txt: No
- WP Time Capsule WP Time Capsule Our plugin WPTimeCapsule is installed in more 30000 WordPress sites. When our backup servers send requests to trigger the backup on the WordPress sites, it is being blocked robots.txt: No
- WP Umbrella Bot WP Umbrella WP Umbrella Bot is a tool for managing and maintaining WordPress websites, operated by WP Umbrella. robots.txt: No
- XY Archive Compliance Bot XY Archive Compliance The XY Archive Compliance Bot is a website archiver operated by XY Archive Compliance for customers with archive compliance requirements. robots.txt: No
- Yokoy Group Webhooks Yokoy Group AG Yokoy Group Webhooks is a bot operated by Yokoy Group AG, generating requests to book expenses or invoices to customer's ERP system. robots.txt: No
- Zapier Zapier Inc. Zapier is an automation tool operated by Zapier Inc. that moves information between web apps automatically. robots.txt: No
- Zendesk Webhook Zendesk Zendesk Webhook bot for development of Zendesk ticketing system and apps. robots.txt: No
How this catalogue is built
Profiles are prepared with AI assistance from what operators publish: documentation pages, help articles and bot info pages. Where a source exists, the profile links to it, and anything the operator doesn't document is marked as unknown.
Waitlist open
Know which of these bots visit you
Can AI See It shows which bots from this catalogue hit your site, which are fake and what each platform sends back.
Social media bots 35
Link preview generation