About IndexIntelligenceBot

Who runs this crawler

It is operated by Cworc GmbH, Hubertusweg 47, 14552 Michendorf, Germany, as part of Index Intelligence — a service that helps site owners understand why their pages are or are not indexed by Google.

It identifies itself with the user agent IndexIntelligenceBot/1.0 (+https://whyindexed.me/bot).

Why the crawler visited your site

Someone with access to your domain connected it to Index Intelligence and started a scan. The crawler fetches your robots.txt, your sitemaps, and a bounded set of pages to run technical SEO and indexability checks.

It only reads publicly reachable URLs with GET requests. It never submits forms, signs in, comments, or writes anything to your site.

How the crawler behaves

It obeys robots.txt — both rules targeting IndexIntelligenceBot and rules targeting all crawlers (*).

It sends one request at a time per host, with roughly a 0.7-second pause between page requests.

Each page fetch is capped at about 3 MB and 15 seconds; sitemap fetches are capped at 20 seconds. It follows only a few redirects before giving up.

It requests HTML pages and sitemap files only, runs only when a site owner asks for a scan, and does not download assets in bulk.

Verifying the crawler

Genuine requests carry the exact user agent IndexIntelligenceBot/1.0 (+https://whyindexed.me/bot), which links back to this page.

It runs from shared, dynamic cloud IP addresses, so we publish no IP allowlist — verify by matching the user agent instead.

How to block the crawler

Add the following to your site's robots.txt. IndexIntelligenceBot re-reads robots.txt on its next visit and stops crawling.

User-agent: IndexIntelligenceBot
Disallow: /

Blocking this crawler does not affect how Google indexes your site. Index Intelligence reads your Google Search Console data separately, through Google's authorised API — not by crawling. Blocking only disables the technical SEO page checks.

Contact

Questions about the crawler? Email remzi.aru@shlau.com.