NimbleSEO crawlers & fetchers
Every user agent this product sends over the wire, what it fetches, and how to allow or block it.
NimbleSEO fetchers work for our customers on their own sites and watched pages — we do not run a web-scale discovery crawler. The site-audit crawler honors robots.txt by default and every fetcher identifies itself with one of the product tokens below, so you can allow or block each one individually in your robots.txt.
Mozilla/5.0 (compatible; NimbleSEO-SiteCrawler/1.0; +https://nimble-seo.com/bot)
Site-audit crawler. Crawls sites that a NimbleSEO customer has connected to their own workspace to find technical SEO/GEO issues.
Crawl behavior: Honors robots.txt (RFC 9309) by default, with bounded per-origin rate and page caps.
NimbleSEO-AuthorBot/1.0 (+https://nimble-seo.com/bot)
Single-page fetches to read public author/byline information for E-E-A-T analysis. User-initiated; it does not crawl beyond the requested page.
Crawl behavior: On-demand single-URL fetch triggered by a user action, not a recurring crawl.
Mozilla/5.0 (compatible; NimbleSeoAuditBot/1.0; +https://nimble-seo.com/bot)
Single-site audit fetches made through the public audit API on behalf of the requesting user.
Crawl behavior: On-demand, user-initiated; bounded to the audited site.
Mozilla/5.0 NimbleSEO-DiffWatcher/1.0
Checks specific pages a customer chose to watch for content changes (competitor page diffs).
Crawl behavior: Fetches only explicitly configured URLs at a low cadence.
NimbleSEO-KG/1.0
Occasional lookups against public knowledge-graph endpoints during entity sync.
Crawl behavior: API-endpoint requests, not website crawling.
To block the site-audit crawler for your whole site, add this to your robots.txt:
User-agent: NimbleSEO-SiteCrawler Disallow: /
The same pattern works for any token above. If you believe a NimbleSEO fetcher is misbehaving, email security@nimble-seo.com with the timestamp and your access-log line — we investigate every report.