Configured crawler and fetcher headers, what they request, and which paths check robots.txt.
This is an inventory of headers configured in our application, not a record of observed traffic or a claim that every feature is enabled. Requests serve connected-site tools, selected pages, configured feeds and provider APIs. Robots handling differs by path, as described below. A user-agent string alone does not authenticate a request.
Mozilla/5.0 (compatible; NimbleSEO-SiteCrawler/1.0; +https://nimbleseo.ai/bot)
Connected-site technical crawling. The listed header is the default; job configuration can supply another user agent.
Crawl behavior: Checks robots.txt by default before page fetches, with bounded page and origin limits. Job configuration can disable robots checking.
NimbleSEO-AuthorBot/1.0 (+https://nimbleseo.app)
NimbleSEO-AuthorBot/1.0 (+https://nimbleseo.ai/bot)
Public author/byline extraction and competitor author observations.
Crawl behavior: The author extractor checks Disallow prefixes for matching agents and *, and proceeds if robots retrieval fails. Competitor author observations use the shared robots parser and stop on network/server errors.
NimbleSEOBot/1.0 (+https://nimbleseo.app)
Customer-selected competitor pages watched for content changes.
Crawl behavior: Checks robots.txt before the selected page; network/server errors or an unreadable policy prevent the page fetch.
Mozilla/5.0 (compatible; NimbleSeoAuditBot/1.0; +https://nimbleseo.ai/bot)
One requested page for public API and interactive audits.
Crawl behavior: No robots.txt permission check in this fetch path.
NimbleSEO-Onboarding/1.0 (+analyze-my-site)
Reads the requested homepage for onboarding or the public homepage checker.
Crawl behavior: No robots.txt permission check in this fetch path.
NimbleSEO-CitationVerifier/2.0
NimbleSEO-CitationVerifier/1.0 (+https://nimbleseo.ai/bot)
Checks cited URLs for article generation and citation review.
Crawl behavior: No robots.txt permission check in this fetch path.
Mozilla/5.0 (compatible; NimbleSEO-ChangeMonitor/1.0; +https://nimbleseo.ai/bot)
Fetches configured pages when their stored monitoring schedule is due.
Crawl behavior: No robots.txt permission check in this fetch path.
Mozilla/5.0 (compatible; NimbleSEO-AuditMonitor/1.0; +https://nimbleseo.ai/bot)
Fetches configured pages for scheduled audit observations.
Crawl behavior: No robots.txt permission check in this fetch path.
NimbleSEO-Bot/1.0
NimbleSEO-Bot/1.0 (Sitemap Scanner)
NimbleSEO-Bot/1.0 (LLMs.txt Generator)
Mozilla/5.0 (compatible; NimbleSEO-Bot/1.0)
Sitemap scanning, llms.txt generation, content-gap reads, robots inspection, structured-data checks and the synchronous site audit.
Crawl behavior: The synchronous site audit checks robots.txt before page fetches and stops on network/server errors or truncated policy. The other listed tool paths do not check robots permission before fetching; reading robots.txt for inspection is not an enforcement check.
NimbleSEO-RedirectBot/1.0
Checks requested redirect targets and chains.
Crawl behavior: No robots.txt permission check in this fetch path.
NimbleSEO-SitemapBot/1.0
Reads an existing sitemap for the sitemap-generation tool.
Crawl behavior: No robots.txt permission check in this fetch path.
Mozilla/5.0 (compatible; NimbleSEO-Crawler/1.0)
Fetches site pages for the internal link-map tool.
Crawl behavior: Checks Disallow prefixes for wildcard and Googlebot groups; it does not match its own NimbleSEO-Crawler token. This restricted parser proceeds without restrictions if robots retrieval fails.
NimbleSEO-data-widget/1.0 (https://nimbleseo.ai)
NimbleSEO-data-widget/1.0
Mozilla/5.0 (compatible; NimbleSEO-data-widget/1.0; +https://nimbleseo.ai)
Reads Wikipedia summaries and selected competitor pricing pages for data widgets.
Crawl behavior: Pricing-page reads check a restricted Disallow-prefix policy; robots retrieval failures permit the read. Wikipedia API requests do not use that check.
NimbleSEO-AlgoFeedBot/1.0 (+security@nimbleseo.ai)
Reads configured search-update feeds, industry reports and patent records.
Crawl behavior: Provider/API request, not a website discovery crawl. No robots.txt permission check in this fetch path.
NimbleSEO/1.0 (community-intelligence; contact: support@nimbleseo.com)
Mozilla/5.0 (compatible; NimbleSEO/1.0)
NimbleSEO/2.0 (+https://nimbleseo.ai/bot; citation-structure)
Content summarization, competitor content reads, citation-structure comparisons and community API reads.
Crawl behavior: No robots.txt permission check in this fetch path.
NimbleSeo-KGResolver/1.0
Queries a knowledge-graph API to look up entities.
Crawl behavior: Provider/API request, not a website discovery crawl. No robots.txt permission check in this fetch path.
Mozilla/5.0 (compatible; GEO-Visibility-Bot/1.0)
Queries the Phind search endpoint when that adapter is available and authorized.
Crawl behavior: Provider/API request, not a website discovery crawl. No robots.txt permission check in this fetch path.
Mozilla/5.0 (compatible; NimbleSeoGapFinder/1.0)
Reads a competitor sitemap and a bounded set of child sitemaps.
Crawl behavior: No robots.txt permission check in this fetch path.
Mozilla/5.0 (compatible; NimbleSeoAgent/1.0)
Reads a requested public URL for an agent web-fetch tool.
Crawl behavior: No robots.txt permission check in this fetch path.
Mozilla/5.0 (compatible; NimbleSeoBot/1.0; +https://nimbleseo.ai/bot)
Reads retained top-ranking page URLs for structural observations.
Crawl behavior: No robots.txt permission check in this fetch path.
Mozilla/5.0 (compatible; NimbleSEO-Autocomplete/1.0; +https://nimbleseo.ai/bot)
Queries the Google suggestion endpoint for keyword suggestions.
Crawl behavior: Provider/API request, not a website discovery crawl. No robots.txt permission check in this fetch path.
NimbleSEO-LinkChecker/1.0
Checks links supplied through the editor.
Crawl behavior: No robots.txt permission check in this fetch path.
NimbleSEO-Google-Indexing-Eligibility/1.0
Reads a requested page to inspect structured data for Indexing API eligibility.
Crawl behavior: No robots.txt permission check in this fetch path.
InventoryScanner/1.0
Reads connected-site sitemaps and pages for the content inventory.
Crawl behavior: No robots.txt permission check in this fetch path.
Mozilla/5.0 ai-seo-alt-bot
Reads requested image data for alternative-text tooling.
Crawl behavior: No robots.txt permission check in this fetch path.
Mozilla/5.0 ai-seo-bot
Reads requested pages and images for bulk image tooling.
Crawl behavior: No robots.txt permission check in this fetch path.
AISEO-PolicyVerifier/1.0 (+https://example.com/ai-crawl)
Fetches published policy URLs selected for deployment verification.
Crawl behavior: No robots.txt permission check in this fetch path.
AISEO-PromptUniverseBot/1.0 (+https://example.com/bot)
Reads supplied competitor pages to derive editable prompt candidates.
Crawl behavior: No robots.txt permission check in this fetch path.
SUS-Knowledge/1.0 (+https://example.com/sus-bot)
Reads configured knowledge-source feeds.
Crawl behavior: No robots.txt permission check in this fetch path.
nimble-seo/0.1
nimble-seo/1.0
Newsletter feed import (1.0) and Stack Exchange API reads (0.1).
Crawl behavior: No robots.txt permission check in this fetch path.
nimble-seo-geoip/1.0
Queries the configured IP-location service.
Crawl behavior: Provider/API request, not a website discovery crawl. No robots.txt permission check in this fetch path.
NimbleSEO-Webhooks/1.0
Delivers event payloads to configured customer webhook URLs.
Crawl behavior: Outbound delivery, not a crawler; robots.txt is not checked.
nimbleseo-app-cron/1
Dispatches internal scheduled application tasks.
Crawl behavior: Internal requests, not a third-party website crawler; robots.txt is not checked.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
Reads public YouTube caption endpoints using the browser-style header shown here.
Crawl behavior: This path does not identify itself with a NimbleSEO product token and does not check robots.txt.
Configured crawl jobs can use a custom header. Browser automation uses browser headers, and authenticated provider SDKs or requests without an explicit header can use their own defaults. Reddit uses an operator-configured header. Site-audit simulations can send configured third-party crawler strings; those requests are NimbleSEO simulations, not visits from the named crawler. Provider-mediated fetches can identify with the provider’s header instead of ours.
Some legacy headers contain example.com or older domain comments. They are reproduced literally here for log matching; use the security contact on this page to report traffic.
For site-audit jobs using the default token with robots checking enabled, this rule disallows page crawling:
User-agent: NimbleSEO-SiteCrawler Disallow: /
Robots.txt is a voluntary crawl policy, not an access-control mechanism. Paths described above as having no permission check do not enforce these rules. Use server or firewall controls when access must be prevented. If you believe a NimbleSEO fetcher is misbehaving, email security@nimbleseo.ai with the timestamp and your access-log line — we investigate every report.