1. Crawler identity and purpose
The product token is SearchProofBot and production requests use the User-Agent SearchProofBot/1.0 (+https://searchproofai.com/crawler-policy). The crawler is used only to audit search and AI visibility readiness for domains requested by users and provide evidence; it is not used to build a general search index, advertising profile, or dataset for sale to train AI models.
2. Who initiates crawling and authority
Collection starts when a user enters a domain for a free/deep audit or when a saved profile is rerun. Users must own/administer the site or have a legitimate business interest. Reports of unauthorized analysis are investigated; jobs or accounts may be stopped or restricted and proof of domain control requested.
3. Access scope and page limits
- Same-site HTTP/HTTPS HTML pages available without separate sign-in
- Up to 5 pages for a free audit and 100 for deep analysis, with separate caps on attempts and discovery queues
- Static HTML first; a limited external browser renderer is used only when configured
- The crawler does not supply credentials or session cookies or submit forms, and it blocks private IPs, localhost, cloud metadata, risky ports, and blocked file types. However, a login or admin path that is publicly reachable without authentication may be considered public HTML unless blocked by robots.txt; private resources must be protected by authentication and SearchProofBot should be explicitly blocked where needed.
4. URL discovery and same-site restriction
Candidates come from the start URL, Sitemaps listed in robots.txt, sitemap indexes, and internal links in visited HTML. Scheme, host, DNS/IP safety, and same-site status are revalidated for every URL and redirect. Tracking parameters and duplicates are normalized, while PDFs, archives, images, executables, and other blocked extensions are skipped.
5. robots.txt and page directives
Before collection, the top-level /robots.txt is requested and applicable Disallow plus reasonable Crawl-delay for SearchProofBot or * are applied. The current implementation does not claim complete RFC 9309 support for every Allow precedence and error-cache rule; operators with complex directives should use an explicit SearchProofBot Disallow and the official blocking channel. noindex, nofollow, canonical, and X-Robots-Tag are recorded as audit evidence, not treated as authority to bypass access controls. robots.txt is not access control; private resources require authentication.
User-agent: SearchProofBot
Disallow: /
# Or block a path
User-agent: SearchProofBot
Disallow: /private-area/6. Request rate, size, and safeguards
- Sequential per-site requests with at least the default 250 ms delay; robots Crawl-delay is applied up to 2 seconds
- Default limits: 10-second connect, 20-second request, 5 redirects, and 5 MiB response
- DNS/IP revalidation on every redirect and blocking of private, loopback, link-local, metadata, and risky targets
- Further requests stop on cancellation, limits, or repeated failure
7. Fields collected and stored
The service may store requested/final URL, HTTP status, content type, response time, title, meta description, language, canonical, robots/X-Robots directives, heading structure, visible text, internal/external links, image alt text, structured data, content hash, collection time, performance metrics, and HTML evidence for reproducibility. Results include scores, rule evidence, and relevant URLs. Bodies are processed only within size limits and unsupported types are skipped.
8. Data not collected or separately used
- Login credentials, session cookies, form submissions, payment data, or private pages requiring authentication
- Separate extraction of incidental personal data for person profiling or marketing lists
- Sale of collected HTML as a general search index, foundation-model training dataset, or third-party data product
9. Retention, deletion, and separation from AI observations
Guest HTML evidence older than seven days is removed by a daily cleanup job, subject to a short delay from backlog or deletion retries. Account-saved analyses, snapshots, and reports remain until account deletion, with statutory records segregated; individual permanent deletion will be announced if that feature becomes available. Deep-analysis AI queries and responses are a separate stage and may be sent through Gabia AI Hub to selected models. Free SEO audits do not call external AI; unavailable channels are marked unmeasured. Independent web collection by an AI provider is separate from SearchProofBot.
10. Blocking, rate adjustment, deletion, and abuse reports
Site operators may set Disallow for SearchProofBot in robots.txt. Requests for emergency stopping, a slower rate, deletion of stored evidence, unauthorized analysis, or spoofed User-Agent reports may be sent to support@searchproofai.com. We may request the domain, log time/IP/User-Agent/URL, requested action, and minimal ownership proof such as DNS TXT, administrator email, or Search Console. After verification, ongoing jobs or related accounts may be restricted, non-statutory evidence deleted, or operating settings adjusted, and available action and completion will be communicated. Keep the robots.txt block in place until any persistent block is confirmed.