Start with the purpose of the crawler
Different crawlers can serve different purposes, even when operated by the same provider. Search crawlers discover material for search experiences. User-directed fetchers retrieve a page in response to a request. Training crawlers have a different purpose. Allowing training is not a prerequisite for search eligibility.
| Experience | Search crawler | User-directed retrieval |
|---|---|---|
| Google Search | Googlebot | Depends on the Google product and request. |
| ChatGPT search | OAI-SearchBot | ChatGPT-User |
| Claude search | Claude-SearchBot | Claude-User |
| Perplexity search | PerplexityBot | Perplexity-User |
Check both robots rules and the server response
Read the policy at the root of the relevant hostname. A crawler-specific group can take precedence over the wildcard group, so private-path restrictions need to be considered in the group the crawler actually follows. A successful robots check does not prove that DNS, TLS, a firewall or a challenge page will permit the fetch.
Compare the requested URL, redirects, final response and returned content. A 200 response containing only a challenge or an empty application shell is not the same as receiving the intended page. Make important product and company information available as readable HTML.
A user-agent string does not verify a visitor
Anyone can claim a crawler name in an HTTP header. Use the operator’s documented verification method when identifying traffic or creating a specific network exception. Do not remove authentication, rate limits or customer-data protections simply because a request claims to be a bot.
Protect private content independently
Robots rules are instructions to cooperative crawlers, not access control. Reports, account records and payment operations still require server-side authentication and ownership checks. Keep private URLs out of public sitemaps and public structured data.
After technical changes, verify the live response and the relevant platform’s reporting tools. Successful fetch tests demonstrate access under the tested conditions; they do not demonstrate an actual visit by the operator or inclusion in an index.
Sources and review
Technical guidance is based on the sources below. Platform behaviour can change; the review date applies to this guide, not to any particular website’s index status.