AI crawler access: what to check

Start with the actual response

Request the preferred URL and record status, content type, redirects and indexing headers. A real page should return its useful content; a missing page should return 404 rather than a successful homepage. Inspect the original HTML for the page’s explanation and links, then compare it with the rendered page.

Google documents its JavaScript rendering process. That capability is not a reason to assume every AI crawler executes your scripts. Our recommendation for a small marketing site is to serve essential copy and navigation in HTML and keep decorative JavaScript optional.

Separate search controls from training controls

OpenAI

OpenAI identifies OAI-SearchBot for search and GPTBot for training-related crawling. Their robots controls are independent. ChatGPT-User serves user-initiated requests and is not the automatic search crawler.

Perplexity

Perplexity documents PerplexityBot and Perplexity-User separately. Review its current descriptions and published verification information before writing access rules.

Anthropic

Anthropic distinguishes ClaudeBot, Claude-SearchBot and Claude-User. A rule intended for training should not inadvertently block its search crawler.

Google

Google-Extended is a robots control token, not a separate HTTP user agent. Do not confuse it with Googlebot or assume its setting alone determines Google Search eligibility.

Consult each provider’s current documentation; names and policies can change. Do not claim that a broad wildcard permission is “search only.”

Keep robots rules understandable

A permissive wildcard allows compliant crawlers, including training crawlers, unless a more specific rule restricts them. More specific groups can replace the wildcard group for a matching crawler, so test the final combined file rather than reading individual lines in isolation. Google explains its robots matching rules.

Do not block the CSS or scripts needed to understand public pages. A robots disallow is not authentication and is not a reliable method to remove confidential material from search. Keep private reports behind actual access control.

Review the edge as well as the file

A CDN can add robots text, challenge requests or apply WAF rules before the origin responds. Compare the published robots file with source configuration and review verified-bot events when account access is available. Do not disable site-wide protection just to make one diagnostic pass.

Sending a provider’s user-agent string from a laptop only tests that request. It does not authenticate the requester or prove that the provider’s infrastructure is allowed. Use official verification methods and provider IP information where documented, alongside actual logs. A failed request from one testing environment is not proof that all crawlers are blocked.

Normalize URLs without losing attribution

Choose a preferred HTTPS host, preserve paths and necessary query parameters in redirects, and use absolute canonicals without tracking queries. Google treats canonicalization as a set of signals; a tag is not an access control or an unconditional command.

A sitemap should list real canonical pages, not every URL variation. Google’s sitemap guidance explains supported formats and submission. Submission does not guarantee indexing.

Turn a check into evidence

Keep the date, exact URL, observed response and test environment. Mark what requires dashboard or crawler-log access. Recheck after changes and distinguish a local preview from production. Our technical self-audit demonstrates those distinctions without inventing AI-answer results.