What we observe
We request a small sample of the pages we read, up to five, without a browser and under our own name, and record what came back: the page, a challenge, a refusal, a redirect or no answer. Where robots.txt does not ask the AI crawlers that answer questions to stay away from a page, and the edge challenged or refused that request, the file and the edge disagree. The disagreement is what we report, never either side alone.
Why it is here: the evidence
Documented A platform’s own published document or code says this is read: a standard the crawlers follow, or the documentation or source code of the company that runs the system.
Source: Google Search Central, AI features and your website: crawling must be allowed in robots.txt and by any CDN or hosting infrastructure. Source grade A: read at the source: the document, the code or the study itself.
Last re-checked ; due again within 30 days.
What we cannot see
What a named AI crawler receives. We never send another company’s crawler name, and an edge that verifies a crawler’s identity may treat a verified crawler differently. A page outside the sample has no reading: not measured, never served.
How to fix it
Decide which of the two you meant and make the other agree: let the edge serve the pages robots.txt opens, or say no in robots.txt as well.
- Layer
- Read: Can an AI crawler fetch your pages and understand what is on them?
- Level
- One of the checks of Level 1: Can an automated visitor that says who it is get your pages and read them?
- Where its finding lives
- Search & findability
- How common it is
- Not published yet. We publish a share only once at least 30 sites stand behind it.
- Check id
chk-ai-delivery
Cite as: Invara, “How AI sees your site”, method version 1, check chk-ai-delivery. https://clearsitedigital.com/how-ai-sees-sites/chk-ai-delivery