What we observe
We fetch /robots.txt and read its rules for the user-agent tokens AI companies publish for their crawlers, in two groups: crawlers that fetch a page when someone asks a question, and crawlers that collect pages for training. For each group the file allows them, does not mention them (which leaves them allowed), asks some to stay away, or asks all of them to. We also read every page we visit for a noai or noimageai directive, in the robots meta tag or the X-Robots-Tag header. Anyone can repeat this by opening the site’s /robots.txt.
Why it is here: the evidence
Documented A platform’s own published document or code says this is read: a standard the crawlers follow, or the documentation or source code of the company that runs the system.
Source: The robots.txt standard (RFC 9309). Source grade A: read at the source: the document, the code or the study itself.
Last re-checked ; due again within 30 days.
What we cannot see
Whether any crawler obeys the file. robots.txt is a request; honouring it is each operator’s choice, and a read from outside cannot see that.
How to fix it
Write the two groups as separate rules: keeping training crawlers out is a content-licensing choice and need not keep out the crawlers that fetch a page to answer a question.
- Layer
- Read: Can an AI crawler fetch your pages and understand what is on them?
- Level
- One of the checks of Level 1: Can an automated visitor that says who it is get your pages and read them?
- Where its finding lives
- Search & findability
- How common it is
- Not published yet. We publish a share only once at least 30 sites stand behind it.
- Check id
chk-ai-access
Cite as: Invara, “How AI sees your site”, method version 1, check chk-ai-access. https://clearsitedigital.com/how-ai-sees-sites/chk-ai-access