What we observe
On every page we read we check the HTTP status, the robots meta tag and the X-Robots-Tag header for noindex, the canonical link, and whether robots.txt blocks the page. A page counts as one a search engine may show when it answered and nothing on it says noindex.
Why it is here: the evidence
Documented A platform’s own published document or code says this is read: a standard the crawlers follow, or the documentation or source code of the company that runs the system.
Source: Google Search Central, AI features and your website: a page must be indexed and eligible to show in Search. Source grade A: read at the source: the document, the code or the study itself.
Last re-checked ; due again within 30 days.
What we cannot see
Whether a page is in fact in any index, or shown for any question. We read what the page declares; the index belongs to the search engine.
How to fix it
Take noindex off the pages meant to be found, point each page’s canonical link at its own preferred address, and let robots.txt reach them.
- Layer
- Read: Can an AI crawler fetch your pages and understand what is on them?
- Level
- One of the checks of Level 1: Can an automated visitor that says who it is get your pages and read them?
- Where its finding lives
- Search & findability
- How common it is
- Not published yet. We publish a share only once at least 30 sites stand behind it.
- Check id
chk-canonical-robots
Cite as: Invara, “How AI sees your site”, method version 1, check chk-canonical-robots. https://clearsitedigital.com/how-ai-sees-sites/chk-canonical-robots