How to Fix the Technical Issues Blocking AI Crawlers From Your Site

Every other guide in this series assumes something that is often not true: that AI platforms can actually reach your website.

They frequently cannot. A robots.txt rule inherited from a staging config, a CDN blocking unfamiliar user agents, pages rendered entirely in client-side JavaScript, a site that answers slowly enough for a crawler to give up — none of these show up in your analytics, and all of them remove you from AI answers completely.

This is the cheapest problem in AI visibility to fix and the most expensive to leave alone.

Why This Comes Before Content

The research panel in the visibility report puts it bluntly: URL accessibility scores 9.5 out of 10 as the top AI citation factor. That figure is easy to misread as "accessibility ranks you highly". It does not.

Step 1 — Run the Checks

The AI Model Registration screen verifies that your domain is discoverable by and registered with the AI platforms. It runs a set of checks grouped into six categories, filterable from the top:

You can run an individual Check or Verify All in one pass. Run everything the first time — the categories interact, and a Missing result in one often explains a weak result in another.

Step 2 — What Each Status Means

Every check returns one of four states, and they are not equally urgent.

Opening a check shows its detail — usually a sentence or two of explanation, and where problems were found, a list of the specific issues behind the result rather than one dense paragraph.

AI Crawlers

This is the category that decides whether anything else matters. It checks whether the AI user agents are permitted to fetch your pages.

Three causes account for nearly every failure here:

Discovery Files

Discovery files tell crawlers what exists and what matters. Three carry most of the weight:

Read yours in a browser before doing anything else — most people have never looked. It should explicitly allow the AI user agents you want citing you, disallow only what genuinely should not be public, and point to your sitemap.

Complete, current, and referenced from robots.txt. A sitemap listing pages that no longer exist is worse than a short one that is accurate.

A Markdown file at your root giving AI crawlers a curated map of your most important content. It is an emerging convention rather than a standard, which is exactly why adding it is cheap and low-risk: a short list of your key pages with a line of context each, deployed at your root, and verified by opening it in a browser.

Structured Data and Knowledge Graph

Access gets you read. These two decide whether the model knows who it just read.

Structured Data checks whether your facts exist in machine-readable form as well as prose — Organization, Article, FAQ, Product and the rest. Prose says what you do; schema states it in a form that needs no interpretation.

Knowledge Graph checks whether you exist as a recognised entity rather than only as a domain. This is what separates a brand a model can talk about confidently from a URL it happens to have crawled.

Entity confusion is the quiet killer here.

If your business name is shared with a bigger company, a product, or a common word, models routinely blend you with something else. Consistent naming, a complete Organization schema, and sameAs links to your real profiles are what disambiguate you — and they are the reason the social links in your footer are worth wiring up properly.

Search Engines and Platforms

The last two categories are the ones teams assume are handled.

Search Engines matters because several AI surfaces build on conventional indexing — Google AI Overviews most obviously. Being unindexed there removes you from those answers regardless of how well the rest is configured.

Platforms checks presence on the third-party sources assistants read about you. This is off your own site, so it is the slowest category to improve and the one that most influences how confidently a model describes you.

Reading Registration Coverage

The summary reports Total Checks and Registration Coverage. Coverage is a useful headline and a poor priority list.

Ninety per cent coverage with the AI Crawlers category failing is worse than sixty per cent with access clean — the failing ten per cent is the door. Read the categories, not the percentage.

Fixing Them in Order

Crawler access and response speed. Until a platform can fetch your pages reliably, every other fix is theoretical.

robots.txt pointing at an accurate sitemap, and an llms.txt listing what you actually want cited. An afternoon, once.

Organization schema with consistent naming and sameAs links to your real profiles. This is the fix that most improves how a model describes you, not just whether it finds you.

Re-run after every migration, redesign, CDN change or robots.txt edit. These settings revert silently and nothing on the site looks different when they do.

Common Mistakes

Googlebot being welcome says nothing about the AI user agents. They are separate agents and frequently governed by separate rules.

Categories are not interchangeable. One failing access check outweighs several passing cosmetic ones.

The most common cause of a sudden visibility drop is an infrastructure change that reintroduced a block nobody noticed.

It helps crawlers find your best content. It does not make weak content citable, and it does not substitute for allowing access in the first place.

Frequently Asked Questions