Can AI and LLMs Access My Page?

The short answer

Often not, and usually without anyone at the company knowing. A page can be perfectly visible in a browser and still be refused to every AI crawler that matters, because the decision is made by a firewall rule or a bot-protection product long before your page is rendered. The people who set that rule were blocking scrapers, not AI search, and nothing in your analytics tells you it happened.

Four things decide whether an AI system can read your page, and they fail independently:

• Your robots.txt may disallow crawlers such as GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, sometimes inherited from a template nobody has read in two years.
• Your CDN or WAF may refuse those crawlers with a 403 or a challenge page regardless of what robots.txt permits. This is the most common cause and the least visible one.
• Your rendering may deliver an empty shell to anything that does not run JavaScript, which is most crawlers most of the time. See making a SPA readable for AI crawlers.
• Your response speed may exceed what a live fetcher will wait for. When a person asks ChatGPT or Perplexity about your page, the fetcher gives up long before a human would.

Any one of these is enough to keep you out of AI answers. Checking only the first one, which is what most tools do, tells you almost nothing.

Why reading your robots.txt does not answer the question

robots.txt is a request, not a mechanism. It states a policy that well-behaved crawlers honour, and it has no idea what your infrastructure actually does. In practice the two disagree constantly: a robots.txt that explicitly welcomes OAI-SearchBot sitting behind a bot-protection rule that refuses it at the edge. The policy says yes, the server says no, and the server wins.

The opposite happens too. Plenty of sites believe they have blocked AI training crawlers, while the rule only ever applied to a User-Agent string those crawlers stopped sending.

If you want to know whether AI can access your page, the only honest test is to make the request and look at the response.

How to check whether AI can actually reach your page

Glippy's live AI crawler access check sends 12 requests for the same URL, from the same IP, in the same session. Every request is identical except for one header: the User-Agent. That design is the whole point, and it is what makes the result trustworthy.

1. Install the Glippy Chrome extension and open the page you want to test.
2. Scan the page, then click Run live crawler check in the AI crawler access panel. Chrome asks for permission for that site the first time.
3. Wait about a minute. Glippy reports each crawler as readable, blocked, or inconclusive, plus a verdict for the session as a whole.

The check is free, runs from your own network, and never runs on its own.

Rather not install anything? The AI Crawler Accessibility Checker runs the same 12 requests against your home page, from Glippy's own edge, and reports the same states and verdicts. The trade-off is the vantage point: those requests leave from a datacenter address rather than yours, which is precisely the traffic a lot of firewalls refuse. When ours is refused, the result says so and stops there, because at that point nothing is known about the AI crawlers either way.

What gets requested

Eight AI crawlers, using the User-Agent strings their vendors publish:

• Search indexers - OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity). These build the indexes that AI answers cite. If you are asking whether AI can find you, these are the three that matter.
• Live fetchers - ChatGPT-User, Claude-User, Perplexity-User. These fetch your page in real time, at the moment a person asks an assistant about it.
• Training crawlers - GPTBot, ClaudeBot. These collect content for model training. Refusing them is a legitimate business decision, so Glippy reports it and never counts it against you.

Alongside those, four control requests make the answer readable rather than just noisy:

• A plain browser, to prove the page is reachable from here at all.
• An unrecognised bot, Glippy's own honest identity, which separates "your site blocks these crawlers by name" from "your site blocks anything that looks automated".
• A spoofed Googlebot, which reveals whether your rules trust User-Agent strings and quietly exempt the search engines everyone allowlists.
• The plain browser again, last, to catch the case where the site started refusing us partway through and the whole run is unreliable.

Why the controls matter more than the status codes

A 403 on its own is not evidence of anything. It may only mean the IP you are testing from has a poor reputation, which happens to office networks, VPNs and cloud servers all the time. Tools that report a bare refusal as "GPTBot is blocked" produce confident, wrong answers, and people rewrite robots.txt files that were never the problem.

Only a difference between requests that are identical except for the User-Agent tells you something about your site's policy. That is why the browser baseline runs first: if your site refuses a plain browser from this network, Glippy reports that its vantage point is blocked and draws no conclusion at all about the AI crawlers. Saying "we could not tell" is the correct answer more often than the industry likes to admit.

What the result tells you, and what to do about it

• Everything readable. AI crawlers can reach the page. Now the question becomes whether they can understand it, which is what the rest of Glippy's 16 categories measure.
• Blocked by name. An unknown bot got through but named AI crawlers did not, so a rule targets them specifically. Find it in your WAF, CDN rule set or bot-management product, not in robots.txt.
• Allowlist block. Everything is refused unless it claims to be a named search engine. Real AI crawlers send exactly the kind of string those rules reject, so add them to the allowlist by verified IP range, which OpenAI, Anthropic and Perplexity all publish.
• Blanket bot block or identity verification. Every non-browser identity is refused, including a spoofed Googlebot. This is the honest limit of any external test: a site that verifies crawlers by IP will refuse a spoof and still admit the real crawler. Your server access logs settle it.
• Slow responses. Nothing is blocked, but the fetchers that were served took long enough that an assistant may abandon the request. That costs you the citation just as effectively as a 403.

Glippy also names the CDN or bot-protection vendor behind a refusal, and flags the case where your firewall and your robots.txt contradict each other, which is usually the fastest route to the rule that needs changing.

The limits of testing from outside

An external probe cannot send requests from OpenAI's or Anthropic's networks, and all three major vendors publish the IP ranges their crawlers fetch from. A firewall that checks those lists will refuse Glippy's test and admit the real crawler, which is a good outcome and one the tool reports as such rather than as a block. Your own server access logs remain the ground truth. What the live check gives you is the fastest reliable signal you can get without them, and an explicit statement whenever it does not know.