Crawler information & allow-list guide
GlippyBot is the user-agent Glippy sends when it fetches a page. Glippy fetches pages in two different ways, from two different places, and this page describes both, because the answer to "can I allow-list this by IP" and "why does it not obey my robots.txt" is different for each.
If you landed here because an AI crawler User-Agent showed up in your logs from a Cloudflare address, skip to the hosted check. That was us, and the section explains exactly what we sent and how to stop it.
The Desktop app, the Chrome extension and the MCP server run on the machine of the person using Glippy. Requests come from that person's residential or corporate IP, not from Glippy infrastructure, and there is no fixed IP range. Somebody is sitting in front of the tool when they happen. This is the mode the rest of this page called "GlippyBot" before the hosted check existed, and everything below about robots.txt and crawl rate applies to it.
The AI Crawler Accessibility Checker on this website runs on Glippy's own Cloudflare Worker. Its requests leave from Cloudflare's edge network, on our infrastructure, not from a Glippy user's machine.
Anyone can type any domain into it. We do not verify that the person running it owns the domain, in the same way that a DNS lookup tool or an SSL checker does not. It sends a fixed, small number of requests to a single URL, it is rate limited per domain, every request carries an X-Glippy-Probe header, and some of those requests deliberately carry AI crawler User-Agents. All four of those things are documented below in full.
For the hosted check the reliable identifier is a header, not the User-Agent. Every request it sends, whatever User-Agent that request carries, includes:
X-Glippy-Probe: https://www.glippy.dev/bot
That value is constant and it never varies within or between runs. It is on the plain browser request, on the GlippyBot request and on all eight AI crawler requests, so one filter on X-Glippy-Probe catches a whole run and every future run. A request carrying this header is us. A request carrying an AI crawler User-Agent without this header did not come from the hosted check, and whether it is the real crawler is a question for that vendor's published IP list, linked below.
Two GlippyBot strings are in use and they are not identical. Match on the substring GlippyBot rather than on either full string, in robots.txt and in WAF rules alike.
The local crawler, on every page it fetches:
Mozilla/5.0 (compatible; GlippyBot/1.0.0; +https://www.glippy.dev/bot)
The version segment matches the installed Glippy Desktop release, so older clients send a different version number.
The GlippyBot control request inside the agent access check, in every mode including the hosted one:
Mozilla/5.0 (compatible; GlippyBot/1.0; +https://glippy.dev/bot)
Note the fixed 1.0 and the missing www. That is a wart we intend to remove, and it is the reason an exact-string rule is the wrong rule: whichever full string you pinned, the other one would slip past it, and so would the next version bump.
robots.txt, sitemap.xml, and llms.txt.robots.txt rules by default (this can be toggled by the user, but the default is on).GET request, accepts the response, and never executes JavaScript or submits forms.The check measures one thing: whether your server treats a request carrying an AI crawler identity differently from the same request carrying an ordinary browser identity. There is no way to measure that without sending both, so it sends both, to one URL, and compares them.
429 Too Many Requests is retried once, so 23 requests is the hard ceiling for a single run. A normal run is 12, and a run against a server that refuses the first plain browser request is 1.GET, and never requests a second URL.The five minute rule is the one that bounds what your server sees, since it holds no matter how many people run the check. The ordinary case is one run and then silence, and the busiest plausible hour is a handful of runs against a single URL. If you measure meaningfully more than that, it is a bug rather than a design, and we would genuinely like the timestamps: [email protected].
Because we sent it, and you should know that before you read anything into it.
Eight of the twelve requests in a run carry the published User-Agent of a real AI crawler:
OAI-SearchBot ChatGPT-User GPTBot (OpenAI) Claude-SearchBot Claude-User ClaudeBot (Anthropic) PerplexityBot Perplexity-User (Perplexity)
Those requests are Glippy, from Cloudflare's edge, with X-Glippy-Probe attached. They are not OpenAI, Anthropic or Perplexity fetching your site. They do not mean your content was retrieved for an answer or for training, and they should not be counted as AI crawler traffic in your analytics. The one Googlebot-labelled request is a control in the same way and is not Google: Google verifies its crawler by reverse DNS, so ours fails that test by construction, which is exactly what makes it useful for telling a site that checks identity apart from a site that trusts a name.
The hosted check does not read your robots.txt, and a Disallow line will not stop it. A robots.txt rule is an instruction to the crawler it names, and the check exists to find out what your server does to a request before robots.txt is ever consulted. This matters if you disallow GPTBot: when GPTBot then shows up in your logs anyway, it is us, from a Cloudflare address, with the header attached, and not GPTBot ignoring your rules. Filtering on X-Glippy-Probe removes the whole run from your reporting in one rule.
The same twelve-request check also ships in the Chrome extension and the MCP server, where a Glippy user runs it against a site from their own machine. Those runs send the same eight crawler identities to the same single URL, but they arrive from that person's own address rather than from ours, so a crawler User-Agent from an ordinary residential or office IP can be a local run of this check rather than the vendor. That vantage point is the point of the local version: it measures what your firewall does to a visitor's network, which is the network the answer is actually about.
Each vendor publishes the addresses its crawlers actually fetch from, which is how you tell a real visit from a spoofed one: openai.com/gptbot.json, claude.com/crawling/bots.json, perplexity.com/perplexitybot.json.
GlippyBot has no fixed IP range in either mode. The local crawler runs on each Glippy user's machine, so those requests come from that user's residential or corporate IP. The hosted check runs on Cloudflare Workers, so those requests come from Cloudflare's global edge and the source address can be any address in Cloudflare's published ranges, which are shared with a large part of the web and are not ours. An IP rule is therefore the wrong tool in both directions: use the User-Agent for the local crawler, and the X-Glippy-Probe header for the hosted check.
If your WAF (Cloudflare, Akamai, SiteGround, DataDome, Imperva, etc.) is challenging or blocking the local crawler, the cleanest fix is an allow-rule on the User-Agent substring. Most WAF dashboards let you create a rule like:
"If User-Agent contains GlippyBot, then skip the bot challenge."
If you prefer to control crawler behavior at the protocol level, use robots.txt. Note that robots.txt governs the local crawler only: the hosted check reads a single URL and does not consult it, so use the header rule further down to control that one.
User-agent: GlippyBot Allow: /
User-agent: GlippyBot Disallow: /
The local crawler honors Disallow directives by default, so if you disallow it, no Glippy user will be able to audit your site from the app or the extension. The hosted check will still reach your home page, because it never asks for the file; block it with the header rule below.
User-agent: GlippyBot Disallow: /admin/ Disallow: /api/ Disallow: /private/ Allow: /
One rule, matched on the header, catches every request in every run including the ones carrying crawler User-Agents. As a Cloudflare WAF expression:
(http.request.headers["x-glippy-probe"][0] eq "https://www.glippy.dev/bot") Action: Block
Or in nginx:
if ($http_x_glippy_probe) { return 403; }
What happens then is worth knowing, because it is the whole design of the check: the first request in a run is a plain browser request, and if that one is refused, the run stops there and the result says our vantage point could not see the site at all. It does not, and cannot, report your site as blocking AI crawlers. Blocking us costs you a result you did not ask for and costs you nothing else.
If you would rather not maintain a rule, mail us the domain at [email protected] and we will exclude it from the hosted check.
The local crawler defaults to 5 concurrent requests with no artificial delay between them, capped at 5,000 pages per crawl. The user can lower these limits in the app. If you'd like a global rate hint, set Crawl-delay in your robots.txt:
User-agent: GlippyBot Crawl-delay: 2
The hosted check ignores Crawl-delay for the same reason it ignores Disallow: it does not read the file. Its rate is fixed in code at one request at a time, roughly four per second at most, twelve per run, and the per-domain limits above.
If a Glippy user reported GlippyBot being blocked on your site, the in-app banner will show them the exact User-Agent string and the IP address making the request. They can forward those details to you, and you can allow-list them in your WAF.
Glippy publishes no IP list, because it has none: see above for why, in both modes. For the hosted check, X-Glippy-Probe is the canonical identifier and it is the only one that covers all twelve requests. For the local crawler the User-Agent is the identifier, and the honest caveat is that a User-Agent is a claim rather than a proof: match on the GlippyBot substring, treat it as a hint rather than an identity, and remember that anyone can send it. In practice a malicious crawler is far more likely to spoof a major search engine than a small tool.
If you have questions about how Glippy fetches pages, want your domain excluded from the hosted check, want to request a rate-limit change for a specific deployment, or believe the hosted check has been pointed at your site abusively, contact [email protected]. For an abuse report, the timestamps and the X-Glippy-Probe header from your logs are what let us find the run.
Check your site for AI search readiness
Start free with the Glippy Chrome extension for instant page checks. Scaling up? Automate audits across many URLs and your whole sitemap with the Glippy MCP server.