What Is Web Bot Auth and How Do You Implement It in 2026?

A signed bot request carries an Ed25519 signature you can check in about forty lines of Node, with no vendor in the loop. This page shows you how to verify one, how to allow verified agents at your CDN instead, and how to publish a key directory if you operate an agent yourself.

Check Your Site Now Browse the Glossary

CategoryAgent & AI protocols StatusRFC 9421 published Maintained byIETF, Cloudflare Glippy checkAgent Interactivity (category 10)

Web Bot Auth (HTTP Message Signatures, RFC 9421) is a profile that lets an automated client sign every outbound HTTP request with an Ed25519 key, so a server identifies it from cryptography rather than from a user agent string or an IP range. The client sends three headers, Signature, Signature-Input and Signature-Agent, and the server resolves the Signature-Agent URL to a published JSON Web Key Set to check the signature. It exists so site owners can allow named agents one at a time, and so agent operators can pass bot mitigation without asking every site for an IP allowlist.

Why Web Bot Auth matters for AI visibility

Every identity signal an origin had before this was either free text or an inference. A user agent string is typed by the client, so anything at all can claim to be GPTBot. Reverse DNS and published IP ranges are much stronger, but they break the moment an agent runs in a headless browser on shared cloud egress, or on an end user's own machine, which is exactly where user-driven agents live. A signature moves identity into the request itself: the client proves possession of a private key whose public half is published at a domain it names, and once that key is cached you can check it in well under a millisecond.

For AI visibility this changes what a crawler policy can be. A robots.txt line or a firewall block is all or nothing at the product-token level, so the usual outcome is that you either accept bulk training scrapers along with the answer engines that cite you, or you turn both away. Per-request signatures let you separate them and apply different rules to each. Two limits are worth stating plainly. A valid signature proves only that the sender holds a key published at that URL and that the covered parts of the request were not altered; it is not a claim about the operator's behaviour. And an unsigned request is not evidence of anything, because most crawlers still do not sign at all.

Where the spec lives

Web Bot Auth is RFC 9421 plus a draft that narrows it. Read the RFC for the signature base and the algorithm registry, and the draft for everything specific to bots.

RFC 9421: HTTP Message Signatures - the Proposed Standard, February 2024. Section 2.3 defines created, expires, keyid and tag; section 3.1 defines the signature base you have to rebuild byte for byte.
draft-meunier-webbotauth-httpsig-protocol - the bot profile: the Signature-Agent header, the well-known key directory, and Appendix E test vectors you can run your verifier against before you trust it.
draft-meunier-webbotauth-registry - the Signature Agent Card: optional metadata about an operator's purpose, rate expectations and the robots.txt product token it answers to. Reach for it when a key alone is not enough to write a policy.
Cloudflare Web Bot Auth documentation - what a verifier already running at scale accepts, and how an operator registers a directory URL with the Verified Bots programme.
cloudflare/web-bot-auth - reference implementations in TypeScript and Rust, plus a Cloudflare Worker verifier, a Caddy plugin and a signing browser extension. Start here if you would rather not hand-roll the parser.
A live example key directory - a real directory served with the real media type. Point your fetch and parse code at this before you point it at production traffic.

Three ways to implement Web Bot Auth

The three jobs are genuinely different. Verify signatures in your own code if you want the agent identity available to your application logic. Delegate to your CDN if you want the outcome without owning any cryptography. Publish a directory and sign if you operate the agent rather than the site. Most teams do the first or the second, and only reach the third if they ship a crawler of their own.

Verify a signed request in Node with no dependencies

This is the whole mechanism in one file. It parses the three headers, rebuilds the RFC 9421 signature base, resolves the Signature-Agent URL to its key directory and checks the Ed25519 signature. It covers the single-signature shape agents send in practice; reach for a structured-fields library once you need multiple signatures or components this does not list.

javascriptverify-web-bot-auth.mjs

// verify-web-bot-auth.mjs - Node 20+, no dependencies
import { createHash, createPublicKey, verify } from 'node:crypto';

const DIRECTORY = '/.well-known/http-message-signatures-directory';
const cache = new Map();
const thumbprint = (k) => createHash('sha256')
  .update(JSON.stringify({ crv: k.crv, kty: k.kty, x: k.x })).digest('base64url');

async function keysFor(dir) {
  const hit = cache.get(dir);
  if (hit?.until > Date.now()) return hit.keys;
  const res = await fetch(dir, { redirect: 'error' });
  if (res.status !== 200) throw new Error('directory returned ' + res.status);
  const { keys } = await res.json();
  cache.set(dir, { keys, until: Date.now() + 3600e3 });
  return keys;
}

export async function verifyWebBotAuth(req) {
  const [, label, raw] = req.headers['signature-input'].match(/^\s*([^=\s]+)=(.+)$/s);
  const p = Object.fromEntries([...raw.matchAll(/;(\w+)=("[^"]*"|[\d.]+)/g)]
    .map(([, k, v]) => [k, v[0] === '"' ? v.slice(1, -1) : Number(v)]));
  const now = Math.floor(Date.now() / 1000);
  if (p.tag !== 'web-bot-auth') throw new Error('not tagged web-bot-auth');
  if (!(p.created <= now + 30 && now < p.expires)) throw new Error('outside its validity window');

  const agents = Object.fromEntries([...req.headers['signature-agent']
    .matchAll(/(?:^|,)\s*([A-Za-z0-9_-]+)=("[^"]+")/g)].map(([, k, v]) => [k, v]));
  const url = new URL(req.url, 'https://' + req.headers.host);
  let agent;
  const base = raw.slice(1, raw.indexOf(')')).split(' ').map((id) => {
    const name = id.match(/^"([^"]+)"/)[1];
    if (name === 'signature-agent') agent = agents[id.match(/;key="([^"]+)"/)?.[1]] ?? req.headers['signature-agent'];
    const v = { '@authority': url.host, '@method': req.method, '@path': url.pathname, 'signature-agent': agent }[name];
    if (v === undefined) throw new Error('unsupported component ' + id);
    return `${id}: ${v}`;
  });
  if (!agent) throw new Error('signature does not cover signature-agent');
  base.push(`"@signature-params": ${raw}`);

  const dir = new URL(DIRECTORY, JSON.parse(agent)).href;
  const jwk = (await keysFor(dir)).find((k) => thumbprint(k) === p.keyid);
  if (!jwk) throw new Error(p.keyid + ' is not published at ' + dir);
  const pub = createPublicKey({ key: { kty: jwk.kty, crv: jwk.crv, x: jwk.x }, format: 'jwk' });
  const sig = Buffer.from(req.headers.signature.match(new RegExp(`${label}=:([^:]+):`))[1], 'base64');
  if (!verify(null, Buffer.from(base.join('\n')), pub, sig)) throw new Error('bad signature');
  return JSON.parse(agent);
}

What this does: a verified request returns the Signature-Agent origin, which is the stable identity you write allow rules against, and every other path throws. It passes the ed25519 test vector in Appendix E.2.1 of the draft, so you have a known-good input to check your own edits against.

Allow verified signed agents at your CDN

If you sit behind a bot management vendor, verification is already happening upstream of you. Cloudflare validates Ed25519 Web Bot Auth signatures at its edge and exposes the result through ruleset fields, so your only job is deciding what a verified identity buys. These are three custom rule expressions, one per rule, in the order they should run.

textSecurity > WAF > Custom rules

# Rule 1  Action: Skip -> Super Bot Fight Mode, remaining custom rules
(cf.bot_management.verified_bot
  and cf.verified_bot_category in {"AI Assistant" "AI Search" "Search Engine Crawler"})

# Rule 2  Action: Managed Challenge
# Claims an identity it cannot back up: header present, signature did not verify.
(len(http.request.headers["signature-agent"]) > 0
  and not cf.bot_management.verified_bot)

# Rule 3  Action: Managed Challenge
# Verified bulk training crawlers, allowed on public content, challenged elsewhere.
(cf.bot_management.verified_bot
  and cf.verified_bot_category eq "AI Crawler"
  and not starts_with(http.request.uri.path, "/blog/"))

What this does: rule 1 takes answer engines and assistants out of your mitigation path entirely, so a citation-generating fetch never meets a challenge. Rule 2 catches the copycat case, a client that sends a Signature-Agent header it cannot back up, which is the one place a failed signature is genuinely suspicious.

Publish a key directory and sign your agent's requests

The operator side. You need one Ed25519 keypair, a JSON Web Key Set served at a fixed path, and a signer that adds three headers to every outbound request. The kid is derived from the key rather than chosen, so what you publish and what you sign with cannot drift apart.

javascriptagent/web-bot-auth.mjs

// agent/web-bot-auth.mjs - Node 20+, no dependencies
import { createHash, generateKeyPairSync, randomBytes, sign } from 'node:crypto';

// Run generateKeyPairSync once, then load the private key from your secret store.
const { publicKey, privateKey } = generateKeyPairSync('ed25519');
const { crv, kty, x } = publicKey.export({ format: 'jwk' });
const kid = createHash('sha256')
  .update(JSON.stringify({ crv, kty, x })).digest('base64url');

// The origin below must be the one that serves this directory, apex or www, exactly.
const AGENT = 'https://agent.example.com';

// GET /.well-known/http-message-signatures-directory
// Content-Type: application/http-message-signatures-directory+json
// Cache-Control: max-age=86400
export const directory = {
  keys: [{ kty, crv, x, kid, use: 'sig', nbf: 1787000000, exp: 1795000000 }]
};

export function signRequest(method, target, { label = 'sig1', ttl = 300 } = {}) {
  const url = new URL(target);
  const created = Math.floor(Date.now() / 1000);
  const value = JSON.stringify(AGENT);
  const params = `("@method" "@authority" "@path" "signature-agent";key="${label}")` +
    `;created=${created};expires=${created + ttl};keyid="${kid}"` +
    `;alg="ed25519";nonce="${randomBytes(32).toString('base64')}";tag="web-bot-auth"`;
  const base = [
    `"@method": ${method}`,
    `"@authority": ${url.host}`,
    `"@path": ${url.pathname}`,
    `"signature-agent";key="${label}": ${value}`,
    `"@signature-params": ${params}`
  ].join('\n');
  return {
    'signature-agent': `${label}=${value}`,
    'signature-input': `${label}=${params}`,
    signature: `${label}=:${sign(null, Buffer.from(base), privateKey).toString('base64')}:`
  };
}

What this does: it produces the three headers a verifier expects, covering method, authority and path so the signature cannot be lifted onto a different request. The Signature-Agent value must be the exact origin that serves the directory, with no path, because a verifier is not allowed to follow redirects: https://agent.example.com and https://www.agent.example.com are two different identities and only one of them resolves.

Implementation guidelines

These are the things that go wrong once real traffic hits the verifier.

Cache the directory, and know what the cache costs you. Directories are served with Cache-Control: max-age=86400. Removing a key from a directory is the only way to deactivate it, and the draft defines no revocation mechanism, so your cache lifetime is exactly how long a withdrawn key keeps verifying on your site.
Key the cache on the pair, never on the key alone. Store keys against the directory URL and the thumbprint together. A verifier that indexes by keyid alone will verify a request using a key it learned from one operator and attribute it to whichever URL the client claimed, and the impersonated party cannot detect it because its own directory is never fetched.
Never follow a redirect when resolving a directory. The fetch has to return 200 at the URL you constructed. Anything else is a discovery failure. Publish at the exact host you name in Signature-Agent, because an apex to www redirect will silently take you out of the verified set.
Check tag, created and expires before you check the signature. They are nearly free. Discard anything not tagged web-bot-auth, and remember that a signature covering only @authority can be replayed against any method and path on your host until it expires, so keep the window you accept tight.
Derive kid, never type it. At the well-known path the kid must be the RFC 7638 JWK thumbprint of the key beside it, so compute it on both sides and compare. The draft's own illustrative directory carries a kid that does not match its key material, which is a good reason to derive rather than copy.
Fail open on discovery, closed on verification. If the directory fetch times out or returns a non-200, treat the request as unsigned and apply your ordinary bot policy. If a signature is present and does not verify, that is a real signal and you can act on it.
Rotate by overlap. Publish the new key before its first use, keep the old one until its exp passes, then remove it. Swap in one step and every verifier with a warm cache will reject your traffic until that cache expires.

Do this, not that

Do

Serve the directory over HTTPS at /.well-known/http-message-signatures-directory with a 200 status and Content-Type: application/http-message-signatures-directory+json.
Log the verified Signature-Agent origin alongside the user agent string for a week before you enforce anything, so you can see who actually signs.
Cover @method and @path as well as @authority, and set expires in minutes rather than at the 24 hour ceiling the draft allows.
Keep robots.txt and your edge rules consistent: an agent you let through the firewall should not be disallowed for the same paths in robots.txt.

Do not

Do not trust the Signature-Agent value before you resolve it. The client picks that string, so it is a claim until your fetch of the directory succeeds.
Do not accept a signature whose keyid you matched from a cache entry you cannot tie back to that specific directory URL.
Do not treat an unsigned request as hostile. Googlebot, GPTBot and most bulk crawlers are still identified by reverse DNS and published IP ranges.
Do not fall back to HMAC or a key shared between clients. The draft requires a unique asymmetric keypair per client, and Cloudflare verifies Ed25519 only.

How Glippy checks this

Glippy scores signed-agent readiness under Agent Interactivity (category 10), the group of checks about what an autonomous agent can do on your page rather than just read from it. It looks at whether your pages expose machine-actionable affordances, and pairs that with the crawl-access checks that report which named agents your robots.txt and headers actually admit, so you can see the mismatch where an agent you allow at the edge is turned away one layer up. Run the AI agent accessibility checker on any URL to see the category breakdown.

Check your Web Bot Auth setup

Glippy runs 240+ checks across 16 categories on any page, including Agent Interactivity (category 10). No sign-up required.

Try the Glippy Chrome extension

Frequently asked questions

Is Web Bot Auth a finished standard?

Half of it is. RFC 9421, the HTTP Message Signatures specification underneath it, has been a Proposed Standard since February 2024 and is stable. Web Bot Auth itself is an individual Internet-Draft, draft-meunier-webbotauth-httpsig-protocol-02, dated August 2026 and expiring in February 2027, discussed on the IETF web-bot-auth mailing list but not yet adopted as a working group document. The header names and the well-known path have held steady across revisions and are deployed in production, but the well-known URI and the media type are still requests in the draft's IANA Considerations rather than completed registrations.

Which AI agents actually sign their requests today?

You can check any operator yourself by fetching its key directory. As of August 2026, chatgpt.com and www.browserbase.com both serve a live JSON Web Key Set at /.well-known/http-message-signatures-directory, and Cloudflare's signed agents launch named ChatGPT agent, Goose, Browserbase, Anchor Browser and Cloudflare Browser Rendering. Amazon Bedrock AgentCore Browser signs when the feature is switched on for a browser tool. Bulk indexing and training crawlers such as Googlebot and GPTBot are still identified by reverse DNS and published IP ranges, so signature coverage today is mostly user-driven agents rather than crawlers.

Does a valid signature mean I should allow the bot?

No. Verification establishes two things and no more: the sender holds a private key whose public half is published at the URL in the Signature-Agent header, and the covered parts of the request have not been altered in transit. It says nothing about whether the operator respects robots.txt, honours your rate limits, or uses your content the way you would like. Treat a verified identity as a stable name to attach a policy to, not as a reason to skip having one.

Does Web Bot Auth replace robots.txt or an IP allowlist?

It replaces the allowlist and sits alongside robots.txt. robots.txt states what you permit, while a signature states who is asking, so you still need a robots.txt rule for every agent you have an opinion about. As an identity mechanism it does supersede IP ranges, because an Ed25519 key travels with the request whereas an IP range does not survive an agent running on shared cloud egress or on an end user's machine. Keep the two consistent, or you will allow an agent at the edge that your robots.txt tells it to stay away from.

Reviewed against the primary sources on 27 August 2026. These standards move quickly, so check the linked specs before you ship.