What Is Content Signals and How Do You Implement It in 2026?

Q: What is the difference between Content-Signal and Disallow?

Disallow says whether a crawler may fetch a URL at all. Content-Signal says what it may do with the content once it has fetched it: index it, ground a generative answer on it, or train on it. The two interact in one direction only: a path that cannot be crawled carries no usage preference, so blocking a path makes its signals moot. Both are requests, and neither stops a crawler that ignores robots.txt.

Q: How does Content-Signal relate to the IETF Content-Usage rule?

They are two different surfaces for the same idea. Content-Signal is Cloudflare's CC0 policy, with the signals search , ai-input and ai-train taking yes or no . The IETF AI Preferences working group is standardising a Content-Usage robots.txt rule and HTTP header with its own vocabulary, including train-ai , taking y or n . As of August 2026 those are Internet-Drafts, not RFCs, and the vocabulary document still states that its definitions do not reflect working group consensus, so publishing both lines is the safe move.

Content Signals put three separate yes/no answers in your robots.txt: may this page be indexed for search, used as live AI input, or used to train a model. You will leave with the exact syntax, three working configurations, and a clear line between what the directive states and what it actually enforces.

Check Your Site Now Browse the Glossary

CategoryCrawler access, licensing & indexing StatusActive, Cloudflare-led Maintained byCloudflare Glippy checkMachine Readability (category 6)

Content Signals (Content-Signal directives in robots.txt) are comma-separated yes / no preferences that say, for one User-agent group, whether your content may be used for search, ai-input or ai-train. Cloudflare published the policy on 24 September 2025 under a CC0 licence, so any site can copy the wording, and applies a default of search=yes, ai-train=no to the millions of domains using its managed robots.txt. The directive states terms for crawler operators to honour; it is not a technical block, and enforcement still comes from Disallow rules, WAF rules and bot management.

Why Content Signals matters for AI visibility

Blocking an AI crawler with Disallow is a single switch covering three very different bargains. ai-train is the one most publishers want to refuse: your archive becomes model weights and you get nothing back. ai-input is the retrieval step that grounds a generative answer on your page and normally carries a citation, which is the traffic GEO exists to win. search is the classic index-and-link deal. A blanket block on GPTBot or ClaudeBot throws away the second and third to stop the first. Content Signals separates them, so a GEO-sane configuration is usually search=yes, ai-input=yes, ai-train=no. The CC0 preamble that ships with the policy also frames any no as an express reservation of rights under Article 4 of EU Directive 2019/790, which is why you copy the comment block and not only the one directive.

Be clear-eyed about who reads it. As of August 2026 no major crawler operator or model vendor has publicly committed to honouring Content-Signal, and in July 2026 Google's John Mueller said he was not aware of any crawler or LLM using the directive. In parallel the IETF AI Preferences working group is standardising a different surface: a Content-Usage robots.txt rule and HTTP header with its own vocabulary (train-ai, search) and y / n values, at draft -05 and -07 respectively in August 2026 with the vocabulary section still marked as not reflecting working group consensus. So the honest position is this: publishing Content Signals is cheap, machine-readable and creates a dated written record of your terms, and it is worth doing on that basis, but it belongs next to something that actually enforces, not instead of it.

Where the spec lives

The policy text and the generator are Cloudflare's; the standards-track work happens at the IETF. Read the first two before you paste anything into a production robots.txt.

contentsignals.org - the reference site and policy generator. Pick a preset, get the full CC0 comment block plus your directive, including the per-bot and per-path forms.
Cloudflare: Giving users choice with Cloudflare's new Content Signals Policy - the 24 September 2025 launch post. Definitions of the three signals, the CC0 release, and the reason ai-input is left unset in the managed default.
Cloudflare Docs: robots.txt setting - the exact managed file Cloudflare prepends, the default line, the experimental use= extension, and the note that Search Console may report "Syntax not understood".
Cloudflare Docs: Markdown for Agents - the one place a Content-Signal HTTP response header is documented as being read: your origin header wins, and a default is added when you send none.
IETF draft-ietf-aipref-vocab - the standards-track vocabulary for AI usage preferences. Read section 3.2 for the explicit statement that a vocabulary does not make anyone follow it.
IETF draft-ietf-aipref-attach - defines the Content-Usage robots.txt rule and HTTP header, including the path-prefix form and the rule that uncrawlable paths carry no preference at all.

Three ways to implement Content Signals

Start with the static robots.txt block: it is the canonical surface and takes ten minutes. Move to grouped and path-scoped signals when different crawlers or different sections of the site deserve different answers. Emit the HTTP response header when you cannot edit robots.txt for every hostname, or when you want the preference to travel with each response rather than sitting in a file a crawler may never fetch.

Static robots.txt with the CC0 policy preamble

The minimum viable version, and the one most sites should ship. The comment block is the CC0 policy text from contentsignals.org: copy it verbatim, because it is what defines the three terms and turns any no into an express reservation of rights rather than an undocumented flag.

text/robots.txt

# As a condition of accessing this website, you agree to abide by the
# following content signals:

# (a)  If a content-signal = yes, you may collect content for the
#      corresponding use.
# (b)  If a content-signal = no, you may not collect content for the
#      corresponding use.
# (c)  If the website operator does not include a content signal for a
#      corresponding use, the website operator neither grants nor restricts
#      permission via content signal with respect to the corresponding use.

# The content signals and their meanings are:

# search: building a search index and providing search results (e.g., returning
#         hyperlinks and short excerpts from your website's contents). Search
#         does not include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
#           augmented generation, grounding, or other real-time taking of
#           content for generative AI search answers).
# ai-train: training or fine-tuning AI models.

# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Sitemap: https://www.example.com/sitemap.xml

What this does: every crawler that reads robots.txt is told it may index and link to your pages and may ground a generative answer on them, but may not use them to train or fine-tune a model. Leaving Allow: / in place matters: a path a crawler is not allowed to fetch carries no usage preference for it to honour.

Per-crawler groups, path-scoped signals and an AIPREF fallback

Use this once one answer no longer fits the whole site. A Content-Signal line belongs to the User-agent group it sits in, and an optional path prefix before the pairs narrows it further. The trailing Content-Usage lines carry the same intent in the IETF vocabulary, so a parser that implements the draft and not Cloudflare's policy still gets an answer.

text/robots.txt

# Group 1: everyone not matched by a more specific group below.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Content-Signal: /press/ search=yes, ai-input=yes, ai-train=yes
Content-Signal: /research/ search=yes, ai-input=no, ai-train=no
Content-Usage: train-ai=n
Content-Usage: /press/ train-ai=y
Allow: /
Disallow: /account/

# Group 2: a bulk training crawler you do not want at all. The Disallow is
# the instruction that matters; the signal states the terms behind it.
User-agent: CCBot
Content-Signal: search=no, ai-input=no, ai-train=no
Content-Usage: train-ai=n
Disallow: /

# Group 3: a partner crawling under a separate written licence.
User-agent: ExamplePartnerBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Content-Usage: train-ai=y
Allow: /

What this does: press releases are cleared for every use including training, research pages are index-only, and everything else stays citable but off limits for training. Note the value sets differ on purpose: Cloudflare's Content-Signal takes yes and no, while the IETF Content-Usage rule takes y and n, so do not mix them.

Send Content-Signal as an HTTP response header

A per-response version for hostnames whose robots.txt you do not control, or where the preference should travel with the bytes. Cloudflare's Markdown for Agents documents this header as authoritative when your origin sets it. This example also uses Cloudflare's experimental use= extension, which describes what a crawler may keep afterwards: immediate (store nothing), reference (index, excerpt, link back) or full (summarise and reproduce).

nginx/etc/nginx/conf.d/content-signal.conf

# Included from the http block. Sends usage preferences on every response,
# so a crawler that never fetches /robots.txt still sees them.

map $uri $content_signal {
    default   "search=yes, ai-input=yes, ai-train=no, use=reference";
    ~^/press/ "search=yes, ai-input=yes, ai-train=yes, use=full";
    ~^/app/   "search=no, ai-input=no, ai-train=no, use=immediate";
}

server {
    listen 443 ssl;
    server_name www.example.com;

    # "always" sends the header on 4xx and 5xx responses too, not just 2xx.
    add_header Content-Signal $content_signal always;

    location / {
        # add_header inside a location replaces the inherited set,
        # so repeat it in every location that adds headers of its own.
        add_header Content-Signal $content_signal always;
        try_files $uri $uri/ =404;
    }
}

What this does: each response carries its own usage terms, so an agent fetching a single URL has the preference in hand without a second request. Generate the header values and the robots.txt groups from one source, because a crawler that sees both and finds them contradictory has no defined tie-break to fall back on.

Implementation guidelines

The failure modes here are structural rather than syntactic: the line parses fine and still says nothing.

Put the signal inside a group. A Content-Signal line applies to the User-agent group it sits in. A line placed before the first User-agent is orphaned, and RFC 9309 requires parsers to merge groups matching the same token exactly, so keep one group per token instead of repeating User-agent: *.
Disallow beats the signal, and does a different job. A path a crawler may not fetch carries no usage preference at all, so Disallow: /research/ plus ai-train=no for the same path is redundant for compliant crawlers and irrelevant for the rest. Decide per path whether you want access control or usage terms.
Ship the preamble, not just the directive. The CC0 comment block is what defines the three terms and states that a no is an express reservation of rights under Article 4 of EU Directive 2019/790. Without it you have a flag with no published meaning attached to it.
Omit a signal rather than guessing. An unset signal means you neither grant nor restrict that use, which is a real and sometimes correct position. Cloudflare deliberately leaves ai-input out of its managed default for exactly this reason.
Price in what ai-input=no costs. It asks answer engines not to ground on the page, which is the same retrieval step that produces AI citations and referral traffic. Reserve it for content where the licence matters more than the visibility, and never apply it sitewide as a reflex.
Expect Search Console to complain. Google Search Console can report "Syntax not understood" for Content Signals and other newer robots.txt directives. Cloudflare reports no observed effect on crawl rate or SEO, so do not "fix" it by deleting the line.
Remember the caching lag. Preference changes only bite after a crawler refetches robots.txt and then refetches the affected resource, and robots.txt caching under RFC 9309 can be up to 24 hours. Treat an edit as taking days, not minutes, and keep the header and the file in version control together.

Do this, not that

Do

Copy the CC0 policy preamble from contentsignals.org verbatim, comment characters and capitals included.
Keep search=yes and ai-input=yes if you want to be quoted in AI answers, and use ai-train=no for the reservation.
Repeat the signal in every User-agent group you define, including the ones you also Disallow.
Pair the signal with something that enforces: a WAF rule, bot management, verified-bot checks or a paywall.

Do not

Do not place Content-Signal above the first User-agent: line and assume it applies site-wide.
Do not write ai-train=n. Content-Signal takes yes and no; y and n belong to the IETF Content-Usage rule.
Do not invent extra signal names. Only search, ai-input and ai-train are defined, with use= as a Cloudflare extension.
Do not present ai-train=no to stakeholders as proof a model was not trained on you; nothing in robots.txt is auditable.

How Glippy checks this

Glippy scores Content Signals inside Machine Readability (category 6). It reads the site's raw robots.txt, matches any line starting with Content-Signal: case-insensitively, strips trailing comments, then splits the value on commas into key/value pairs and reports what it found. A directive that is present and not restrictive earns the full bonus for the check; a directive containing ai-input=no is reported as an informational note instead, flagging that you are asking AI engines not to use the page for live answers. If no directive exists at all the check is informational with no maximum score attached, so an absent signal never costs you points: this is an emerging standard and Glippy only rewards it. One parsing caveat worth knowing: the check reads every Content-Signal line in the file rather than resolving user-agent groups, so a path-scoped line is reported with its path prefix attached to the first key.

Check your Content Signals setup

Glippy runs 240+ checks across 16 categories on any page, including Machine Readability (category 6). No sign-up required.

Try the Glippy Chrome extension

Frequently asked questions

Does any AI company actually honour Content-Signal?

No major crawler operator or model vendor has publicly committed to reading it. In July 2026 Google's John Mueller said he was not aware of any crawler or LLM using the content-signal directive. Cloudflare applies the defaults to millions of managed robots.txt files, so the signal is widely published but not demonstrably widely read. Publish it as a dated, machine-readable statement of your terms, and use bot management or a WAF rule for anything you need enforced.

What is the difference between Content-Signal and Disallow?

Disallow says whether a crawler may fetch a URL at all. Content-Signal says what it may do with the content once it has fetched it: index it, ground a generative answer on it, or train on it. The two interact in one direction only: a path that cannot be crawled carries no usage preference, so blocking a path makes its signals moot. Both are requests, and neither stops a crawler that ignores robots.txt.

How does Content-Signal relate to the IETF Content-Usage rule?

They are two different surfaces for the same idea. Content-Signal is Cloudflare's CC0 policy, with the signals search, ai-input and ai-train taking yes or no. The IETF AI Preferences working group is standardising a Content-Usage robots.txt rule and HTTP header with its own vocabulary, including train-ai, taking y or n. As of August 2026 those are Internet-Drafts, not RFCs, and the vocabulary document still states that its definitions do not reflect working group consensus, so publishing both lines is the safe move.

Will adding Content Signals hurt my Google rankings?

There is no evidence the syntax does. Google Search Console may occasionally report "Syntax not understood" for Content Signals and other newer robots.txt directives, and Cloudflare states it has observed no impact on crawling rates or SEO as a result. What can cost you visibility is the value you pick rather than the line itself: search=no asks search engines to stop indexing you, and ai-input=no asks answer engines not to ground on your page, which is the retrieval step that produces AI citations.

Reviewed against the primary sources on 27 August 2026. These standards move quickly, so check the linked specs before you ship.