What Is NLWeb and How Do You Implement It in 2026?

Q: Who governs NLWeb now that it has left the microsoft GitHub organisation?

The repository moved from microsoft/NLWeb to the nlweb-ai organisation in July 2025, and the specification is published on the project's own site. Microsoft still drives the work and project support still routes to a Microsoft address, so the move was a rehome rather than a handover. NLWeb has not been contributed to the Linux Foundation's Agentic AI Foundation, which is where MCP, goose and AGENTS.md went in December 2025. The code is MIT licensed either way.

NLWeb puts a natural-language /ask endpoint in front of the Schema.org markup or RSS feed you already publish. This page shows you how to index your own content, what infrastructure it actually needs, and how to expose the same instance over MCP.

Check Your Site Now Browse the Glossary

CategoryAgent & AI protocols StatusOpen source, active Maintained byMicrosoft Glippy checkAgent Interactivity (category 10)

NLWeb (Natural Language Web) is an open specification and Python reference implementation that puts a natural-language query endpoint in front of a website's own content. You point it at the Schema.org markup or the RSS feed you already publish, it indexes that into a vector store, and it answers questions at /ask with Schema.org typed JSON instead of prose. Every instance also speaks the Model Context Protocol at /mcp, so the same index is callable as an agent tool without a second integration.

Why NLWeb matters for AI visibility

Most GEO work is passive. You publish markup, and then you wait for someone else's crawler to fetch it, on someone else's schedule, through whatever cache sits in between. NLWeb inverts that arrangement: you run the retrieval yourself, over your own live data, and an answer comes back grounded in what you actually have published or in stock today. Because the results are typed Schema.org objects rather than a paragraph of text, a client can render them, filter them, or act on them, and it can attribute them back to the url field on each item. The markup you already ship for rich results is the input, so the marginal cost is deployment rather than content.

The reason the deployment is worth doing is the MCP surface. The same process that serves your site search answers tools/list and tools/call on /mcp, which makes your catalogue addressable by any MCP client. The honest caveat is discovery: NLWeb defines no .well-known path and no link relation, and version 0.55 of the spec only fixes the endpoint names. Agents find your endpoint because you registered it with a client, listed it in your llms.txt, or because a platform such as Wix mounted it for you. Publishing the endpoint is step one; telling something it exists is step two, and nothing in the spec does that for you.

Where the spec lives

The canonical specification is published at nlweb.ai (currently version 0.55), but at the time of writing that host is serving an expired TLS certificate, so treat the repository as the reliable primary source. Note also that the project moved out of the microsoft GitHub organisation in July 2025, so older tutorials point at a redirect.

nlweb-ai/NLWeb - the Python reference implementation and the project's real home since the move off the microsoft org. Start here for anything the spec text leaves ambiguous.
REST API reference - the exact parameters the server accepts on /ask and the MCP methods handled on /mcp. Read it before you write a client.
Hello world guide - the shortest path from clone to a running instance, including which three config files you have to change from their Azure defaults.
config_retrieval.yaml - every supported backend with its exact key names and environment variables. The fastest way to see whether your existing database is already a candidate.
Scraping and crawl tools - the incremental crawler that walks your sitemap and pulls the Schema.org JSON-LD out of each page. This is the piece that reuses markup you already have.
Wix NLWeb documentation - a worked example of a managed deployment, useful if you want to see what a hosted endpoint URL and its limits look like before running your own.

Three ways to implement NLWeb

The first example is the proof of concept: index your existing feed or markup into the bundled local vector store and serve /ask from your laptop. The second is the part nobody writes about, the configuration you have to change to move that off a single machine and onto a real backend with a real model provider. The third is the consumption side, showing the same instance answering both an HTTP query and an MCP tool call, which is what you hand to an agent.

Minimum viable: index what you already publish and serve /ask locally

Use this to find out, in under an hour, whether your existing structured data is rich enough to answer questions. Nothing here touches your production site: the crawler reads your public pages, and everything it extracts lands in a local file-backed vector store.

bashterminal

# 1. Get the reference implementation and install its dependencies.
git clone https://github.com/nlweb-ai/NLWeb.git
cd NLWeb
python -m venv myenv
source myenv/bin/activate          # Windows: myenv\Scripts\activate
pip install -r AskAgent/python/requirements.txt
cp .env.template .env              # put OPENAI_API_KEY in here

# 2. The three config files ship with Azure defaults. Change them to:
#    config/config_llm.yaml        preferred_endpoint: openai
#    config/config_embedding.yaml  preferred_provider: openai
#    config/config_retrieval.yaml  write_endpoint: qdrant_local

# 3. Prove the keys and the store resolve before you index anything.
cd AskAgent/python
python testing/check_connectivity.py

# 4a. Already publish an RSS or Atom feed? Load it directly.
python -m data_loading.db_load https://example.com/feed.xml example-blog

# 4b. Already publish Schema.org JSON-LD? Crawl the sitemap and extract it.
python -m scraping.incrementalCrawlAndLoad example.com --max-pages 500 --database qdrant_local

# 5. Serve it. /ask, /mcp, /sites and /who are all on the same port.
python app-aiohttp.py
curl "http://localhost:8000/ask?query=what+do+you+sell&site=example-blog&streaming=false"

What this does: step 4b is the one that matters for GEO, because it turns the JSON-LD you already emit for rich results into vectors an agent can search semantically. The crawler is resumable and keeps a crawl_status.json, so you can stop it and restart it against a large sitemap without losing work.

Production backend: a shared vector store and your own model provider

Move here the moment more than one person needs to query the endpoint. The local Qdrant store is a file path bound to one process, so it cannot be shared between instances or survive a stateless redeploy. NLWeb also supports Azure AI Search, Elasticsearch, OpenSearch, Milvus, Snowflake Cortex Search and Cloudflare AutoRAG through the same file; Postgres is shown here because most sites already have one.

yamlconfig/config_retrieval.yaml, config_llm.yaml, config_embedding.yaml

# ---------- config/config_retrieval.yaml ----------
# write_endpoint decides where db_load and the crawler write their vectors.
write_endpoint: postgres

endpoints:
  # Switch the laptop store off once you leave the laptop.
  qdrant_local:
    enabled: false
    database_path: "../data/db"
    index_name: nlweb_collection
    db_type: qdrant

  postgres:
    enabled: true
    # postgresql://HOST:PORT/DATABASE?user=USERNAME&sslmode=require
    api_endpoint_env: POSTGRES_CONNECTION_STRING
    api_key_env: POSTGRES_PASSWORD
    index_name: documents
    db_type: postgres

# ---------- config/config_llm.yaml ----------
preferred_endpoint: openai

endpoints:
  openai:
    api_key_env: OPENAI_API_KEY
    api_endpoint_env: OPENAI_ENDPOINT
    llm_type: openai
    models:
      high: gpt-4.1
      low: gpt-4.1-mini

# ---------- config/config_embedding.yaml ----------
# This model must match the one used at index time or every query misses.
preferred_provider: openai

providers:
  openai:
    api_key_env: OPENAI_API_KEY
    model: text-embedding-3-small

What this does: these three files are the whole infrastructure decision. You need a Python process, a vector store, an embedding model and a chat model, and NLWeb reads which of each to use from here rather than from code. Watch the key names: the LLM file uses preferred_endpoint while the embedding file uses preferred_provider, and getting them the wrong way round fails quietly.

Query it from an agent: the structured /ask body and the MCP endpoint

Once the index is populated, the same process answers two audiences. Your own front end talks HTTP to /ask; an MCP client talks JSON-RPC to /mcp. Send streaming as false while you are testing with curl, and set Accept: text/event-stream when you want Server-Sent Events instead.

bashterminal

# The spec's HTTP binding: POST /ask with a structured v0.55 body.
curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{
    "query":   { "text": "waterproof jackets under 200 euro", "site": "example-shop" },
    "context": { "prev": ["hiking gear"] },
    "prefer":  { "streaming": false, "mode": "list, summarize" },
    "meta":    { "version": "0.55" }
  }'

# Same process, MCP transport. Ask what the endpoint exposes:
curl -X POST http://localhost:8000/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

# Then call the ask tool. The reference server takes a flat string here,
# not the nested query object shown in Appendix A of the spec.
curl -X POST http://localhost:8000/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 2,
    "method": "tools/call",
    "params": {
      "name": "ask",
      "arguments": {
        "query": "waterproof jackets under 200 euro",
        "site": ["example-shop"],
        "generate_mode": "list"
      }
    }
  }'

What this does: the first call returns a _meta block plus a results array of Schema.org typed items, each with a url back to the page it came from, which is the citation an agent needs. The MCP calls prove the same index is reachable as a tool: tools/list returns ask, list_sites and, if enabled, who.

Implementation guidelines

These are the failures that show up after the demo works, when the endpoint is live and someone else is calling it.

Keep the embedding model identical at index and query time. The loader and the server both read config_embedding.yaml. Change model or preferred_provider after you have indexed and queries return nothing without erroring, because you are searching a vector space that no longer matches. Re-run the load rather than patching the config.
Run the connectivity check before every load, not just the first. python testing/check_connectivity.py confirms the LLM, embedding and retrieval endpoints all resolve. Running it before a large crawl saves you from discovering an expired key after 500 pages of fetching.
Rebuild the index on the same schedule as your sitemap. An NLWeb endpoint that answers from last quarter's catalogue is worse than no endpoint, because the answer looks authoritative. The crawler is incremental and resumable, and --reprocess recomputes embeddings from HTML you already have on disk.
Partition with site names and restrict what is exposed. The second argument to db_load is the site name, and callers select it with the site parameter. Use the sites key in config_nlweb.yaml to limit which partitions the endpoint will serve, and never load a partition you would not publish as a public page.
Put authentication and rate limiting in front of it yourself. The specification scopes both out as transport-layer concerns, and the reference server ships neither. Every /ask call runs model inference you pay for, so put it behind a reverse proxy that returns 429 with Retry-After before you expose it publicly.
Send meta, read _meta. Version 0.55 made the asymmetry explicit: requests carry a meta object with version, responses carry _meta with response_type. Handle all four response types, because answer is only one of them; elicitation, promise and failure all arrive with HTTP 200.
Do not assume the reference server matches the spec's tool schema. Appendix A of the spec defines the MCP ask tool with a nested query object, while the shipping Python server advertises a flat query string plus site and generate_mode. Call tools/list against the instance you are targeting and build against what it actually returns.

Do this, not that

Do

Index the same Schema.org objects your pages already emit, so the /ask answer and the page a user lands on cannot disagree.
Set write_endpoint in config_retrieval.yaml before you load anything, since it decides which store the crawler writes to.
Link your /ask or /mcp URL from llms.txt and from your own documentation, because NLWeb has no discovery convention of its own.
Test with streaming=false first, then switch to Accept: text/event-stream and handle the start, result, error and complete events.

Do not

Do not leave qdrant_local enabled in production. It is a file store bound to one disk path and one process, so it will not scale past a single instance.
Do not expose /ask unmetered on a public hostname. Each call is an LLM request on your account, and the endpoint has no built-in quota.
Do not swap the embedding provider on a populated index. Reload the data instead, or every query silently returns an empty result set.
Do not treat NLWeb as a substitute for on-page markup. Crawlers still read the page, and the endpoint is an extra surface rather than a replacement.

How Glippy checks this

Glippy scores NLWeb under Agent Interactivity (category 10), alongside WebMCP, agent cards, Agent Skills and the other surfaces that let an agent do something with your site rather than only read it. The checks look for a reachable NLWeb endpoint and for the structured data that makes one worth building, since a site with no Schema.org markup and no feed has nothing to index. If you are working on this category, the Agent Interactivity checker shows the full list of signals it looks for.

Check your NLWeb setup

Glippy runs 240+ checks across 16 categories on any page, including Agent Interactivity (category 10). No sign-up required.

Try the Glippy Chrome extension

Frequently asked questions

Does NLWeb replace my Schema.org markup?

No, it consumes it. NLWeb's crawler walks your sitemap, extracts the JSON-LD already embedded in each page, embeds it and stores it in a vector database. If you delete the markup, the index has nothing to build from. The same applies to RSS and Atom: NLWeb reads the feed you publish rather than replacing it, so on-page structured data remains the foundation.

What infrastructure do I need to run NLWeb?

Four things: a Python process to run the server, a vector store, an embedding model and a chat model. The smallest working setup is the bundled local Qdrant file store plus one API key, which runs on a laptop. Production means a hosted store, with Azure AI Search, Postgres, Elasticsearch, OpenSearch, Qdrant, Milvus, Snowflake Cortex Search and Cloudflare AutoRAG all configurable in config_retrieval.yaml, plus a paid model endpoint. Budget for inference, because every query runs a model.

Is NLWeb the same thing as MCP?

They solve different halves of the problem. MCP is a transport for connecting a model to tools and says nothing about what a tool should do. NLWeb defines the query contract itself: an ask operation that takes a natural-language question and returns Schema.org typed results. The two meet because every NLWeb instance also serves MCP on /mcp, exposing ask, list_sites and optionally who as tools, so one deployment covers both an HTTP client and an MCP client.

Who governs NLWeb now that it has left the microsoft GitHub organisation?

The repository moved from microsoft/NLWeb to the nlweb-ai organisation in July 2025, and the specification is published on the project's own site. Microsoft still drives the work and project support still routes to a Microsoft address, so the move was a rehome rather than a handover. NLWeb has not been contributed to the Linux Foundation's Agentic AI Foundation, which is where MCP, goose and AGENTS.md went in December 2025. The code is MIT licensed either way.

Reviewed against the primary sources on 27 August 2026. These standards move quickly, so check the linked specs before you ship.