Verifies a robots.txt — and advises how to write it starting from the traffic your site actually receives.

It comes out of a concrete problem: some crawlers were taking 5% of a site's requests without bringing a single visit, and the question «what do I put in robots.txt?» had no data-backed answer. The lists you find online are generic; a site's traffic is specific.

GitHub Design rationale Contributing For coding agents pkg.go.dev

Install

# Homebrew — macOS and Linuxbrew. Homebrew 6+ asks to trust a third-party tap the first
# time: brew trust --cask Allan-Nava/tap/robotsmith
brew install --cask Allan-Nava/tap/robotsmith

# Docker — scratch image, runs as nobody
docker run --rm ghcr.io/allan-nava/robotsmith check example.com
docker run --rm -v "$PWD:/w:ro" ghcr.io/allan-nava/robotsmith lint /w/robots.txt

# Go
go install github.com/Allan-Nava/robotsmith@latest

Go 1.24, no external dependencies — a single static binary, fit for a CI pipeline. Prebuilt binaries for linux and macOS (amd64/arm64) with checksums come with every release; the container image is scratch plus the binary and a CA bundle, with no shell and no package manager inside it.

Why writing four lines by hand is not enough

The risk is asymmetric. Blocking a scraper produces no visible effect; accidentally blocking Googlebot drops the site out of the index within weeks — and you notice once the traffic is already gone. So a robots.txt has to be verified, and the file can be right and still not work:

⚠️ A cache-buster does not help you find that last one: if the query string is not part of the cache key — the normal setup for a static file — even ?cb=123 returns the old copy. The only reliable comparison is public against origin, which is what --origin does.

The algorithm that advises the file

advise does not apply a canned list: it reads the traffic and decides case by case. For every observed crawler it asks one question — does this traffic bring me anything? — and the answer determines the policy.

The non-obvious choices, and why

• ChatGPT-User and OAI-SearchBot are not training scrapers. The first is a fetch triggered by a person, the second feeds search results. Blocking them costs visibility without removing load, so the policy is ALLOW even though the name suggests otherwise.
• Google-Extended and Applebot-Extended are not User-Agents: they exist only for robots.txt (they mean «do not use my content for training»). A UA-based rule for them does nothing; here they are emitted into the file, where they mean something.
• An unknown crawler is not automatically useless. Above 3% of the requests it is proposed active, with the number next to it — the burden of proof flips, it costs too much to ignore; below that it is proposed commented out, because a wrong block is invisible.
• Pre-existing rules are carried over unchanged. They were written for that site and the algorithm does not know why: if the previous file had orphaned rules (after a blank line) they are recovered and flagged, not lost.
• The closing warning accuses nobody. A browser User-Agent with a high share is normal: behind one string there are thousands of people. The tool only says that on that slice it cannot reach a verdict, because telling a person from a disguised scraper needs the per-IP rate — which advise does not look at.

Reviewing the advice, not re-reading the file

Applying advice is a decision someone has to sign off on, and the risk is asymmetric: nobody notices a wrongly blocked scraper, everybody notices a wrongly blocked Googlebot. So --diff answers the reviewer's actual question — what would change? — instead of handing over a second sixty-line file to compare by eye.

$ robotsmith advise --log access.log --current ./robots.txt --diff
+ ChatGPT-User           Allow: /       13.02%  fetch triggered by a person: blocking it costs visibility, not load
! Googlebot              Allow: /       11.20%  brings visits: blocking it costs real traffic
                         ⚠️  the file currently says: Disallow: /
+ Bytespider             Disallow: /     9.13%  takes content for training without bringing visits
+ YisouSpider            Disallow: /    18.20%  unrecognised but heavy crawler (18.2%): to be reviewed
~ YandexBot              Disallow: /search       written for this site, kept verbatim: this tool does not know why it is there

+ added   ! the file says the opposite   ~ kept verbatim (not advised here)

The ~ line matters as much as the others: a group written by a person for a crawler the logs never showed is carried over verbatim and named out loud, because silence is how a rule gets lost.

Example

$ robotsmith advise --ua-counts ua.txt --current https://example.com/robots.txt --host example.com
Observed 182761 requests from 60 distinct user-agents.

What I advise, and why:
  ALLOW    ChatGPT-User             4.49%  fetch triggered by a person: blocking it costs visibility, not load
  ALLOW    Googlebot                3.86%  brings visits: blocking it costs real traffic
  BLOCK    Bytespider               3.15%  takes content for training without bringing visits
  REVIEW   YisouSpider              6.27%  unrecognised but heavy crawler (6.3%): to be reviewed

The advised blocks touch 11.4% of the observed requests.

⚠️ 16.0% of the traffic declares a browser User-Agent: this command cannot tell whether there are
   people or disguised scrapers behind it, because it does not look at the per-IP rate.

The file itself goes to stdout, every message to stderr — so robotsmith advise … > robots.txt produces a clean file:

# robots.txt generated by robotsmith (github.com/Allan-Nava/robotsmith)
# Based on 62,970 observed requests. The advised blocks touch 28.8% of them.
#
# ⚠️ robots.txt is a REQUEST, not a control: whoever disguises itself as a browser ignores it.
#    For those you need a request cap or a WAF.

# ── Allowed on purpose: they bring visits ───────────────────────────
# ai-user — fetch triggered by a person: blocking it costs visibility, not load
User-agent: ChatGPT-User
Allow: /

# ── Blocked: they take content without bringing visits ──────────────
User-agent: Bytespider
Disallow: /
# ⚠️ Unrecognised but heavy crawler: 18.2% of the requests (11,460).
User-agent: YisouSpider
Disallow: /

# ── General rules ───────────────────────────────────────────────────
User-agent: *
Disallow: /admin/
# ⚠️ The lines below were ORPHANED in the previous file (after a blank line
#    inside the group): a strict parser ignored them. Recovered here.
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

Longest match, not first match

The parser is written in-house on purpose. The historical implementations (including Python's stdlib robotparser) apply the first matching rule; RFC 9309 — and Google — use the longest match, with Allow winning ties:

User-agent: *
Allow: /
Disallow: /login

/login comes out allowed under first-match and disallowed under the RFC. A tool that advises what to write has to model how real crawlers behave, so it implements the second one — and lint still flags that layout, because not every crawler is compliant.

Usage

# verify: the file is there, it is fresh, it says the right thing (32 cases)
robotsmith check example.com --origin https://internal.origin/robots.txt

# structural defects of a local or remote file
robotsmith lint ./robots.txt

# advice from the logs (HAProxy or nginx), preserving the current rules
robotsmith advise --log access.log \
  --current https://example.com/robots.txt --host example.com --out robots.txt

# rotated logs, straight off the pipe — gzip is detected by content, not by file name
zcat access.log.*.gz | robotsmith advise --log - --host example.com

# or from a count you already have
awk '{n=split($0,q,"\""); if(n>=5) print q[4]}' access.log | sort | uniq -c | sort -rn > ua.txt
robotsmith advise --ua-counts ua.txt

Flags

Log formats

--log reads the user-agent out of whatever the edge writes. Three shapes are covered by a fixture each, because the failure mode here is silent: a format the parser does not understand produces no crash and no empty output, just advice derived from a fraction of the traffic.

The last row is the one that matters. HAProxy's default httplog carries no user-agent at all, so an advice built on it would be built on nothing — advise says so on stderr instead of printing a short list that looks complete. Add the capture and the traffic appears:

capture request header User-Agent len 200

A report you can send

The person who signs off on blocking a third of the crawler traffic is usually not the person who ran the command, and sixty lines of terminal output do not survive that trip.

robotsmith advise --log access.log --host example.com --report advice.html
robotsmith advise --log access.log --host example.com --report advice.pdf

One layout model, two backends. The bars and rows are computed once; HTML emits inline SVG, PDF emits drawing operators. Rendering it twice a second way is the trap — two renderers drift, and then the numbers in the deck disagree with the numbers in the tool — so a test asserts every share in the HTML appears in the PDF. The HTML is self-contained (inline CSS and SVG, no JavaScript, nothing fetched: it gets emailed and opened offline); the PDF needs no headless browser, keeps its text selectable, and is byte-identical for the same input so it can be diffed in CI.

Bars sorted by volume with direct labels, and never a pie chart: the argument is "this one is worth more than those ten", which a pie destroys at exactly the sizes that matter. Every bar carries its policy word and a glyph, so identity never rests on colour alone — red and green are one colour to a deuteranope, and a printed page has none.

Direction, not just a snapshot

A share says what a crawler costs today; the decision usually hinges on where it is going. 0.4% flat for a year and 0.4% quadrupling this month get the same answer from a share alone — and only the second is worth acting on. --compare takes a stored --json document from an earlier run, so there is no new file format to learn.

$ robotsmith advise --log access.log --compare last-month.json
Against the previous run (182,761 requests):
  grew      Bytespider              1.00% →  4.10%  (×4.1)
  shrank    GPTBot                  2.00% →  0.80%  (×0.4)
  appeared  NewSpider               0.00% →  2.00%
  vanished  OldBot                  3.00% →  0.00%  — a rule still blocking it does nothing

A crawler too small to earn a line but growing past ×2 is raised for review with the growth as its reason. Vanished ones are reported for the opposite reason: a rule that no longer does anything is invisible until somebody audits the file.

Your own policy, written down

The classification table is an opinion. --policy makes disagreeing cheap and, more importantly, written down — first match wins, as in the built-in table:

{
  "schema": "robotsmith.policy/1",
  "rules": [
    {"pattern": "bytespider", "policy": "allow", "why": "we license our content to them"},
    {"pattern": "internal-indexer", "family": "search", "policy": "allow"},
    {"pattern": "semrush", "policy": "ignore", "why": "our SEO team runs it"}
  ]
}

The reasoning then names the rule and the file, so the decision can be argued with: ALLOW Bytespider 13.02% we license our content to them [policy.json: bytespider].

⚠️ The file is strict: an unknown key, an unknown value, a broken pattern or a rule an earlier one already covers is an error, not a warning — a policy file that half-works reads like an applied decision and is not one.

...and verified after it is deployed

The same file closes the loop: advise --policy decides, you deploy, and check --expect verifies what the origin actually serves — quoting the deployed file's own lines, and saying when an answer was merely inherited from * rather than written for that crawler.

$ robotsmith check example.com --expect policy.json
  ✅ bytespider               must be allowed — `Allow: /` (line 2, group `bytespider`)
  ⛔ gptbot                   must be blocked — no rule in the file mentions it, inherited from `*`
                             because: takes without giving

⚠️ It replaces the built-in cases: if you stated your policy, yours is the contract — otherwise a deliberate exception would fail a built-in case forever, and a check that is red by design is one people stop reading.

In a GitHub workflow

- uses: Allan-Nava/robotsmith@v0
  with:
    command: lint
    target: robots.txt

Findings come back as annotations on the offending lines, so a defect shows up in the diff view of the pull request instead of in a log nobody scrolls. command takes check, lint, advise or crawlers; fail-on-findings: false reports without failing the job; the step exposes exit-code and output for a later step to act on. The action downloads the release binary for the runner and verifies it against the release's own checksums.txt before running it — a CI step that curls an unverified binary and executes it is a supply chain nobody audited.

@v0 is a moving tag: the release workflow force-moves it onto every stable release, so a pipeline that uses it keeps getting the newest one without a bump. Pin @v0.5.1 instead when you want a build to be reproducible. ⚠️ v0 is a 0.x line: it moves across minors, and a 0.x minor is allowed to change behaviour — the exit codes and the flags are the part this repo treats as a contract.

There is one more input, binary, and it exists for exactly one caller: this repository's own CI, which has to test action.yml against the branch under review rather than against whatever was last published. ⚠️ Pointing it at a path skips the download and the checksum verification, so it is the wrong answer everywhere else — pin version instead.

Without the action, any CI gets the same thing: robotsmith lint robots.txt --format github.

Machine-readable output

--json puts one document on stdout and nothing else there; the exit code is unchanged. Consumers pin the schema field, which is the only stable promise — the prose is free to be reworded, and a document is never edited in place: a breaking change bumps its number.

robotsmith lint ./robots.txt --json | jq -r '.findings[] | "::error line=\(.line)::\(.message)"'
robotsmith advise --ua-counts ua.txt --json | jq -r '.decisions[] | select(.policy=="block") | .name'

Exit codes

They are a contract: in CI a robots.txt that loses rules or stays stuck in a cache becomes a red build instead of a late discovery. Telling 1 from 4 matters — "the file says the wrong thing" and "the file is not there" call for different actions.

⛔ What this tool does not do

• It does not tell you whether crawlers obey. robots.txt is a request, not a control. That is measured from the logs over the following days.
• It does not stop whoever disguises itself as a browser. By construction: there is no token to write. That calls for a per-IP request cap or a WAF.
• It does not look at the per-IP rate, do fingerprinting, or verify reverse DNS.
• It does not handle Crawl-delay: Google ignores it, and suggesting it would give false confidence.
• It has no server, daemon or database. It is a command that reads, decides and prints.

How it is built