Verifies a robots.txt — and advises how to write it starting from the traffic your site actually receives.

It comes out of a concrete problem: some crawlers were taking 5% of a site's requests without bringing a single visit, and the question «what do I put in robots.txt?» had no data-backed answer. The lists you find online are generic; a site's traffic is specific.

GitHub Design rationale Contributing For coding agents pkg.go.dev

Install

# Homebrew — macOS and Linuxbrew. Homebrew 6+ asks to trust a third-party tap the first
# time: brew trust --cask Allan-Nava/tap/robotsmith
brew install --cask Allan-Nava/tap/robotsmith

# Docker — scratch image, runs as nobody
docker run --rm ghcr.io/allan-nava/robotsmith check example.com
docker run --rm -v "$PWD:/w:ro" ghcr.io/allan-nava/robotsmith lint /w/robots.txt

# Go
go install github.com/Allan-Nava/robotsmith@latest

Go 1.24, no external dependencies — a single static binary, fit for a CI pipeline. Prebuilt binaries for linux and macOS (amd64/arm64) with checksums come with every release; the container image is scratch plus the binary and a CA bundle, with no shell and no package manager inside it.

Why writing four lines by hand is not enough

The risk is asymmetric. Blocking a scraper produces no visible effect; accidentally blocking Googlebot drops the site out of the index within weeks — and you notice once the traffic is already gone. So a robots.txt has to be verified, and the file can be right and still not work:

⚠️ A cache-buster does not help you find that last one: if the query string is not part of the cache key — the normal setup for a static file — even ?cb=123 returns the old copy. The only reliable comparison is public against origin, which is what --origin does.

The algorithm that advises the file

advise does not apply a canned list: it reads the traffic and decides case by case. For every observed crawler it asks one question — does this traffic bring me anything? — and the answer determines the policy.

The non-obvious choices, and why

• ChatGPT-User and OAI-SearchBot are not training scrapers. The first is a fetch triggered by a person, the second feeds search results. Blocking them costs visibility without removing load, so the policy is ALLOW even though the name suggests otherwise.
• Google-Extended and Applebot-Extended are not User-Agents: they exist only for robots.txt (they mean «do not use my content for training»). A UA-based rule for them does nothing; here they are emitted into the file, where they mean something.
• An unknown crawler is not automatically useless. Above 3% of the requests it is proposed active, with the number next to it — the burden of proof flips, it costs too much to ignore; below that it is proposed commented out, because a wrong block is invisible.
• Pre-existing rules are carried over unchanged. They were written for that site and the algorithm does not know why: if the previous file had orphaned rules (after a blank line) they are recovered and flagged, not lost.
• The closing warning accuses nobody. A browser User-Agent with a high share is normal: behind one string there are thousands of people. The tool only says that on that slice it cannot reach a verdict, because telling a person from a disguised scraper needs the per-IP rate — which advise does not look at.

Reviewing the advice, not re-reading the file

Applying advice is a decision someone has to sign off on, and the risk is asymmetric: nobody notices a wrongly blocked scraper, everybody notices a wrongly blocked Googlebot. So --diff answers the reviewer's actual question — what would change? — instead of handing over a second sixty-line file to compare by eye.

$ robotsmith advise --log access.log --current ./robots.txt --diff
+ ChatGPT-User           Allow: /       13.02%  fetch triggered by a person: blocking it costs visibility, not load
! Googlebot              Allow: /       11.20%  brings visits: blocking it costs real traffic
                         ⚠️  the file currently says: Disallow: /
+ Bytespider             Disallow: /     9.13%  takes content for training without bringing visits
+ YisouSpider            Disallow: /    18.20%  unrecognised but heavy crawler (18.2%): to be reviewed
~ YandexBot              Disallow: /search       written for this site, kept verbatim: this tool does not know why it is there

+ added   ! the file says the opposite   ~ kept verbatim (not advised here)

The ~ line matters as much as the others: a group written by a person for a crawler the logs never showed is carried over verbatim and named out loud, because silence is how a rule gets lost.

Example

$ robotsmith advise --ua-counts ua.txt --current https://example.com/robots.txt --host example.com
Observed 182761 requests from 60 distinct user-agents.

What I advise, and why:
  ALLOW    ChatGPT-User             4.49%  fetch triggered by a person: blocking it costs visibility, not load
  ALLOW    Googlebot                3.86%  brings visits: blocking it costs real traffic
  BLOCK    Bytespider               3.15%  takes content for training without bringing visits
  REVIEW   YisouSpider              6.27%  unrecognised but heavy crawler (6.3%): to be reviewed

The advised blocks touch 11.4% of the observed requests.

⚠️ 16.0% of the traffic declares a browser User-Agent: this command cannot tell whether there are
   people or disguised scrapers behind it, because it does not look at the per-IP rate.

The file itself goes to stdout, every message to stderr — so robotsmith advise … > robots.txt produces a clean file:

# robots.txt generated by robotsmith (github.com/Allan-Nava/robotsmith)
# Based on 62,970 observed requests. The advised blocks touch 28.8% of them.
#
# ⚠️ robots.txt is a REQUEST, not a control: whoever disguises itself as a browser ignores it.
#    For those you need a request cap or a WAF.

# ── Allowed on purpose: they bring visits ───────────────────────────
# ai-user — fetch triggered by a person: blocking it costs visibility, not load
User-agent: ChatGPT-User
Allow: /

# ── Blocked: they take content without bringing visits ──────────────
User-agent: Bytespider
Disallow: /
# ⚠️ Unrecognised but heavy crawler: 18.2% of the requests (11,460).
User-agent: YisouSpider
Disallow: /

# ── General rules ───────────────────────────────────────────────────
User-agent: *
Disallow: /admin/
# ⚠️ The lines below were ORPHANED in the previous file (after a blank line
#    inside the group): a strict parser ignored them. Recovered here.
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

Longest match, not first match

The parser is written in-house on purpose. The historical implementations (including Python's stdlib robotparser) apply the first matching rule; RFC 9309 — and Google — use the longest match, with Allow winning ties:

User-agent: *
Allow: /
Disallow: /login

/login comes out allowed under first-match and disallowed under the RFC. A tool that advises what to write has to model how real crawlers behave, so it implements the second one — and lint still flags that layout, because not every crawler is compliant.

Usage

# verify: the file is there, it is fresh, it says the right thing (32 cases)
robotsmith check example.com --origin https://internal.origin/robots.txt

# structural defects of a local or remote file
robotsmith lint ./robots.txt

# advice from the logs (HAProxy or nginx), preserving the current rules
robotsmith advise --log access.log \
  --current https://example.com/robots.txt --host example.com --out robots.txt

# rotated logs, straight off the pipe — gzip is detected by content, not by file name
zcat access.log.*.gz | robotsmith advise --log - --host example.com

# or from a count you already have
awk '{n=split($0,q,"\""); if(n>=5) print q[4]}' access.log | sort | uniq -c | sort -rn > ua.txt
robotsmith advise --ua-counts ua.txt

Flags

Machine-readable output

--json puts one document on stdout and nothing else there; the exit code is unchanged. Consumers pin the schema field, which is the only stable promise — the prose is free to be reworded, and a document is never edited in place: a breaking change bumps its number.

robotsmith lint ./robots.txt --json | jq -r '.findings[] | "::error line=\(.line)::\(.message)"'
robotsmith advise --ua-counts ua.txt --json | jq -r '.decisions[] | select(.policy=="block") | .name'

Exit codes

They are a contract: in CI a robots.txt that loses rules or stays stuck in a cache becomes a red build instead of a late discovery. Telling 1 from 4 matters — "the file says the wrong thing" and "the file is not there" call for different actions.

⛔ What this tool does not do

• It does not tell you whether crawlers obey. robots.txt is a request, not a control. That is measured from the logs over the following days.
• It does not stop whoever disguises itself as a browser. By construction: there is no token to write. That calls for a per-IP request cap or a WAF.
• It does not look at the per-IP rate, do fingerprinting, or verify reverse DNS.
• It does not handle Crawl-delay: Google ignores it, and suggesting it would give false confidence.
• It has no server, daemon or database. It is a command that reads, decides and prints.

How it is built