What a run looks like
$ galera-doctor audit --config clusters.json --cluster compress --state /var/lib/galera-doctor/compress.json BAD compress 3 node(s) BAD cluster/uuid compress nodes report different cluster state UUIDs: 5b1e2a8c-111… (cl-02, sg-01) vs 9999aaaa-222… (ov-03) ↳ this is a partition, not a lag: the groups have diverged and one side has to be reinitialised from the other BAD systables/drift mysql.column_stats definition differs across nodes: aaaaaaaaaaaa… (ov-03, sg-01) vs cccccccccccc… (cl-02) ↳ Galera does not replicate this: fix it per node — no wsrep_* metric will ever show it WARN flow/paused sg-01 flow-controlled 2.10% of the last 10m0s ↳ this node is intermittently the slowest in the cluster; look at its disk and its replication threads WARN schema/no-pk schema 1 table(s) without a primary key: app.events OK cluster/size compress 3 member(s), expected 3
Three nodes, one cluster, every replication counter green — and one of them has
been holding a different definition of a system table since somebody ran
mysql_upgrade on two of the three.
The states it exists for
| What is wrong | Why no wsrep_* metric shows it |
|---|---|
| A system table's definition differs between nodes | Galera does not replicate maintenance on the server's own mysql.* tables.
Two nodes can disagree about mysql.column_stats for months while every
replication counter stays green — until one node's query plans go strange. |
| One name, two clusters | Each half reports a consistent size and a Primary status. The divergence is only
visible by comparing the wsrep_cluster_state_uuid of every node
to each other. |
| The proxy and the cluster disagree | ProxySQL says ONLINE, the node says Joined. Each dashboard is fine. Traffic is going to a node that is not synced. |
| The gcache is too small for the write rate | 512 MB is either forty minutes or ninety seconds. Nobody finds out until a node restarts and needs a full SST, taking a donor out of service with it. |
| Flow control that already happened | wsrep_flow_control_paused covers the time since the last status reset, so
an incident from March reads the same today. |
A total is not a rate
The wsrep counters only go up, and they reset on restart. A threshold over
wsrep_flow_control_paused goes red once and stays red — and a check that stays
red is a check people stop reading. So --state FILE remembers the counters
between runs and the checks grade the interval:
WARN flow/paused sg-01 flow-controlled 2.10% of the last 10m0s
Without a baseline the same check reports the lifetime figure and refuses to judge it:
OK flow/paused sg-01 0.9% of the time since the last status reset (not graded: no baseline) ↳ run again with --state to grade the interval between runs instead of the lifetime total
A counter that went backwards, a node whose uptime shrank, a state file from another format version: all of them mean no baseline — never a negative rate and never a spectacular one.
Install
$ go install github.com/Allan-Nava/galera-doctor/cmd/galera-doctor@latest # or the static binary, and the image that contains nothing else $ go build -o galera-doctor ./cmd/galera-doctor $ docker build -t galera-doctor .
Audit one node without a config file at all:
$ galera-doctor audit --node "sg-01=audit:***@tcp(10.11.1.5:3306)/"
Read-only, mechanically
One gate for every query
Every statement goes through a single function that refuses anything but
SHOW and SELECT.
CI greps the source
The build fails on a writing statement or an Exec anywhere in the tree.
No DSN in the output
Nodes are identified by name and driver errors are redacted — findings end up in tickets.
redact before renderIt says what it could not read
An unreachable node is an ERROR, because every cluster-wide statement below
it was made without that node.
What it is not
- Not a monitoring system. No daemon, no scraper, no history beyond the one state file it uses for rates. It is what you run from cron, a CI job or an incident.
- Not a generic MySQL health check. Connections, buffer pool, slow queries and replica lag belong in checkfleet. This one only knows about Galera.
- Not a repair tool. It reports; a human decides. Bootstrapping a cluster, dropping a peer or resyncing a node are operations with consequences, and they are not one flag away here.
Docs
- Install — binary,
go install, container - Usage — flags, config, output, exit status
- Checks — every check, what it grades and why
- Rates, not totals — the state file, and why a counter is never a verdict
- Permissions and safety — the grants it needs, the writes it cannot do
- Intent — why it exists, and what it will never become
- Roadmap — generated from
BACKLOG.md