galera-doctor

galera-doctor

A read-only audit of a MariaDB/MySQL Galera cluster that reports the states the cluster's own metrics cannot show you.

one static Go binary one dependency SHOW and SELECT only writes: none MIT

What a run looks like

$ galera-doctor audit --config clusters.json --cluster compress --state /var/lib/galera-doctor/compress.json
BAD   compress  3 node(s)
  BAD   cluster/uuid       compress  nodes report different cluster state UUIDs: 5b1e2a8c-111… (cl-02, sg-01) vs 9999aaaa-222… (ov-03)
        ↳ this is a partition, not a lag: the groups have diverged and one side has to be reinitialised from the other
  BAD   systables/drift    mysql.column_stats  definition differs across nodes: aaaaaaaaaaaa… (ov-03, sg-01) vs cccccccccccc… (cl-02)
        ↳ Galera does not replicate this: fix it per node — no wsrep_* metric will ever show it
  WARN  flow/paused        sg-01     flow-controlled 2.10% of the last 10m0s
        ↳ this node is intermittently the slowest in the cluster; look at its disk and its replication threads
  WARN  schema/no-pk       schema    1 table(s) without a primary key: app.events
  OK    cluster/size       compress  3 member(s), expected 3

Three nodes, one cluster, every replication counter green — and one of them has been holding a different definition of a system table since somebody ran mysql_upgrade on two of the three.

The states it exists for

What is wrongWhy no wsrep_* metric shows it
A system table's definition differs between nodes Galera does not replicate maintenance on the server's own mysql.* tables. Two nodes can disagree about mysql.column_stats for months while every replication counter stays green — until one node's query plans go strange.
One name, two clusters Each half reports a consistent size and a Primary status. The divergence is only visible by comparing the wsrep_cluster_state_uuid of every node to each other.
The proxy and the cluster disagree ProxySQL says ONLINE, the node says Joined. Each dashboard is fine. Traffic is going to a node that is not synced.
The gcache is too small for the write rate 512 MB is either forty minutes or ninety seconds. Nobody finds out until a node restarts and needs a full SST, taking a donor out of service with it.
Flow control that already happened wsrep_flow_control_paused covers the time since the last status reset, so an incident from March reads the same today.

A total is not a rate

The wsrep counters only go up, and they reset on restart. A threshold over wsrep_flow_control_paused goes red once and stays red — and a check that stays red is a check people stop reading. So --state FILE remembers the counters between runs and the checks grade the interval:

  WARN  flow/paused   sg-01   flow-controlled 2.10% of the last 10m0s

Without a baseline the same check reports the lifetime figure and refuses to judge it:

  OK    flow/paused   sg-01   0.9% of the time since the last status reset (not graded: no baseline)
        ↳ run again with --state to grade the interval between runs instead of the lifetime total

A counter that went backwards, a node whose uptime shrank, a state file from another format version: all of them mean no baseline — never a negative rate and never a spectacular one.

Install

$ go install github.com/Allan-Nava/galera-doctor/cmd/galera-doctor@latest

# or the static binary, and the image that contains nothing else
$ go build -o galera-doctor ./cmd/galera-doctor
$ docker build -t galera-doctor .

Audit one node without a config file at all:

$ galera-doctor audit --node "sg-01=audit:***@tcp(10.11.1.5:3306)/"

Read-only, mechanically

One gate for every query

Every statement goes through a single function that refuses anything but SHOW and SELECT.

a property, not a promise

CI greps the source

The build fails on a writing statement or an Exec anywhere in the tree.

enforced on every commit

No DSN in the output

Nodes are identified by name and driver errors are redacted — findings end up in tickets.

redact before render

It says what it could not read

An unreachable node is an ERROR, because every cluster-wide statement below it was made without that node.

never a silent omission

What it is not

Docs