Tool: drbd-verify.py

Overview

drbd-verify.py runs DRBD online verifies across the resources of a cluster and investigates any differences it finds. Unlike a plain drbdadm verify, which only compares the local node against its peers, this tool performs every verify needed to compare each pair of nodes (1 verify for 2 nodes, 3 for 3 nodes, 6 for 4 nodes, and so on).

For every pair of nodes that turns out to have out-of-sync (OOS) blocks, it then tries to explain the difference and recommend how to fix it. It reports the affected files where it can, uses heuristics such as the Shannon entropy of the differing blocks and filesystem checks (fsck) to judge which side holds the real data, and prints the exact drbdadm commands needed to resync. With --do-it it can run those commands for you.

This is useful whenever you need to confirm that two or more replicas of a volume actually hold identical data — for example when investigating suspected data divergence, such as the DRBD Unrotated UUID condition.

Requirements

• Python 3
• A working kubectl and running LINSTOR satellite pods. Nodes are reached with kubectl exec into the satellite pod, so the DRBD tools do not need to be installed on the hosts themselves.

The tool is intended for interactive use: it refreshes its progress display every few seconds. A full verify reads every replica end to end and can take a long time. Resources with an ongoing resync or verify are skipped automatically.

Get the Tool

curl -O https://charts.linstor.io/drbd-verify.py

Usage

Run the script from a machine that has kubectl access to the cluster. Pick any node whose drbdsetup status is used to discover the resources and their peers:

# All resources visible from <node>
python3 drbd-verify.py --kubectl --node <node>

# A single resource (use the PV / volumeHandle name)
python3 drbd-verify.py --kubectl --node <node> --resource <resource>

The satellite namespace defaults to linbit-sds; override it with --namespace if you deployed elsewhere. The exec target defaults to ds/linstor-satellite.{node} and can be changed with --pod-pattern.

Useful options

• -r, --resource — operate only on the named resource(s).
• --skip-verify — skip the (slow) drbdsetup verify and instead analyze the blocks DRBD already reports as out-of-sync.
• --do-it — execute the suggested resync commands instead of only printing them.
• -j, --json — emit only JSON, suppressing the progress output.

Interpreting the Results

For each pair of nodes the tool reports the amount of out-of-sync data in KiB. A value of 0 means the two replicas are identical. For any pair that differs, it decides which node to treat as the source of truth by comparing the Shannon entropy of the differing blocks: the side with higher-entropy data (real, compressed or encrypted content, as opposed to zeroes) is chosen as the resync source, and the other side is overwritten. Filesystem-check error and warning counts, plus the list of affected files, are reported for context.

The recommended resync commands are collected under resync_suggestions in the output, and a full copy of the results is written to drbd-verify-result_YYYY-MM-DD_HHMM.json in the current directory.

A resync overwrites the target node's copy of the divergent blocks. Review the recommendation before running with --do-it. If you are unsure which replica is correct, or need help interpreting the output, contact LINBIT support.