Postmortem: Three Validators That Were Template Data

Summary

From 29 May to 5 September 2026 the public decentralization report for the EmpoorioChain testnet stated that the network had three validators in us-east-1, eu-west-1 and ap-southeast-1. The network had two, both in one data centre in Logroño, and had never run on AWS. The report was wrong for 99 days.

How it happened

data/Genesis/validators.json was scaffolded early with placeholder entries — the well-known Substrate development accounts (Bob, Charlie, Dave) with invented regions. The placeholders were never replaced. The report generator read the file and rendered it as inventory.

The verification script cited by the report, scripts/testnet_validator_audit.py, was supposed to reconcile the file with the chain. It did not query the chain. It read the file, formatted it, and exited 0. Run against a deliberately refused connection, it produced the same three validators and the same success code.

Why it was not caught

The output was plausible. Three validators in three regions is what a modest testnet looks like. CI saw exit 0. Nobody ran the script with a bad endpoint to see whether it could fail — because nobody asks a passing check what would make it fail.

Detection

The 114-finding audit of 5 September ran the script against port 1. It succeeded. That was the finding.

Fix

Script rewritten to read Session::Validators and Aura::Authorities from one or more live RPC endpoints, compare with the inventory, and exit 1 on disagreement or on no response.
Inventory replaced with the two real accounts, hosts, ports and provider.
Report reissued with a correction block at the top and status not suitable for mainnet: 2 validators, zero fault tolerance, gate requires 10 and four independent operators.
Grafana dashboard's validator count sourced from chain state, not from the file.

Lessons

Ask what would have to be true for this check to fail. If nothing, it is decorative. This question is now in the review checklist for every verification step.
Never hand-edit an inventory the chain can provide. Validator count, supply, spec version, genesis, chain id — all are read from a node now (RED.json is regenerated, not typed).
A public report must carry its own correction. The May report was not deleted; it was superseded with the error described, so that readers can calibrate.

Based on DECENTRALIZATION_REPORT.md (correction block, 2026-09-05).