Near-Miss: The Authority-Set Bug the Test Suite Passed and try-runtime Caught

Summary

During the September 2026 upgrade sequence, a runtime candidate passed the full test suite (3,870 runtime tests plus 309 integration tests, zero failures) and would have stopped the public testnet from producing blocks had it been deployed. It was caught by try-runtime fast-forward run against a snapshot of the live chain state. No incident occurred. This is a near-miss report, written because the ecosystem's rule is that near-misses teach the same lessons as incidents at a fraction of the cost.

What the change did

The candidate altered how the Aura authority set was derived after a session change. In the mock runtimes used by tests, the authority set is small, static and created at genesis; the change behaved correctly there. Against the live chain's state — two validators, real session keys, a history of session rotations — the derived set was empty after the first session boundary. No authorities, no block author, no blocks.

Why the suite passed

Unit and integration tests run against constructed state. They can exercise a pallet's logic exhaustively and still never see the actual state the runtime will meet: real session history, real key rotations, real storage layout accumulated over thousands of blocks and several migrations. The bug lived in the interaction between the change and that history.

What caught it

try-runtime fast-forward loads a snapshot of the live chain's state, applies the candidate runtime, and produces blocks — many of them — driving session changes, era transitions and on_initialize hooks the way the real chain would. At the first session boundary, block production failed. The upgrade was pulled before proposal.

The rule

The failures that kill a chain are not seen by the suite. Run the chain. Concretely:

• try-runtime (on-runtime-upgrade and fast-forward) against a testnet snapshot is mandatory before every runtime upgrade proposal, and the upgrade policy makes the report a required manifest field for any upgrade with a storage migration.
• Post-upgrade hooks verify their own migration's effect, not facts about the world — four hooks that asserted "storage is empty" without checking whether their migration had run were rewritten the same month.
• The public RPC nodes rate-limit and ban; a try-runtime snapshot is taken from an operator's own node, not from the public endpoint.

What would have happened

With two validators and GRANDPA at two thirds, an empty Aura set halts authoring; GRANDPA has nothing new to finalize; every client shows submitted and nothing else. Recovery would require an emergency runtime upgrade — which requires block production to execute. The realistic path would have been a coordinated restart from a snapshot: the very thing the in-place upgrade model exists to avoid.

Lessons

1. Tests prove the logic; only the chain proves the deployment.
2. A near-miss with a written report is worth more than a green build.
3. The emergency-class drill with rollback — still outstanding on the mainnet gate — is the rehearsal for the day try-runtime is not run.

Based on the September 2026 engineering notes ("ejecutar contra la cadena real") and RUNTIME_UPGRADE_POLICY.md.