Engineering Notes #14: A Full Disk Looks Like a Compiler Error

Disk

A full EmpoorioChain build with --all-targets grows target/ to 48–50 GB; target/debug/incremental alone reached 19 GB. When the disk fills, cargo fails with No space left on device — but the message appears among ordinary compile errors and looks like one. On 1 September 2026 this happened twice; the first time, several pallets and an XCM executor crate were diagnosed as broken. They were fine. The disk was full.

What works:

• CARGO_INCREMENTAL=0 for every cargo invocation on this workspace — removes the 19 GB outright and does not slow a clean build.
• Reclaim without losing work: rm -rf target/debug/{deps,build,examples,incremental} recovered 44 GB (from 5.3 GiB free to 46 GiB) and regenerates as needed.
• df -h before a long build, not after the failure.

Load

With several development sessions compiling Rust simultaneously, the machine's load average reached 86 — on hardware comfortable below 8. The symptom is not a hang: it is a queue. Three-minute builds take half an hour; wait loops tick for hours. Ten sessions compiling do not finish faster; they all finish much slower.

Rule: one heavy build at a time. Check uptime before starting another.

Related: measuring while someone is writing

While any background process has write access to the tree, do not compile, test or commit. Four times in the Aephoron project a measurement was taken on a tree mid-edit: a three-hour build broke on a half-written function; git add -A swept in-flight work into unrelated commits; a test went red because an agent was deliberately verifying by mutation with a guard removed, and the red was read as a design flaw. Before forming a hypothesis about a surprising result, compare source timestamps with the build's finish time (ls -lT versus the Finished … in mark). A source newer than the binary invalidates the measurement.

Why these are in one note

All three are the same failure: an environmental condition producing an error that looks like a code error. Disk, load, concurrency. Check the environment first.

From the EmpoorioChain build notes, 2026-09-01, and the Aephoron measurement notes.