What happened
api_server.py is the Ailoos coordinator's central file — routers, mounts, middleware. On 1 August 2026 it needed a small change. The local copy was uploaded over the remote. Production broke; rollback in under a minute. The local copy had accumulated unshipped work in progress — a new router, an environment-variable change — that was never meant to deploy.
Later in the same session the file needed another change. This time the remote copy was fetched and diffed first. It had diverged again. The change was applied as a two-line patch to a copy of the remote file, verified, and deployed. Nothing broke.
The rule
For any file that is central — mounts, imports, routers, configuration loaders — and lives on a production host:
scpthe remote file to a temporary location.diffit against the local copy.- If there is divergence unrelated to your change, build a minimal patch (exact-line replacement) applied to a copy of the remote file — never to the diverged local copy.
- Verify the patch's diff is only what you intended.
- Back up the remote with a timestamp before overwriting.
- Checksum after copying.
Never assume the local copy matches what is deployed, even if it matched last time.
Why it keeps happening
Because the coordinator's code is bind-mounted into its container and "deploying" is copying a file. That is fast and it is also how local work-in-progress becomes production without anyone deciding it should. The permanent fix — deploy from a versioned artefact, not a working copy — is the direction; the procedure above is the guard until then.
The general form
The local tree is not production. Engineering Notes #1 said it about library versions; this one says it about the files themselves. Measure the real thing, diff the real thing, patch the real thing.
From the Ailoos federated-training deployment, 2026-08-01.