- update.sh restarted the long-lived services before clearing the caches, so
they booted from the old cached config and kept it for the life of the
process — the release's configuration never reached the workers.
- install.sh put the Gitea token in the clone URL, which is visible in the
process list to every local user while the clone runs. It goes through a
short-lived askpass helper now.
- The collector's uniqueness lock expired at exactly the collection interval,
so a delayed queue could run two collectors on the same baseline and count
the same delta twice — throttling a customer for traffic they never used.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The provisioning worker boots before install.sh writes
CLUPILOT_WG_HUB_PUBKEY and holds the empty value for the life of its process,
so every host onboarded afterwards would have got an empty PublicKey in its
wg0.conf.
- A failed update left the checkout at the target, so the rerun compared a
commit with itself and skipped exactly the dependency and image steps that had
not finished. The comparison base is now the last commit actually deployed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The installer still needs root for apt and the firewall, but it now creates a
clupilot service account that owns the checkout and runs Docker, maps the
container user to it (HOST_UID/GID), and update.sh refuses to run as root —
root-owned files in the checkout are files the application cannot write.
The account has no password login: docker group membership is root-equivalent
on the host, so it is reachable only via sudo -u or an SSH key.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- update.sh recreated the app container only at the very end, so an update that
changes the PHP runtime would have installed dependencies and migrated inside
the old one.
- install.sh randomised DB_PASSWORD but left DB_ROOT_PASSWORD at the value
committed in .env.example — the same root credential on every installation,
reachable from any container on the compose network.
- Settings cached the fallback after a database failure, so one blip could
leave a hidden site publicly visible until someone cleared the cache. Only a
successful read is cached now, proven by a test that drops the table and
brings it back.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex was right on all three, and the first was a design error on my part: the
checkout is bind-mounted into every container, so 'git merge' swaps the running
code instantly — 'migrations before traffic' was not achievable by ordering.
The updater now goes down → merge → dependencies → migrate → assets → restart →
up, trading a short deliberate outage for never serving code whose schema does
not exist yet. If a step fails the site STAYS down: coming back up with new code
on an old schema is worse than staying dark until someone looks.
It also records the last successfully deployed commit. A run that died halfway
left the checkout ahead, so the next run said 'already up to date' and skipped
the rest forever.
And it installs dependencies when a lockfile moved: vendor/ and node_modules/
live in the bind mount and shadow the image, so rebuilding the image never
updated them — migrations could run against stale packages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
install.sh sets up a bare Debian/Ubuntu server end to end: Docker, git,
WireGuard, clone, generated secrets, stack, migrations, hub keypair, the Owner
account and a closed firewall. Re-runnable: it keeps an existing .env and never
regenerates the hub key, which would disconnect every onboarded host.
update.sh pulls and applies. Migrations run before the new containers take
traffic, the image is rebuilt only when its definition changed, and the queue
workers are restarted — they are long-running processes that otherwise keep the
old classes in memory, which cost us an hour during development.
clupilot:create-admin creates or promotes an Owner, so re-running the installer
fixes a lost role instead of failing on a taken address.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo