The owner's phone shows a blank page on the VPN. That page is the reverse proxy
refusing the request, and it refuses because the phone is not on the tunnel: the
client config routes only the management subnet, so a request to the console's
public hostname leaves over the mobile network and arrives from a carrier
address. It is also why the peer reads "last contact: never" — WireGuard only
handshakes when it has traffic to send, and nothing is ever routed in.
The obvious fixes are both wrong. Adding the server's public address to
AllowedIPs routes the WireGuard endpoint into the tunnel it is trying to
establish. A public DNS record pointing at 10.66.0.1 publishes the internal
subnet to anyone enumerating the domain — the owner raised that himself, and he
was right.
So the console is served INSIDE the tunnel instead. A small resolver and a small
gateway share the hub container's network namespace, which is where wg0 lives,
and answer on 10.66.0.1 directly. A client therefore needs no route beyond the
subnet it already has, the host's docker bridge is never exposed, and the
request reaches the application with its real 10.66.0.x source — which is what
the console's own allowlist checks. The gateway reuses the certificate the
public Caddy already renews: a certificate is bound to the name, not to the
address serving it, so nothing new is issued and nothing internal reaches a
public zone.
Most of this commit is the arithmetic of not lying about it. Review found
fourteen ways the arrangement could report itself working while it was not:
enabling the compose profile after the services were started rather than before;
starting a gateway with empty certificate paths, which can only crash; treating
"container created" as "container running"; carrying a stale readiness marker
through a failed restart; reusing the previous hostname's certificate after a
rename; leaving the profile enabled when the hostname was cleared, in a script
that exits early precisely when nothing else would fix it; reading the renewal
timestamp as a user who cannot traverse Caddy's storage; and a `find | head`
that aborts the whole installer under pipefail the moment two issuers hold a
certificate for the same name.
One of them was mine and worth naming: to let the container read the
certificate, I had made the TLS private key world-readable. On a multi-user host
that hands the console's identity to every local account.
The client is told about the resolver only when both services are confirmed
running, because a config naming a resolver that does not exist takes the
device's entire name resolution with it for as long as the tunnel is up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
STATUS_HOST binds /status to a single hostname. Every other host redirects
there rather than 404ing: a status page is the one address people keep in a
bookmark and reach for when something is already wrong, so breaking old links
to prove a point about separation would be exactly backwards. Unset, nothing
changes.
The /legal/status redirect goes through route() instead of a literal path. A
relative redirect lands on whatever host the visitor is already on — which,
once the page has a hostname of its own, is precisely where it no longer
answers.
The installer told the operator to point Stripe's webhook at the customer
portal. Stripe posts server-to-server and never sees the portal; the endpoint
is the api hostname, which the installer never even asked for. It asks now, and
prints the address that actually receives the events. Getting this wrong does
not fail loudly — payments simply stop being recorded.
It also writes ADMIN_HOST_EXCLUSIVE=false on a fresh install. Switching the
console onto its own hostname before the DNS for it exists makes the console
unreachable under any other name, and that is not a thing to have on by default.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The console was unreachable, and not for the reason it looked like. Two causes,
both mine.
A failed `artisan optimize` — the one that hit duplicate route names — left a
broken route cache behind, so every request to the console host answered 500.
Rebuilt.
Underneath that: the reverse proxy has its own console allowlist, hard-coded,
and it runs BEFORE the application. Everything added on the console's own
access page was therefore ineffective, and when the owner's address changed
they were turned away by the proxy before reaching the page that would have
fixed it. The proxy now imports a fragment generated from the same list the
console manages, regenerated by the host agent and reloaded when it changes.
Getting that safe took most of the review. It refuses to rewrite an ambiguous
Caddyfile rather than replacing some other site's matcher; it never falls back
to a loopback-only list when the application cannot be reached, because that
list validates cleanly and locks out every remote operator; it retries a reload
that failed instead of assuming it worked; and an installation that upgrades
without rerunning the installer is told, because otherwise the whole mechanism
is invisibly absent.
Prices are entered in euros. The form asked for cents, so €799 was typed as
79900 and one slipped digit was a factor of ten on an invoice. Conversion
happens in one place, on the string rather than through a float — (float)
'79.90' × 100 is 7989.999… and casting truncates to 7989, one cent short on
exactly the prices people charge — and it refuses an amount the column cannot
hold instead of failing at the database.
Every error code has a page now, in the site's own language and typeface,
self-contained so it still renders when the asset manifest is the thing that
broke. There was only a bare white 404.
The VPN list stops being a seven-column table nobody could fit: names broke
across two lines and so did the headings. These are attributes of one access,
not quantities compared down a column, so each access is a row — identity on
one line, measurements in mono on the next.
The status page says what it measures rather than what it promises. "New orders
are delivered without failures" read as a marketing claim on a page whose only
job is to be believed, and said nothing about the last 24 hours.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The panel had no way to update the installation, and could not have one on its
own: it runs as www-data inside a container, and the update restarts that
container. A button that shelled out would kill its own web server
mid-response, and would need the application to hold host-level credentials —
a far worse thing to own than an out-of-date checkout.
So the button is a mailbox. The panel writes a request into the checkout; a
systemd timer on the host, running as the service account, consumes it, runs
deploy/update.sh and writes back what happened. The same timer answers the
question the panel cannot answer alone — is there anything to update — which
needs a git fetch, and therefore credentials the application does not have.
The parts that were wrong before review, all of which would have shipped as a
button that looks fine and does nothing:
- The agent was only installed by install.sh, which the installed base never
runs again. Every existing server would have shown the button permanently
disabled. It has its own root entry point now, install.sh delegates to it,
and update.sh says so when the unit is missing.
- On a release-pinned server the checkout is detached, so it followed
"origin/HEAD" and then ran an update that deliberately stays on the pinned
tag: updates advertised, nothing applied. It now compares TAGS in release
mode and passes the target release through.
- On a branch other than main it advertised that branch's commits and then
deployed main's, because update.sh defaults to main.
- A request written while the agent was down was executed whenever the agent
next started, days later.
- A single status file meant the routine five-minute check overwrote a failed
update with "idle" before anyone saw it.
- An agent that had been stopped left the button enabled forever, because a
status file written once counted as an agent.
- The agent's error messages were German strings rendered into an English
interface; it reports codes now and the panel translates them.
Also: the add-address button in the console-access panel was as tall as the
field plus its hint, because the hint was a sibling inside the flex row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The file is sourced, so ADMIN_NAME=Boban Blaskovic is read as a command and
dies with "Blaskovic: command not found" — naming neither the file nor the
variable — and set -e ends the install there. A display name with a space is
entirely ordinary; the example just happened to use "Administrator", one word,
so nobody hit it.
Found while installing on the live server.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by running it on a fresh Debian 12 with systemd, no docker, no git and no
users — which is the first time it has ever been run anywhere.
`optional_env` ended in `[[ -n "$2" ]] && set_env …`. With an empty value the
function returns 1, and under `set -e` that ends the install, silently, with a
half-written root-owned .env. Absent is the NORMAL case for those variables —
they are the optional ones — so this killed every installation that did not
happen to supply a Hetzner token. An `if` returns 0 and does not.
Then it waited for `php -v` before migrating. PHP answers in seconds; the
entrypoint is still running `composer install` on a fresh checkout, which takes
minutes. So `artisan migrate` ran against a checkout with no vendor/ and died
on a missing autoload.php. It now waits for vendor/autoload.php, exactly as the
dependent containers already do, and says where to look if it never appears.
After both: exit 0, 59 migrations applied, the plan catalogue seeded and
consistent, the Owner account created with its role, and the portal answering
200 while the console redirects to login and the unconfigured Stripe webhook
fails closed with 400.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
VERSION at the repository root is the release number and the only place it is
written down — a file rather than `git describe`, because a source archive, a
shallow CI checkout and a container built without .git have no tags to
describe. Cutting a release is editing that file and merging it; CI creates the
annotated v<VERSION> tag once green, only on the commit that raised the number,
and never moves it. Servers are pinned to these.
Two modes, and the difference is the point. RELEASE=v1.0.0 pins a server to an
immutable tag, checked out detached: it moves when someone decides it moves.
BRANCH=main follows the edge, where CI tags a commit only after it is pushed.
The mode is remembered, so re-running the updater on a pinned box does not
quietly walk it back onto main.
Going backwards is refused, by commit ancestry rather than by comparing version
strings — a tag can be cut from anywhere, and only ancestry says whether history
is going back. The schema has already moved forward by then, and the migrations
needed to reverse it are not in the older checkout at all.
What the console reports comes from a manifest written atomically after every
step succeeded, never from live git: git says what the files are, not whether
the deployment came up. So a failed update keeps reporting the version that is
actually serving — including its version number, not the newer one the checkout
has already moved to. A deployment pinned to the tag reads "1.0.0 (abc1234)";
anything else reads "1.0.0-dev (abc1234) · main", because every commit after the
tag still carries VERSION=1.0.0 and is not that release. Reporting it as one is
how a bug gets filed against the wrong code.
Codex reviewed the design before it was written and the result six times after.
Its findings, all real: the version had to come from the manifest too, not just
the commit; branch names need JSON escaping or the manifest silently vanishes; a
`case` glob does not anchor and accepted 1x.2y.3garbage; pinning to the commit a
server already sits on skipped recording the mode, and then skipped detaching;
a manifest that failed to write would never be repaired; an unparseable
timestamp would 500 every admin page; and an empty tag list read as a
connectivity failure.
466 tests green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The installer clones BRANCH, which defaults to main — and the repository has
no main. Git's own "Remote branch not found" reads like a bad token or an
unreachable server, and the branch is the one thing here that is routinely
wrong. It now asks first and, when the branch is missing, prints the ones that
do exist and how to pass one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It still named two webhook events; the application handles six, and the four
new ones are the billing cycle itself — renewals, failed payments, status
changes and endings. It also said nothing about stripe:sync-catalogue, without
which Stripe has no prices and nothing can be sold at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things that bit us on every deploy:
- artisan down now renders a branded page that says an update is running and
reloads itself, instead of Laravel's bare 503.
- A page left open across a deploy carries a Livewire snapshot and CSRF token
the new code rejects — the user got "419 Page Expired" and a dead interface.
A 419 from a /livewire/ request now reloads the page; the session is still
valid, so that is all it takes. Hooked at the fetch layer rather than through
Livewire's request hook, whose failure callback is not invoked for this case
in the installed version — verified against a real 419 in the browser, not a
simulated one.
- Vite no longer empties its output directory: wiping it is what left an open
page without its stylesheet mid-deploy, which is the "design is completely
broken" symptom. update.sh also rebuilds the caches with optimize:clear +
optimize instead of leaving them half-warm.
Plus CI: .gitea/workflows/tests.yml (Pest + asset build, and a tested- tag only
on a green main) and an opt-in act_runner service under the ci profile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
--env-file makes the whole run non-interactive, which is what a bare server
needs: install git, clone, run, done. The file also carries the operational
secrets (Hetzner DNS, Stripe, SMTP, the Proxmox key path); those are optional,
so a first install can happen before the Stripe account exists and the features
stay dark until the values are filled in.
The installer warns when the file is readable by more than its owner, and
clupilot.env is gitignored — it holds every secret an installation will need.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- update.sh restarted the long-lived services before clearing the caches, so
they booted from the old cached config and kept it for the life of the
process — the release's configuration never reached the workers.
- install.sh put the Gitea token in the clone URL, which is visible in the
process list to every local user while the clone runs. It goes through a
short-lived askpass helper now.
- The collector's uniqueness lock expired at exactly the collection interval,
so a delayed queue could run two collectors on the same baseline and count
the same delta twice — throttling a customer for traffic they never used.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The provisioning worker boots before install.sh writes
CLUPILOT_WG_HUB_PUBKEY and holds the empty value for the life of its process,
so every host onboarded afterwards would have got an empty PublicKey in its
wg0.conf.
- A failed update left the checkout at the target, so the rerun compared a
commit with itself and skipped exactly the dependency and image steps that had
not finished. The comparison base is now the last commit actually deployed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The installer still needs root for apt and the firewall, but it now creates a
clupilot service account that owns the checkout and runs Docker, maps the
container user to it (HOST_UID/GID), and update.sh refuses to run as root —
root-owned files in the checkout are files the application cannot write.
The account has no password login: docker group membership is root-equivalent
on the host, so it is reachable only via sudo -u or an SSH key.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- update.sh recreated the app container only at the very end, so an update that
changes the PHP runtime would have installed dependencies and migrated inside
the old one.
- install.sh randomised DB_PASSWORD but left DB_ROOT_PASSWORD at the value
committed in .env.example — the same root credential on every installation,
reachable from any container on the compose network.
- Settings cached the fallback after a database failure, so one blip could
leave a hidden site publicly visible until someone cleared the cache. Only a
successful read is cached now, proven by a test that drops the table and
brings it back.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex was right on all three, and the first was a design error on my part: the
checkout is bind-mounted into every container, so 'git merge' swaps the running
code instantly — 'migrations before traffic' was not achievable by ordering.
The updater now goes down → merge → dependencies → migrate → assets → restart →
up, trading a short deliberate outage for never serving code whose schema does
not exist yet. If a step fails the site STAYS down: coming back up with new code
on an old schema is worse than staying dark until someone looks.
It also records the last successfully deployed commit. A run that died halfway
left the checkout ahead, so the next run said 'already up to date' and skipped
the rest forever.
And it installs dependencies when a lockfile moved: vendor/ and node_modules/
live in the bind mount and shadow the image, so rebuilding the image never
updated them — migrations could run against stale packages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
install.sh sets up a bare Debian/Ubuntu server end to end: Docker, git,
WireGuard, clone, generated secrets, stack, migrations, hub keypair, the Owner
account and a closed firewall. Re-runnable: it keeps an existing .env and never
regenerates the hub key, which would disconnect every onboarded host.
update.sh pulls and applies. Migrations run before the new containers take
traffic, the image is rebuilt only when its definition changed, and the queue
workers are restarted — they are long-running processes that otherwise keep the
old classes in memory, which cost us an hour during development.
clupilot:create-admin creates or promotes an Owner, so re-running the installer
fixes a lost role instead of failing on a taken address.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude
nexxo