The console announced an update for "spätestens um 15:21" while the clock on
the wall said 17:21. Not a formatting slip in one string: fourteen views and
four components printed database values straight out, so every absolute time in
the product was two hours out all summer and one hour out all winter.
Two of those views looked as though they had handled it. They called
->timezone(config('app.timezone'))
which reads like "convert to local time" and, app.timezone being the STORAGE
zone that must stay UTC, converts to UTC — a no-op wearing the costume of a
fix, which is worse than no call at all because it stops the next person
looking. That exact string is now banned by test.
Worse than the labels were two FORMS. Maintenance windows and plan versions
filled their datetime-local fields from UTC and parsed what came back as UTC. A
datetime-local field carries no offset — it is the digits a person reads off
their own clock — so an operator typing 21:00 scheduled a window for 23:00
their time, and nothing anywhere said so. App\Support\LocalTime now holds
toField() and fromField() side by side, because getting one end right is not
half a fix, it is a fresh bug.
The mechanism is one Carbon macro, ->local(). It copies before converting:
Illuminate\Support\Carbon is mutable, so without that, rendering a timestamp
would rewrite the model attribute as a side effect and anything comparing or
saving it afterwards would be an hour or two out. Both Carbon classes get the
same body — Carbon keeps macros in one global table, so the second registration
replaces the first for every Carbon class, and two different bodies meant the
immutable version, safe for itself, silently became the mutable one's
implementation too. My own test caught that.
The existing tests had not caught any of this because they built their expected
values with the same wrong call the views used — a test that recomputes the
implementation asserts nothing. They now assert the wall clock, and the queued
update additionally asserts that the UTC time is NOT shown.
Recorded as R19 in CLAUDE.md and enforced by tests/Feature/DisplayTimezoneTest:
no Blade or Livewire component may format an absolute time without ->local(),
the UTC no-op is banned, storage stays UTC, both sides of the DST boundary are
checked, ->local() must not mutate, and a form field must round-trip to the
same instant.
APP_DISPLAY_TIMEZONE, default Europe/Vienna. 615 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gateway is restarted twice in a deployment — once by `up -d`, then again
after the hub whose network namespace it lives in — and `docker compose exec`
into a container that is still coming up fails outright rather than waiting.
Twenty seconds did not cover that, so the first run of the new probe reported a
gateway that answers as down, and the resolver stayed out of client configs for
one more deployment.
The health site also now binds the hub address instead of only matching on it.
Nothing was published either way, but the file should do what its comment says.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The console has been unreachable over the VPN, and the cause was the readiness
probe rather than the tunnel. The gateway binds to the WireGuard hub address
alone; the probe asked 127.0.0.1, where nothing has ever listened. It could only
fail, so VPN_READY stayed false, and on that basis the application withheld the
`DNS = 10.66.0.1` line from every client config it issued. A phone then built
the tunnel, asked its normal resolver for the console hostname, got the public
address and had its connection closed — indistinguishable from "this site does
not exist", which is exactly what it looked like.
Probing over TLS on the bare address would not have worked either: the site
matches on the console's hostname, so a request without SNI is offered no
certificate. The gateway now answers a plain-HTTP health port on the hub
address, which removes TLS, SNI and name resolution from the question and
answers only what is being asked — is this gateway listening, in this network
namespace, right now. Caddy refuses to start when the certificate is unreadable,
so a health port that answers still proves the whole file loaded.
Also here, all found while looking:
- Icons pushed their label onto a second line and rendered a size larger than
asked for. Tailwind's preflight makes an svg display:block, and `.size-4` and
`.size-5` have equal specificity, so stylesheet order decided — and it emits
size-4 first. Every icon written as 16px was silently 20px. Recorded as R18.
- Four error pages printed `errors.404.hint` in place of a sentence: the lang
files give those codes a null hint and Laravel returns the key for a null line.
- The Developer role had no label in either language, so the dropdown showed
`admin_settings.role_developer`.
- The secrets area held one key, which is not worth a password gate. It now
carries the credentials that actually stop the business when they expire —
DNS, monitoring, SMTP — and the test button appears only where a checker
exists, instead of reporting on Stripe whatever was being looked at.
- The update button said nothing about when a queued run would start or where a
running one had got to; both are now shown, and a failure names its step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An empty 404 is itself information: it says something terminates TLS here and
chose not to answer. The console and the websocket endpoint now close the
connection instead, which is what a hostname that serves nothing looks like.
The websocket endpoint answers a genuine upgrade — verified live, 101 — and
nothing else; a browser opening it gets a closed connection.
Reviewing that turned up two holes that mattered more than the thing being
reviewed.
The compose defaults published the application and Reverb on every interface.
Docker publishes ports ahead of UFW, so those backends were reachable from the
internet with the firewall closed — and reaching a backend directly skips the
proxy, and with it every hostname and address rule keeping the console private.
The defaults are loopback now, and an update rebinds an existing installation
that is behind a proxy actually running and holding 443. A development box
without one is left alone, because taking its port away would look like the
machine had broken.
And the console's own "open" switch was emitting 0.0.0.0/0 into the proxy's
allowlist. One click would have put the console on the public internet, which
is the exact state the owner's rule exists to prevent. Switching it off relaxes
the application's check; the proxy keeps its list, always.
The reference proxy config carries the arrangement: QUIC early data off, since
an address-based decision taken on 0-RTT can answer 425 and some browsers do
not retry — an intermittent lockout from the console is the worst kind — and
explicit handling for plain HTTP, whose automatic redirect otherwise announces
both hidden names to anyone who asks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things reported success while the tunnel console was unreachable.
The services were restarted the moment the hub container counted as started,
but the hub brings wg0 up as part of its start command — so they were binding an
address that did not exist yet. They now wait for the interface.
And readiness was a container-state check. A container can be up and running
while the process inside it listens in a network namespace that was torn down
underneath it: nothing errors, the address simply refuses connections, and
"running" reports everything fine. It asks the gateway now, and only a real
answer counts — because the consequence of getting this wrong is a client
config naming a resolver that is not there, which takes the device's whole name
resolution with it for as long as the tunnel is up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both VPN services share the provisioning container's network namespace, and a
process holds the namespace it started in. Restarting the hub — which every
deploy does, to pick up new code — therefore leaves them listening inside a
namespace that no longer exists.
Nothing errors. The containers stay up, Caddy's log still reports it is serving
on the tunnel address, and connections to that address are simply refused. It
looks precisely like a gateway that never worked, which is how it presented on
the live server after the first successful start.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The owner's phone shows a blank page on the VPN. That page is the reverse proxy
refusing the request, and it refuses because the phone is not on the tunnel: the
client config routes only the management subnet, so a request to the console's
public hostname leaves over the mobile network and arrives from a carrier
address. It is also why the peer reads "last contact: never" — WireGuard only
handshakes when it has traffic to send, and nothing is ever routed in.
The obvious fixes are both wrong. Adding the server's public address to
AllowedIPs routes the WireGuard endpoint into the tunnel it is trying to
establish. A public DNS record pointing at 10.66.0.1 publishes the internal
subnet to anyone enumerating the domain — the owner raised that himself, and he
was right.
So the console is served INSIDE the tunnel instead. A small resolver and a small
gateway share the hub container's network namespace, which is where wg0 lives,
and answer on 10.66.0.1 directly. A client therefore needs no route beyond the
subnet it already has, the host's docker bridge is never exposed, and the
request reaches the application with its real 10.66.0.x source — which is what
the console's own allowlist checks. The gateway reuses the certificate the
public Caddy already renews: a certificate is bound to the name, not to the
address serving it, so nothing new is issued and nothing internal reaches a
public zone.
Most of this commit is the arithmetic of not lying about it. Review found
fourteen ways the arrangement could report itself working while it was not:
enabling the compose profile after the services were started rather than before;
starting a gateway with empty certificate paths, which can only crash; treating
"container created" as "container running"; carrying a stale readiness marker
through a failed restart; reusing the previous hostname's certificate after a
rename; leaving the profile enabled when the hostname was cleared, in a script
that exits early precisely when nothing else would fix it; reading the renewal
timestamp as a user who cannot traverse Caddy's storage; and a `find | head`
that aborts the whole installer under pipefail the moment two issuers hold a
certificate for the same name.
One of them was mine and worth naming: to let the container read the
certificate, I had made the TLS private key world-readable. On a multi-user host
that hands the console's identity to every local account.
The client is told about the resolver only when both services are confirmed
running, because a config naming a resolver that does not exist takes the
device's entire name resolution with it for as long as the tunnel is up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
STATUS_HOST binds /status to a single hostname. Every other host redirects
there rather than 404ing: a status page is the one address people keep in a
bookmark and reach for when something is already wrong, so breaking old links
to prove a point about separation would be exactly backwards. Unset, nothing
changes.
The /legal/status redirect goes through route() instead of a literal path. A
relative redirect lands on whatever host the visitor is already on — which,
once the page has a hostname of its own, is precisely where it no longer
answers.
The installer told the operator to point Stripe's webhook at the customer
portal. Stripe posts server-to-server and never sees the portal; the endpoint
is the api hostname, which the installer never even asked for. It asks now, and
prints the address that actually receives the events. Getting this wrong does
not fail loudly — payments simply stop being recorded.
It also writes ADMIN_HOST_EXCLUSIVE=false on a fresh install. Switching the
console onto its own hostname before the DNS for it exists makes the console
unreachable under any other name, and that is not a thing to have on by default.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The console was unreachable, and not for the reason it looked like. Two causes,
both mine.
A failed `artisan optimize` — the one that hit duplicate route names — left a
broken route cache behind, so every request to the console host answered 500.
Rebuilt.
Underneath that: the reverse proxy has its own console allowlist, hard-coded,
and it runs BEFORE the application. Everything added on the console's own
access page was therefore ineffective, and when the owner's address changed
they were turned away by the proxy before reaching the page that would have
fixed it. The proxy now imports a fragment generated from the same list the
console manages, regenerated by the host agent and reloaded when it changes.
Getting that safe took most of the review. It refuses to rewrite an ambiguous
Caddyfile rather than replacing some other site's matcher; it never falls back
to a loopback-only list when the application cannot be reached, because that
list validates cleanly and locks out every remote operator; it retries a reload
that failed instead of assuming it worked; and an installation that upgrades
without rerunning the installer is told, because otherwise the whole mechanism
is invisibly absent.
Prices are entered in euros. The form asked for cents, so €799 was typed as
79900 and one slipped digit was a factor of ten on an invoice. Conversion
happens in one place, on the string rather than through a float — (float)
'79.90' × 100 is 7989.999… and casting truncates to 7989, one cent short on
exactly the prices people charge — and it refuses an amount the column cannot
hold instead of failing at the database.
Every error code has a page now, in the site's own language and typeface,
self-contained so it still renders when the asset manifest is the thing that
broke. There was only a bare white 404.
The VPN list stops being a seven-column table nobody could fit: names broke
across two lines and so did the headings. These are attributes of one access,
not quantities compared down a column, so each access is a row — identity on
one line, measurements in mono on the next.
The status page says what it measures rather than what it promises. "New orders
are delivered without failures" read as a marketing claim on a page whose only
job is to be believed, and said nothing about the last 24 hours.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The panel had no way to update the installation, and could not have one on its
own: it runs as www-data inside a container, and the update restarts that
container. A button that shelled out would kill its own web server
mid-response, and would need the application to hold host-level credentials —
a far worse thing to own than an out-of-date checkout.
So the button is a mailbox. The panel writes a request into the checkout; a
systemd timer on the host, running as the service account, consumes it, runs
deploy/update.sh and writes back what happened. The same timer answers the
question the panel cannot answer alone — is there anything to update — which
needs a git fetch, and therefore credentials the application does not have.
The parts that were wrong before review, all of which would have shipped as a
button that looks fine and does nothing:
- The agent was only installed by install.sh, which the installed base never
runs again. Every existing server would have shown the button permanently
disabled. It has its own root entry point now, install.sh delegates to it,
and update.sh says so when the unit is missing.
- On a release-pinned server the checkout is detached, so it followed
"origin/HEAD" and then ran an update that deliberately stays on the pinned
tag: updates advertised, nothing applied. It now compares TAGS in release
mode and passes the target release through.
- On a branch other than main it advertised that branch's commits and then
deployed main's, because update.sh defaults to main.
- A request written while the agent was down was executed whenever the agent
next started, days later.
- A single status file meant the routine five-minute check overwrote a failed
update with "idle" before anyone saw it.
- An agent that had been stopped left the button enabled forever, because a
status file written once counted as an agent.
- The agent's error messages were German strings rendered into an English
interface; it reports codes now and the panel translates them.
Also: the add-address button in the console-access panel was as tall as the
field plus its hint, because the hint was a sibling inside the flex row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The file is sourced, so ADMIN_NAME=Boban Blaskovic is read as a command and
dies with "Blaskovic: command not found" — naming neither the file nor the
variable — and set -e ends the install there. A display name with a space is
entirely ordinary; the example just happened to use "Administrator", one word,
so nobody hit it.
Found while installing on the live server.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by running it on a fresh Debian 12 with systemd, no docker, no git and no
users — which is the first time it has ever been run anywhere.
`optional_env` ended in `[[ -n "$2" ]] && set_env …`. With an empty value the
function returns 1, and under `set -e` that ends the install, silently, with a
half-written root-owned .env. Absent is the NORMAL case for those variables —
they are the optional ones — so this killed every installation that did not
happen to supply a Hetzner token. An `if` returns 0 and does not.
Then it waited for `php -v` before migrating. PHP answers in seconds; the
entrypoint is still running `composer install` on a fresh checkout, which takes
minutes. So `artisan migrate` ran against a checkout with no vendor/ and died
on a missing autoload.php. It now waits for vendor/autoload.php, exactly as the
dependent containers already do, and says where to look if it never appears.
After both: exit 0, 59 migrations applied, the plan catalogue seeded and
consistent, the Owner account created with its role, and the portal answering
200 while the console redirects to login and the unconfigured Stripe webhook
fails closed with 400.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
VERSION at the repository root is the release number and the only place it is
written down — a file rather than `git describe`, because a source archive, a
shallow CI checkout and a container built without .git have no tags to
describe. Cutting a release is editing that file and merging it; CI creates the
annotated v<VERSION> tag once green, only on the commit that raised the number,
and never moves it. Servers are pinned to these.
Two modes, and the difference is the point. RELEASE=v1.0.0 pins a server to an
immutable tag, checked out detached: it moves when someone decides it moves.
BRANCH=main follows the edge, where CI tags a commit only after it is pushed.
The mode is remembered, so re-running the updater on a pinned box does not
quietly walk it back onto main.
Going backwards is refused, by commit ancestry rather than by comparing version
strings — a tag can be cut from anywhere, and only ancestry says whether history
is going back. The schema has already moved forward by then, and the migrations
needed to reverse it are not in the older checkout at all.
What the console reports comes from a manifest written atomically after every
step succeeded, never from live git: git says what the files are, not whether
the deployment came up. So a failed update keeps reporting the version that is
actually serving — including its version number, not the newer one the checkout
has already moved to. A deployment pinned to the tag reads "1.0.0 (abc1234)";
anything else reads "1.0.0-dev (abc1234) · main", because every commit after the
tag still carries VERSION=1.0.0 and is not that release. Reporting it as one is
how a bug gets filed against the wrong code.
Codex reviewed the design before it was written and the result six times after.
Its findings, all real: the version had to come from the manifest too, not just
the commit; branch names need JSON escaping or the manifest silently vanishes; a
`case` glob does not anchor and accepted 1x.2y.3garbage; pinning to the commit a
server already sits on skipped recording the mode, and then skipped detaching;
a manifest that failed to write would never be repaired; an unparseable
timestamp would 500 every admin page; and an empty tag list read as a
connectivity failure.
466 tests green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The installer clones BRANCH, which defaults to main — and the repository has
no main. Git's own "Remote branch not found" reads like a bad token or an
unreachable server, and the branch is the one thing here that is routinely
wrong. It now asks first and, when the branch is missing, prints the ones that
do exist and how to pass one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It still named two webhook events; the application handles six, and the four
new ones are the billing cycle itself — renewals, failed payments, status
changes and endings. It also said nothing about stripe:sync-catalogue, without
which Stripe has no prices and nothing can be sold at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things that bit us on every deploy:
- artisan down now renders a branded page that says an update is running and
reloads itself, instead of Laravel's bare 503.
- A page left open across a deploy carries a Livewire snapshot and CSRF token
the new code rejects — the user got "419 Page Expired" and a dead interface.
A 419 from a /livewire/ request now reloads the page; the session is still
valid, so that is all it takes. Hooked at the fetch layer rather than through
Livewire's request hook, whose failure callback is not invoked for this case
in the installed version — verified against a real 419 in the browser, not a
simulated one.
- Vite no longer empties its output directory: wiping it is what left an open
page without its stylesheet mid-deploy, which is the "design is completely
broken" symptom. update.sh also rebuilds the caches with optimize:clear +
optimize instead of leaving them half-warm.
Plus CI: .gitea/workflows/tests.yml (Pest + asset build, and a tested- tag only
on a green main) and an opt-in act_runner service under the ci profile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
--env-file makes the whole run non-interactive, which is what a bare server
needs: install git, clone, run, done. The file also carries the operational
secrets (Hetzner DNS, Stripe, SMTP, the Proxmox key path); those are optional,
so a first install can happen before the Stripe account exists and the features
stay dark until the values are filled in.
The installer warns when the file is readable by more than its owner, and
clupilot.env is gitignored — it holds every secret an installation will need.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- update.sh restarted the long-lived services before clearing the caches, so
they booted from the old cached config and kept it for the life of the
process — the release's configuration never reached the workers.
- install.sh put the Gitea token in the clone URL, which is visible in the
process list to every local user while the clone runs. It goes through a
short-lived askpass helper now.
- The collector's uniqueness lock expired at exactly the collection interval,
so a delayed queue could run two collectors on the same baseline and count
the same delta twice — throttling a customer for traffic they never used.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The provisioning worker boots before install.sh writes
CLUPILOT_WG_HUB_PUBKEY and holds the empty value for the life of its process,
so every host onboarded afterwards would have got an empty PublicKey in its
wg0.conf.
- A failed update left the checkout at the target, so the rerun compared a
commit with itself and skipped exactly the dependency and image steps that had
not finished. The comparison base is now the last commit actually deployed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The installer still needs root for apt and the firewall, but it now creates a
clupilot service account that owns the checkout and runs Docker, maps the
container user to it (HOST_UID/GID), and update.sh refuses to run as root —
root-owned files in the checkout are files the application cannot write.
The account has no password login: docker group membership is root-equivalent
on the host, so it is reachable only via sudo -u or an SSH key.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- update.sh recreated the app container only at the very end, so an update that
changes the PHP runtime would have installed dependencies and migrated inside
the old one.
- install.sh randomised DB_PASSWORD but left DB_ROOT_PASSWORD at the value
committed in .env.example — the same root credential on every installation,
reachable from any container on the compose network.
- Settings cached the fallback after a database failure, so one blip could
leave a hidden site publicly visible until someone cleared the cache. Only a
successful read is cached now, proven by a test that drops the table and
brings it back.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex was right on all three, and the first was a design error on my part: the
checkout is bind-mounted into every container, so 'git merge' swaps the running
code instantly — 'migrations before traffic' was not achievable by ordering.
The updater now goes down → merge → dependencies → migrate → assets → restart →
up, trading a short deliberate outage for never serving code whose schema does
not exist yet. If a step fails the site STAYS down: coming back up with new code
on an old schema is worse than staying dark until someone looks.
It also records the last successfully deployed commit. A run that died halfway
left the checkout ahead, so the next run said 'already up to date' and skipped
the rest forever.
And it installs dependencies when a lockfile moved: vendor/ and node_modules/
live in the bind mount and shadow the image, so rebuilding the image never
updated them — migrations could run against stale packages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
install.sh sets up a bare Debian/Ubuntu server end to end: Docker, git,
WireGuard, clone, generated secrets, stack, migrations, hub keypair, the Owner
account and a closed firewall. Re-runnable: it keeps an existing .env and never
regenerates the hub key, which would disconnect every onboarded host.
update.sh pulls and applies. Migrations run before the new containers take
traffic, the image is rebuilt only when its definition changed, and the queue
workers are restarted — they are long-running processes that otherwise keep the
old classes in memory, which cost us an hour during development.
clupilot:create-admin creates or promotes an Owner, so re-running the installer
fixes a lost role instead of failing on a taken address.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo
Claude