The gateway is restarted twice in a deployment — once by `up -d`, then again
after the hub whose network namespace it lives in — and `docker compose exec`
into a container that is still coming up fails outright rather than waiting.
Twenty seconds did not cover that, so the first run of the new probe reported a
gateway that answers as down, and the resolver stayed out of client configs for
one more deployment.
The health site also now binds the hub address instead of only matching on it.
Nothing was published either way, but the file should do what its comment says.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The console has been unreachable over the VPN, and the cause was the readiness
probe rather than the tunnel. The gateway binds to the WireGuard hub address
alone; the probe asked 127.0.0.1, where nothing has ever listened. It could only
fail, so VPN_READY stayed false, and on that basis the application withheld the
`DNS = 10.66.0.1` line from every client config it issued. A phone then built
the tunnel, asked its normal resolver for the console hostname, got the public
address and had its connection closed — indistinguishable from "this site does
not exist", which is exactly what it looked like.
Probing over TLS on the bare address would not have worked either: the site
matches on the console's hostname, so a request without SNI is offered no
certificate. The gateway now answers a plain-HTTP health port on the hub
address, which removes TLS, SNI and name resolution from the question and
answers only what is being asked — is this gateway listening, in this network
namespace, right now. Caddy refuses to start when the certificate is unreadable,
so a health port that answers still proves the whole file loaded.
Also here, all found while looking:
- Icons pushed their label onto a second line and rendered a size larger than
asked for. Tailwind's preflight makes an svg display:block, and `.size-4` and
`.size-5` have equal specificity, so stylesheet order decided — and it emits
size-4 first. Every icon written as 16px was silently 20px. Recorded as R18.
- Four error pages printed `errors.404.hint` in place of a sentence: the lang
files give those codes a null hint and Laravel returns the key for a null line.
- The Developer role had no label in either language, so the dropdown showed
`admin_settings.role_developer`.
- The secrets area held one key, which is not worth a password gate. It now
carries the credentials that actually stop the business when they expire —
DNS, monitoring, SMTP — and the test button appears only where a checker
exists, instead of reporting on Stripe whatever was being looked at.
- The update button said nothing about when a queued run would start or where a
running one had got to; both are now shown, and a failure names its step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The owner's phone shows a blank page on the VPN. That page is the reverse proxy
refusing the request, and it refuses because the phone is not on the tunnel: the
client config routes only the management subnet, so a request to the console's
public hostname leaves over the mobile network and arrives from a carrier
address. It is also why the peer reads "last contact: never" — WireGuard only
handshakes when it has traffic to send, and nothing is ever routed in.
The obvious fixes are both wrong. Adding the server's public address to
AllowedIPs routes the WireGuard endpoint into the tunnel it is trying to
establish. A public DNS record pointing at 10.66.0.1 publishes the internal
subnet to anyone enumerating the domain — the owner raised that himself, and he
was right.
So the console is served INSIDE the tunnel instead. A small resolver and a small
gateway share the hub container's network namespace, which is where wg0 lives,
and answer on 10.66.0.1 directly. A client therefore needs no route beyond the
subnet it already has, the host's docker bridge is never exposed, and the
request reaches the application with its real 10.66.0.x source — which is what
the console's own allowlist checks. The gateway reuses the certificate the
public Caddy already renews: a certificate is bound to the name, not to the
address serving it, so nothing new is issued and nothing internal reaches a
public zone.
Most of this commit is the arithmetic of not lying about it. Review found
fourteen ways the arrangement could report itself working while it was not:
enabling the compose profile after the services were started rather than before;
starting a gateway with empty certificate paths, which can only crash; treating
"container created" as "container running"; carrying a stale readiness marker
through a failed restart; reusing the previous hostname's certificate after a
rename; leaving the profile enabled when the hostname was cleared, in a script
that exits early precisely when nothing else would fix it; reading the renewal
timestamp as a user who cannot traverse Caddy's storage; and a `find | head`
that aborts the whole installer under pipefail the moment two issuers hold a
certificate for the same name.
One of them was mine and worth naming: to let the container read the
certificate, I had made the TLS private key world-readable. On a multi-user host
that hands the console's identity to every local account.
The client is told about the resolver only when both services are confirmed
running, because a config naming a resolver that does not exist takes the
device's entire name resolution with it for as long as the tunnel is up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The map listed only *.dev.clupilot.com, with the production names commented out
under "add the real public hostnames here before launch". Launching without
them removes the outer of the two layers that keep /admin off the public
internet — the ADMIN_HOSTS check in the app would still answer 404, but the
whole point of the pair is that neither is relied on alone.
admin.clupilot.com stays out of the list on purpose: the list names the hosts
that are PUBLIC, and the console is not one of them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
While the product is still being built, the marketing site and the customer
portal should not be reachable — but they must stay reachable for us.
- A toggle in the console (site.manage, Owner/Admin) stored in a new
app_settings table, because this has to be flippable without a deploy.
- Outsiders get a placeholder with 503 + noindex, not 200: a 200 invites search
engines to index the placeholder as the site's content, which is far harder
to undo than to prevent.
- Anyone on the management VPN, and any signed-in operator, sees the real site.
The console, Livewire's endpoint, the Stripe webhook and the health check are
always reachable — otherwise the switch could only be flipped once.
- robots.txt is generated by the app and follows the switch. It had to stop
being a static file: nginx short-circuited it and Laravel's stock file said
'Disallow:' with an empty value, so crawlers were never told anything.
- Settings reads fall back to the default when the table is unavailable. The
gate reads one on every request, so a deploy running new code before migrate
would otherwise answer the entire site with a 500.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With the Vite dev server off by default, a fresh checkout had no public/build
(gitignored, and the image does not build it), so every @vite page failed with
ViteManifestNotFoundException. The entrypoint now builds once when the manifest
is absent, non-fatally — a broken build must not wedge the container into a
restart loop where the logs are unreachable.
Proven by deleting public/build and restarting: assets rebuilt automatically,
page 200.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cancel buttons were dead: wire-elements registers Livewire.on('closeModal'),
but the buttons used Alpine's $dispatch, which fires a DOM event. Livewire
bridges its own events TO the DOM, not back, so nothing ever received them.
Proven in the browser both ways — before: modal stays open after Abbrechen;
after: it closes.
Domain operation behind Zoraxy:
- trustProxies for FOR/PORT/PROTO so Laravel sees https (otherwise it builds
http:// URLs into an https page). X_FORWARDED_HOST is deliberately NOT
trusted — the console is gated on the request host, and trusting it would let
anyone reach /admin through a public domain by forging the header.
- APP_URL + VITE_REVERB_* point at the dev domains (wss via ws.dev).
- Vite dev server is now opt-in (VITE_AUTOSTART): over https the browser cannot
load assets from http://<ip>:5173. Built assets are the default, and the
entrypoint clears a stale public/hot, which otherwise 404s every asset.
Verified in the browser over https: login, styled pages, zero console errors,
zero unencrypted requests, /admin reachable only on admin.dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Only the marketing site and the customer portal are public; /admin and the
Proxmox hosts stay on the private network. nginx was a single server_name _
catch-all, so /admin was served on all four dev domains.
Two layers:
- nginx denies /admin on the known-public hostnames before PHP is reached
(denylist, so an allowlist typo cannot lock the operators out).
- ADMIN_HOSTS is now populated, making the app layer a strict allowlist that
also covers /livewire/update, which nginx cannot attribute to /admin.
Verified live against the running stack: /admin is 404 on www/app/api/ws and on
any unlisted host, 302->login on admin.dev / the private IP / localhost, while
/ and /dashboard stay reachable everywhere.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- MONITORING_ATTEMPTS is capped by the run's max_attempts, so a large value can
no longer burn the retry budget and fail the very provisioning the degradation
exists to protect (covered by a new test at the budget boundary)
- /health is now dependency-free (200 in ~2ms with Kuma absent, verified: the
container healthcheck stays healthy); Kuma state is exclusively in /ready
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified with a reachable Kuma and deliberately wrong credentials:
/ready -> 503 'Kuma login rejected: Incorrect username or password.'
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified against a real Kuma by stopping it mid-test:
- get_monitors() reads the library's CACHED monitorList event, so it reported
'up' long after Kuma died. Reachability now does a real HTTP round-trip plus a
live socket check.
- the cached list could yield an id for a monitor that no longer exists; a match
is now confirmed with a live get_monitor before it is returned, so CluPilot can
never record a target that is never checked.
- liveness (/health, always 200 while serving) split from readiness (/ready, 503
when Kuma is unreachable) — the container healthcheck must not restart a
healthy bridge just because a dependency is down.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- attempt is 0-based, so compare one-based: ATTEMPTS=2 really means two tries
- bridge accepts add_monitor returning {monitorID}|{id}|int (v1.2.1 returns a
dict — re-verified: first create on a fresh Kuma is HTTP 200 with the id)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.
Verified end-to-end against a real Uptime Kuma 1.x:
create -> id, create again -> same id (idempotent), list, status, delete
and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.
Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
monitor legitimately reports 'pending' — confirmed in the e2e run); it records
the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Image installs wireguard-tools/iproute2; the provisioning worker brings up
wg0 before queue:work so LocalWireguardHub can manage peers.
- ConfirmRemoveHost deletes each run under its run:<uuid> lock so an in-flight
worker can't keep mutating the server or write to a deleted run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo
Claude