Zwei Ursachen, eine davon habe ich mit v1.4.2 selbst gelegt.
1. DER NETZ-ALIAS. Damit in docker/nginx/default.conf `terminal:8082` stehen
konnte statt des Namens des Nachbarcontainers, bekam queue-provisioning einen
Eintrag unter `networks: aliases:`. Das ist Teil seiner NETZKONFIGURATION —
und Compose baut einen Container neu, sobald die sich aendert. Neu gebaut
heisst neue Adresse im Compose-Netz, heisst neu geschriebene
Weiterleitungsregeln fuer den veroeffentlichten UDP-Port 51820. Auf dem liegt
jede bestehende WireGuard-Sitzung.
Der Preis fuer einen sprechenderen Namen in einer Konfigurationszeile war
also ein Abriss saemtlicher Tunnel beim Ausrollen — der des Betreibers am
Telefon wie der jedes Hosts. Alias entfernt, nginx spricht die Bruecke unter
`queue-provisioning:8082` an. Nachgemessen: danach laesst `docker compose
up -d` den Tunnel-Container unangetastet, und nginx erreicht die Bruecke
weiterhin (426 statt 502).
2. DER ARBEITER ALS PROZESS 1. wg0 stand im Namensraum von queue-provisioning,
und dessen Prozess 1 war `exec php artisan queue:work provisioning`. Endete
der Arbeiter, endete der Container — und mit ihm der Namensraum und jede
Sitzung darin. Ein Arbeiter endet oefter, als man denkt: `--timeout=2100`
beendet ihn bei einem langen Provisionierungs-Schritt, ein fataler Fehler
beendet ihn, ein Speicherlimit beendet ihn, `queue:restart` beendet ihn
absichtlich. Aus jedem dieser vier Faelle wurde bisher "das VPN ist
unzuverlaessig".
Der Rumpf liegt jetzt in docker/provisioning-worker.sh: wg0 einmal hochziehen,
danach den Arbeiter in einer Schleife halten, SIGTERM sauber weiterreichen.
Nachgemessen: Arbeiter getoetet -> Container hat NULL Neustarts, wg0 steht
weiter, Arbeiter ist von allein wieder da.
Beide Male bleibt der Netz-Namensraum stehen. Damit verschwindet nebenbei auch
der Grund, aus dem update.sh die Bruecke hinterher neu starten musste — die
Zeile bleibt trotzdem, sie kostet nichts und traegt den Fall, dass der Container
doch einmal neu gebaut wird.
NICHT GETAGGT, mit Absicht: diese Aenderung fasst genau das an, was den Zugang
des Betreibers traegt. Sie gehoert ausgerollt, wenn jemand davorsitzt und die
Konsole des Anbieters als Rueckfahrkarte hat — nicht unbeaufsichtigt vom
Update-Agenten, waehrend der Betreiber unterwegs ist.
2508 Tests gruen.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Der Beiwagen nach dem Vorbild von kuma-bridge: Python, ein Zweck, kein
Laravel. Er steht im Netz-Namensraum von queue-provisioning, weil dort wg0
lebt und nur von dort ein Host ueberhaupt erreichbar ist.
Drei Punkte, die das Review von Aufgabe 2 offen gelassen hat, sind hier
entschieden:
- Das Ticket reist als Sec-WebSocket-Protocol, nicht in der Adresszeile.
Die Adresse eines Upgrade-Antrags schreibt jeder Reverse Proxy mit, und
auf der Strecke stehen zwei, von denen nur einer aus diesem Repo
konfiguriert wird.
- /terminal/ws bleibt an der Wurzel, unabhaengig von AdminArea::prefix():
nginx kann die Konfiguration der Anwendung nicht lesen, der Ort muss ein
fester Text sein. In beiden Dateien steht die Begruendung.
- Die Bruecke sendet ausschliesslich Binaerrahmen.
Abweichungen vom Plan, jeweils gemessen statt vermutet:
- Der Schluessel in Redis traegt REDIS_PREFIX (clupilot-database-), nicht
CACHE_PREFIX: TerminalTicket schreibt ueber Redis::connection('cache'),
und phpredis stellt die Praefix-Option dieser Verbindung voran.
- Port 8082 statt 8081: im selben Namensraum lauscht der VPN-Gateway schon
auf VPN_HEALTH_PORT, und der zweite Zuhoerer auf einem Port stirbt.
- SSH ueber paramiko.Transport statt SSHClient. SSHClient prueft gegen
known_hosts, die dieser Container nicht hat und aus einem Fingerabdruck
auch nicht bilden kann; RejectPolicy verbaende nie, AutoAddPolicy
meldete sich mit einem Root-Schluessel an, bevor irgendetwas geprueft
ist. Der Transport erlaubt die richtige Reihenfolge: Handschlag,
Fingerabdruck, dann erst Anmeldung.
- Der Fingerabdruck wird gebildet wie in PhpseclibRemoteShell, nicht wie
bei OpenSSH: gehasht wird "<algorithmus> <base64-blob>", nicht der Blob.
- proxy_pass ueber eine Variable mit resolver, damit nginx nicht beim
Start scheitert, wenn die Bruecke gerade nicht laeuft.
- /terminal/ws antwortet auf einem oeffentlichen Hostnamen mit 404,
dieselbe Regel wie /admin eine Zeile darueber.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gateway is restarted twice in a deployment — once by `up -d`, then again
after the hub whose network namespace it lives in — and `docker compose exec`
into a container that is still coming up fails outright rather than waiting.
Twenty seconds did not cover that, so the first run of the new probe reported a
gateway that answers as down, and the resolver stayed out of client configs for
one more deployment.
The health site also now binds the hub address instead of only matching on it.
Nothing was published either way, but the file should do what its comment says.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The console has been unreachable over the VPN, and the cause was the readiness
probe rather than the tunnel. The gateway binds to the WireGuard hub address
alone; the probe asked 127.0.0.1, where nothing has ever listened. It could only
fail, so VPN_READY stayed false, and on that basis the application withheld the
`DNS = 10.66.0.1` line from every client config it issued. A phone then built
the tunnel, asked its normal resolver for the console hostname, got the public
address and had its connection closed — indistinguishable from "this site does
not exist", which is exactly what it looked like.
Probing over TLS on the bare address would not have worked either: the site
matches on the console's hostname, so a request without SNI is offered no
certificate. The gateway now answers a plain-HTTP health port on the hub
address, which removes TLS, SNI and name resolution from the question and
answers only what is being asked — is this gateway listening, in this network
namespace, right now. Caddy refuses to start when the certificate is unreadable,
so a health port that answers still proves the whole file loaded.
Also here, all found while looking:
- Icons pushed their label onto a second line and rendered a size larger than
asked for. Tailwind's preflight makes an svg display:block, and `.size-4` and
`.size-5` have equal specificity, so stylesheet order decided — and it emits
size-4 first. Every icon written as 16px was silently 20px. Recorded as R18.
- Four error pages printed `errors.404.hint` in place of a sentence: the lang
files give those codes a null hint and Laravel returns the key for a null line.
- The Developer role had no label in either language, so the dropdown showed
`admin_settings.role_developer`.
- The secrets area held one key, which is not worth a password gate. It now
carries the credentials that actually stop the business when they expire —
DNS, monitoring, SMTP — and the test button appears only where a checker
exists, instead of reporting on Stripe whatever was being looked at.
- The update button said nothing about when a queued run would start or where a
running one had got to; both are now shown, and a failure names its step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The owner's phone shows a blank page on the VPN. That page is the reverse proxy
refusing the request, and it refuses because the phone is not on the tunnel: the
client config routes only the management subnet, so a request to the console's
public hostname leaves over the mobile network and arrives from a carrier
address. It is also why the peer reads "last contact: never" — WireGuard only
handshakes when it has traffic to send, and nothing is ever routed in.
The obvious fixes are both wrong. Adding the server's public address to
AllowedIPs routes the WireGuard endpoint into the tunnel it is trying to
establish. A public DNS record pointing at 10.66.0.1 publishes the internal
subnet to anyone enumerating the domain — the owner raised that himself, and he
was right.
So the console is served INSIDE the tunnel instead. A small resolver and a small
gateway share the hub container's network namespace, which is where wg0 lives,
and answer on 10.66.0.1 directly. A client therefore needs no route beyond the
subnet it already has, the host's docker bridge is never exposed, and the
request reaches the application with its real 10.66.0.x source — which is what
the console's own allowlist checks. The gateway reuses the certificate the
public Caddy already renews: a certificate is bound to the name, not to the
address serving it, so nothing new is issued and nothing internal reaches a
public zone.
Most of this commit is the arithmetic of not lying about it. Review found
fourteen ways the arrangement could report itself working while it was not:
enabling the compose profile after the services were started rather than before;
starting a gateway with empty certificate paths, which can only crash; treating
"container created" as "container running"; carrying a stale readiness marker
through a failed restart; reusing the previous hostname's certificate after a
rename; leaving the profile enabled when the hostname was cleared, in a script
that exits early precisely when nothing else would fix it; reading the renewal
timestamp as a user who cannot traverse Caddy's storage; and a `find | head`
that aborts the whole installer under pipefail the moment two issuers hold a
certificate for the same name.
One of them was mine and worth naming: to let the container read the
certificate, I had made the TLS private key world-readable. On a multi-user host
that hands the console's identity to every local account.
The client is told about the resolver only when both services are confirmed
running, because a config naming a resolver that does not exist takes the
device's entire name resolution with it for as long as the tunnel is up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The map listed only *.dev.clupilot.com, with the production names commented out
under "add the real public hostnames here before launch". Launching without
them removes the outer of the two layers that keep /admin off the public
internet — the ADMIN_HOSTS check in the app would still answer 404, but the
whole point of the pair is that neither is relied on alone.
admin.clupilot.com stays out of the list on purpose: the list names the hosts
that are PUBLIC, and the console is not one of them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
While the product is still being built, the marketing site and the customer
portal should not be reachable — but they must stay reachable for us.
- A toggle in the console (site.manage, Owner/Admin) stored in a new
app_settings table, because this has to be flippable without a deploy.
- Outsiders get a placeholder with 503 + noindex, not 200: a 200 invites search
engines to index the placeholder as the site's content, which is far harder
to undo than to prevent.
- Anyone on the management VPN, and any signed-in operator, sees the real site.
The console, Livewire's endpoint, the Stripe webhook and the health check are
always reachable — otherwise the switch could only be flipped once.
- robots.txt is generated by the app and follows the switch. It had to stop
being a static file: nginx short-circuited it and Laravel's stock file said
'Disallow:' with an empty value, so crawlers were never told anything.
- Settings reads fall back to the default when the table is unavailable. The
gate reads one on every request, so a deploy running new code before migrate
would otherwise answer the entire site with a 500.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With the Vite dev server off by default, a fresh checkout had no public/build
(gitignored, and the image does not build it), so every @vite page failed with
ViteManifestNotFoundException. The entrypoint now builds once when the manifest
is absent, non-fatally — a broken build must not wedge the container into a
restart loop where the logs are unreachable.
Proven by deleting public/build and restarting: assets rebuilt automatically,
page 200.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cancel buttons were dead: wire-elements registers Livewire.on('closeModal'),
but the buttons used Alpine's $dispatch, which fires a DOM event. Livewire
bridges its own events TO the DOM, not back, so nothing ever received them.
Proven in the browser both ways — before: modal stays open after Abbrechen;
after: it closes.
Domain operation behind Zoraxy:
- trustProxies for FOR/PORT/PROTO so Laravel sees https (otherwise it builds
http:// URLs into an https page). X_FORWARDED_HOST is deliberately NOT
trusted — the console is gated on the request host, and trusting it would let
anyone reach /admin through a public domain by forging the header.
- APP_URL + VITE_REVERB_* point at the dev domains (wss via ws.dev).
- Vite dev server is now opt-in (VITE_AUTOSTART): over https the browser cannot
load assets from http://<ip>:5173. Built assets are the default, and the
entrypoint clears a stale public/hot, which otherwise 404s every asset.
Verified in the browser over https: login, styled pages, zero console errors,
zero unencrypted requests, /admin reachable only on admin.dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Only the marketing site and the customer portal are public; /admin and the
Proxmox hosts stay on the private network. nginx was a single server_name _
catch-all, so /admin was served on all four dev domains.
Two layers:
- nginx denies /admin on the known-public hostnames before PHP is reached
(denylist, so an allowlist typo cannot lock the operators out).
- ADMIN_HOSTS is now populated, making the app layer a strict allowlist that
also covers /livewire/update, which nginx cannot attribute to /admin.
Verified live against the running stack: /admin is 404 on www/app/api/ws and on
any unlisted host, 302->login on admin.dev / the private IP / localhost, while
/ and /dashboard stay reachable everywhere.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- MONITORING_ATTEMPTS is capped by the run's max_attempts, so a large value can
no longer burn the retry budget and fail the very provisioning the degradation
exists to protect (covered by a new test at the budget boundary)
- /health is now dependency-free (200 in ~2ms with Kuma absent, verified: the
container healthcheck stays healthy); Kuma state is exclusively in /ready
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified with a reachable Kuma and deliberately wrong credentials:
/ready -> 503 'Kuma login rejected: Incorrect username or password.'
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified against a real Kuma by stopping it mid-test:
- get_monitors() reads the library's CACHED monitorList event, so it reported
'up' long after Kuma died. Reachability now does a real HTTP round-trip plus a
live socket check.
- the cached list could yield an id for a monitor that no longer exists; a match
is now confirmed with a live get_monitor before it is returned, so CluPilot can
never record a target that is never checked.
- liveness (/health, always 200 while serving) split from readiness (/ready, 503
when Kuma is unreachable) — the container healthcheck must not restart a
healthy bridge just because a dependency is down.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- attempt is 0-based, so compare one-based: ATTEMPTS=2 really means two tries
- bridge accepts add_monitor returning {monitorID}|{id}|int (v1.2.1 returns a
dict — re-verified: first create on a fresh Kuma is HTTP 200 with the id)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.
Verified end-to-end against a real Uptime Kuma 1.x:
create -> id, create again -> same id (idempotent), list, status, delete
and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.
Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
monitor legitimately reports 'pending' — confirmed in the e2e run); it records
the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Image installs wireguard-tools/iproute2; the provisioning worker brings up
wg0 before queue:work so LocalWireguardHub can manage peers.
- ConfirmRemoveHost deletes each run under its run:<uuid> lock so an in-flight
worker can't keep mutating the server or write to a deleted run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo
Claude