Zwei Ursachen, eine davon habe ich mit v1.4.2 selbst gelegt.
1. DER NETZ-ALIAS. Damit in docker/nginx/default.conf `terminal:8082` stehen
konnte statt des Namens des Nachbarcontainers, bekam queue-provisioning einen
Eintrag unter `networks: aliases:`. Das ist Teil seiner NETZKONFIGURATION —
und Compose baut einen Container neu, sobald die sich aendert. Neu gebaut
heisst neue Adresse im Compose-Netz, heisst neu geschriebene
Weiterleitungsregeln fuer den veroeffentlichten UDP-Port 51820. Auf dem liegt
jede bestehende WireGuard-Sitzung.
Der Preis fuer einen sprechenderen Namen in einer Konfigurationszeile war
also ein Abriss saemtlicher Tunnel beim Ausrollen — der des Betreibers am
Telefon wie der jedes Hosts. Alias entfernt, nginx spricht die Bruecke unter
`queue-provisioning:8082` an. Nachgemessen: danach laesst `docker compose
up -d` den Tunnel-Container unangetastet, und nginx erreicht die Bruecke
weiterhin (426 statt 502).
2. DER ARBEITER ALS PROZESS 1. wg0 stand im Namensraum von queue-provisioning,
und dessen Prozess 1 war `exec php artisan queue:work provisioning`. Endete
der Arbeiter, endete der Container — und mit ihm der Namensraum und jede
Sitzung darin. Ein Arbeiter endet oefter, als man denkt: `--timeout=2100`
beendet ihn bei einem langen Provisionierungs-Schritt, ein fataler Fehler
beendet ihn, ein Speicherlimit beendet ihn, `queue:restart` beendet ihn
absichtlich. Aus jedem dieser vier Faelle wurde bisher "das VPN ist
unzuverlaessig".
Der Rumpf liegt jetzt in docker/provisioning-worker.sh: wg0 einmal hochziehen,
danach den Arbeiter in einer Schleife halten, SIGTERM sauber weiterreichen.
Nachgemessen: Arbeiter getoetet -> Container hat NULL Neustarts, wg0 steht
weiter, Arbeiter ist von allein wieder da.
Beide Male bleibt der Netz-Namensraum stehen. Damit verschwindet nebenbei auch
der Grund, aus dem update.sh die Bruecke hinterher neu starten musste — die
Zeile bleibt trotzdem, sie kostet nichts und traegt den Fall, dass der Container
doch einmal neu gebaut wird.
NICHT GETAGGT, mit Absicht: diese Aenderung fasst genau das an, was den Zugang
des Betreibers traegt. Sie gehoert ausgerollt, wenn jemand davorsitzt und die
Konsole des Anbieters als Rueckfahrkarte hat — nicht unbeaufsichtigt vom
Update-Agenten, waehrend der Betreiber unterwegs ist.
2508 Tests gruen.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Aus dem Gesamt-Review, und der erste Befund ist der, der still zugeschlagen
haette.
`terminal` lebt im Netz-Namensraum von `queue-provisioning`, und ein Prozess
bleibt in dem Namensraum, in dem er gestartet ist. `update.sh` startet den Hub
neu — danach lauscht die Bruecke in einem, den es nicht mehr gibt. Nichts meldet
dabei einen Fehler: `docker compose ps` sagt weiter "healthy", weil die
Lebendpruefung ueber Loopback INNERHALB des verwaisten Namensraums laeuft. Nach
aussen antwortet nginx mit 502, und der Betreiber liest "Keine Verbindung —
laeuft der Terminal-Dienst?", waehrend der Dienst behauptet, es gehe ihm gut.
Genau dieselbe Falle, die zwei Bloecke tiefer schon fuer vpn-dns/vpn-gateway
behandelt ist; die Bruecke fehlte in der Behandlung.
Nachgemessen statt geglaubt: Hub neu gestartet -> Docker sagt "healthy", curl aus
dem Namensraum bekommt gar keine Antwort. Nach `restart terminal`: 200.
Und ein zweiter Ausrollfehler daneben: gebaut wurde nur `app`. `docker compose
up -d` baut nur Images, die es noch GAR NICHT gibt — beim ersten Ausrollen faellt
das nicht auf, danach nie wieder. Eine Aenderung an docker/terminal/ saehe
ausgeliefert aus, und es liefe das alte Image.
Ausserdem:
- Die Meldung zu 4502 zaehlte zwei Ursachen auf, der Code deckt fuenf. Die
Bruecke schickt 4502 fuer JEDE gescheiterte Anmeldung, auch fuer einen
abgewiesenen Schluessel — und das ist der wahrscheinlichste Fall, wenn ein Host
neu aufgesetzt wurde. "antwortet nicht" war dort schlicht falsch: die Maschine
hat geantwortet und abgelehnt. Titel und Text legen sich nicht mehr fest.
- R19: der Kommentar an der Kopfzeile der Spalte nannte "Berechtigung,
Betriebsbereitschaft" als Grund, warum der Knopf nicht ueberall steht.
Letzteres entscheidet seit dem Entsperren nichts mehr, und zwanzig Zeilen
tiefer begruendete der Kommentar am Knopf ausfuehrlich das Gegenteil.
- Der Kommentar am Retry-Knopf erklaerte die Reihenfolge von .hidden gegen
.inline-flex fuer zu unsicher, waehrend die Buehne dreissig Zeilen hoeher genau
darauf baut. Tailwind gibt .hidden als letzte Display-Klasse aus; der wahre
Grund fuer den Wrapper ist, dass die Klassen des Knopfes aus einem geteilten
Bauteil kommen.
- REDIS_URL stand fest auf Datenbank 1, waehrend PHP REDIS_CACHE_DB liest. Wer
die anfasst, legt auf der einen Seite ab, wo die andere nicht sucht.
2507 Tests gruen, compose config und bash -n sauber.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Der Beiwagen nach dem Vorbild von kuma-bridge: Python, ein Zweck, kein
Laravel. Er steht im Netz-Namensraum von queue-provisioning, weil dort wg0
lebt und nur von dort ein Host ueberhaupt erreichbar ist.
Drei Punkte, die das Review von Aufgabe 2 offen gelassen hat, sind hier
entschieden:
- Das Ticket reist als Sec-WebSocket-Protocol, nicht in der Adresszeile.
Die Adresse eines Upgrade-Antrags schreibt jeder Reverse Proxy mit, und
auf der Strecke stehen zwei, von denen nur einer aus diesem Repo
konfiguriert wird.
- /terminal/ws bleibt an der Wurzel, unabhaengig von AdminArea::prefix():
nginx kann die Konfiguration der Anwendung nicht lesen, der Ort muss ein
fester Text sein. In beiden Dateien steht die Begruendung.
- Die Bruecke sendet ausschliesslich Binaerrahmen.
Abweichungen vom Plan, jeweils gemessen statt vermutet:
- Der Schluessel in Redis traegt REDIS_PREFIX (clupilot-database-), nicht
CACHE_PREFIX: TerminalTicket schreibt ueber Redis::connection('cache'),
und phpredis stellt die Praefix-Option dieser Verbindung voran.
- Port 8082 statt 8081: im selben Namensraum lauscht der VPN-Gateway schon
auf VPN_HEALTH_PORT, und der zweite Zuhoerer auf einem Port stirbt.
- SSH ueber paramiko.Transport statt SSHClient. SSHClient prueft gegen
known_hosts, die dieser Container nicht hat und aus einem Fingerabdruck
auch nicht bilden kann; RejectPolicy verbaende nie, AutoAddPolicy
meldete sich mit einem Root-Schluessel an, bevor irgendetwas geprueft
ist. Der Transport erlaubt die richtige Reihenfolge: Handschlag,
Fingerabdruck, dann erst Anmeldung.
- Der Fingerabdruck wird gebildet wie in PhpseclibRemoteShell, nicht wie
bei OpenSSH: gehasht wird "<algorithmus> <base64-blob>", nicht der Blob.
- proxy_pass ueber eine Variable mit resolver, damit nginx nicht beim
Start scheitert, wenn die Bruecke gerade nicht laeuft.
- /terminal/ws antwortet auf einem oeffentlichen Hostnamen mit 404,
dieselbe Regel wie /admin eine Zeile darueber.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
── The 500 ─────────────────────────────────────────────────────────────────
"touch(): Utime failed: Operation not permitted", from BladeCompiler, on
every request for the affected view.
Blade compiles a view and then touch()es the compiled file to the source's
mtime. touch() with an explicit time needs ownership of the file. A compose
service without `user:` runs as ROOT — and queue, scheduler and reverb had
none — so the first worker to render a view wrote a root-owned compiled file
that the web process, as www-data, could never refresh again. The queue is
what renders mails, which is why this surfaced now.
queue, reverb and scheduler run as www-data. queue-provisioning stays root
and says why: it brings wg0 up and runs `wg set` for every peer change, which
needs NET_ADMIN on the running process. It renders no mail.
And update.sh normalises ownership at the END of a run as well as at the
start. The first call heals what a previous run left; the second heals what
this one made — `git checkout` rewrites the tree as the service account while
the old build is still serving.
── The panel that would not go away ─────────────────────────────────────────
The same 500 is why it kept coming back: every poll failed, the watcher
treats a failed request as "still restarting" (which it normally is), and the
overlay stayed up over a console nobody could then reach to find out why.
A restart is seconds. After two minutes of nothing the panel now says so and
offers a way out — for an operator only, behind the same flag as the step and
the log, so a customer on the 503 page is not shown a button suggesting they
can call the deployment off. A successful answer clears it again: one bad
minute must not leave a "something is wrong" notice sitting there for the
rest of the run.
Not an always-present close button. The deployment does not stop because
somebody dismissed a panel, and offering that at the wrong moment is a lie.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The console has been unreachable over the VPN, and the cause was the readiness
probe rather than the tunnel. The gateway binds to the WireGuard hub address
alone; the probe asked 127.0.0.1, where nothing has ever listened. It could only
fail, so VPN_READY stayed false, and on that basis the application withheld the
`DNS = 10.66.0.1` line from every client config it issued. A phone then built
the tunnel, asked its normal resolver for the console hostname, got the public
address and had its connection closed — indistinguishable from "this site does
not exist", which is exactly what it looked like.
Probing over TLS on the bare address would not have worked either: the site
matches on the console's hostname, so a request without SNI is offered no
certificate. The gateway now answers a plain-HTTP health port on the hub
address, which removes TLS, SNI and name resolution from the question and
answers only what is being asked — is this gateway listening, in this network
namespace, right now. Caddy refuses to start when the certificate is unreadable,
so a health port that answers still proves the whole file loaded.
Also here, all found while looking:
- Icons pushed their label onto a second line and rendered a size larger than
asked for. Tailwind's preflight makes an svg display:block, and `.size-4` and
`.size-5` have equal specificity, so stylesheet order decided — and it emits
size-4 first. Every icon written as 16px was silently 20px. Recorded as R18.
- Four error pages printed `errors.404.hint` in place of a sentence: the lang
files give those codes a null hint and Laravel returns the key for a null line.
- The Developer role had no label in either language, so the dropdown showed
`admin_settings.role_developer`.
- The secrets area held one key, which is not worth a password gate. It now
carries the credentials that actually stop the business when they expire —
DNS, monitoring, SMTP — and the test button appears only where a checker
exists, instead of reporting on Stripe whatever was being looked at.
- The update button said nothing about when a queued run would start or where a
running one had got to; both are now shown, and a failure names its step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An empty 404 is itself information: it says something terminates TLS here and
chose not to answer. The console and the websocket endpoint now close the
connection instead, which is what a hostname that serves nothing looks like.
The websocket endpoint answers a genuine upgrade — verified live, 101 — and
nothing else; a browser opening it gets a closed connection.
Reviewing that turned up two holes that mattered more than the thing being
reviewed.
The compose defaults published the application and Reverb on every interface.
Docker publishes ports ahead of UFW, so those backends were reachable from the
internet with the firewall closed — and reaching a backend directly skips the
proxy, and with it every hostname and address rule keeping the console private.
The defaults are loopback now, and an update rebinds an existing installation
that is behind a proxy actually running and holding 443. A development box
without one is left alone, because taking its port away would look like the
machine had broken.
And the console's own "open" switch was emitting 0.0.0.0/0 into the proxy's
allowlist. One click would have put the console on the public internet, which
is the exact state the owner's rule exists to prevent. Switching it off relaxes
the application's check; the proxy keeps its list, always.
The reference proxy config carries the arrangement: QUIC early data off, since
an address-based decision taken on 0-RTT can answer 425 and some browsers do
not retry — an intermittent lockout from the console is the worst kind — and
explicit handling for plain HTTP, whose automatic redirect otherwise announces
both hidden names to anyone who asks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dnsmasq image was pinned to 2.90-r0, which is not published. The gateway
therefore never started — correctly reported as not ready, so no client was
handed a resolver that was not there, but the tunnel console did not come up
either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The owner's phone shows a blank page on the VPN. That page is the reverse proxy
refusing the request, and it refuses because the phone is not on the tunnel: the
client config routes only the management subnet, so a request to the console's
public hostname leaves over the mobile network and arrives from a carrier
address. It is also why the peer reads "last contact: never" — WireGuard only
handshakes when it has traffic to send, and nothing is ever routed in.
The obvious fixes are both wrong. Adding the server's public address to
AllowedIPs routes the WireGuard endpoint into the tunnel it is trying to
establish. A public DNS record pointing at 10.66.0.1 publishes the internal
subnet to anyone enumerating the domain — the owner raised that himself, and he
was right.
So the console is served INSIDE the tunnel instead. A small resolver and a small
gateway share the hub container's network namespace, which is where wg0 lives,
and answer on 10.66.0.1 directly. A client therefore needs no route beyond the
subnet it already has, the host's docker bridge is never exposed, and the
request reaches the application with its real 10.66.0.x source — which is what
the console's own allowlist checks. The gateway reuses the certificate the
public Caddy already renews: a certificate is bound to the name, not to the
address serving it, so nothing new is issued and nothing internal reaches a
public zone.
Most of this commit is the arithmetic of not lying about it. Review found
fourteen ways the arrangement could report itself working while it was not:
enabling the compose profile after the services were started rather than before;
starting a gateway with empty certificate paths, which can only crash; treating
"container created" as "container running"; carrying a stale readiness marker
through a failed restart; reusing the previous hostname's certificate after a
rename; leaving the profile enabled when the hostname was cleared, in a script
that exits early precisely when nothing else would fix it; reading the renewal
timestamp as a user who cannot traverse Caddy's storage; and a `find | head`
that aborts the whole installer under pipefail the moment two issuers hold a
certificate for the same name.
One of them was mine and worth naming: to let the container read the
certificate, I had made the TLS private key world-readable. On a multi-user host
that hands the console's identity to every local account.
The client is told about the resolver only when both services are confirmed
running, because a config naming a resolver that does not exist takes the
device's entire name resolution with it for as long as the tunnel is up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github.* is the documented context alias and exists in 1.20; gitea.* came
later. The runner is pinned instead of :latest — the registration protocol
moves with the server, and a newer runner refuses to register against an older
Gitea.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three things that bit us on every deploy:
- artisan down now renders a branded page that says an update is running and
reloads itself, instead of Laravel's bare 503.
- A page left open across a deploy carries a Livewire snapshot and CSRF token
the new code rejects — the user got "419 Page Expired" and a dead interface.
A 419 from a /livewire/ request now reloads the page; the session is still
valid, so that is all it takes. Hooked at the fetch layer rather than through
Livewire's request hook, whose failure callback is not invoked for this case
in the installed version — verified against a real 419 in the browser, not a
simulated one.
- Vite no longer empties its output directory: wiping it is what left an open
page without its stylesheet mid-deploy, which is the "design is completely
broken" symptom. update.sh also rebuilds the caches with optimize:clear +
optimize instead of leaving them half-warm.
Plus CI: .gitea/workflows/tests.yml (Pest + asset build, and a tested- tag only
on a green main) and an opt-in act_runner service under the ci profile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cancel buttons were dead: wire-elements registers Livewire.on('closeModal'),
but the buttons used Alpine's $dispatch, which fires a DOM event. Livewire
bridges its own events TO the DOM, not back, so nothing ever received them.
Proven in the browser both ways — before: modal stays open after Abbrechen;
after: it closes.
Domain operation behind Zoraxy:
- trustProxies for FOR/PORT/PROTO so Laravel sees https (otherwise it builds
http:// URLs into an https page). X_FORWARDED_HOST is deliberately NOT
trusted — the console is gated on the request host, and trusting it would let
anyone reach /admin through a public domain by forging the header.
- APP_URL + VITE_REVERB_* point at the dev domains (wss via ws.dev).
- Vite dev server is now opt-in (VITE_AUTOSTART): over https the browser cannot
load assets from http://<ip>:5173. Built assets are the default, and the
entrypoint clears a stale public/hot, which otherwise 404s every asset.
Verified in the browser over https: login, styled pages, zero console errors,
zero unencrypted requests, /admin reachable only on admin.dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.
Verified end-to-end against a real Uptime Kuma 1.x:
create -> id, create again -> same id (idempotent), list, status, delete
and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.
Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
monitor legitimately reports 'pending' — confirmed in the e2e run); it records
the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Image installs wireguard-tools/iproute2; the provisioning worker brings up
wg0 before queue:work so LocalWireguardHub can manage peers.
- ConfirmRemoveHost deletes each run under its run:<uuid> lock so an in-flight
worker can't keep mutating the server or write to a deleted run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- RunRunner catches step-resolution errors (removed/renamed pipeline) and
fails the run terminally instead of looping forever every tick.
- queue-provisioning worker gains NET_ADMIN + tun + persistent wireguard
volume + published UDP port so LocalWireguardHub can manage wg0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Dedicated 'provisioning' queue connection (retry_after 2400) + worker
(--timeout=2100); AdvanceRunJob tries=1, timeout=2100 on that queue.
- Run lock TTL raised to 2100s so a long step can't be run concurrently by a
duplicate tick-dispatched job.
- StartHostOnboarding creates host+run in a transaction, dispatches after commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
env_file: .env exported the dev DB/cache config as real container env vars,
which overrode phpunit's forced test env — so the Pest suite ran RefreshDatabase
against the dev MariaDB and wiped it on every run. Laravel already reads .env
from the bind mount; only vite needs VITE_HMR_HOST/VITE_PORT at process level,
now injected explicitly. Tests now use sqlite :memory: and never touch dev data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo
Claude