── The 500 ─────────────────────────────────────────────────────────────────
"touch(): Utime failed: Operation not permitted", from BladeCompiler, on
every request for the affected view.
Blade compiles a view and then touch()es the compiled file to the source's
mtime. touch() with an explicit time needs ownership of the file. A compose
service without `user:` runs as ROOT — and queue, scheduler and reverb had
none — so the first worker to render a view wrote a root-owned compiled file
that the web process, as www-data, could never refresh again. The queue is
what renders mails, which is why this surfaced now.
queue, reverb and scheduler run as www-data. queue-provisioning stays root
and says why: it brings wg0 up and runs `wg set` for every peer change, which
needs NET_ADMIN on the running process. It renders no mail.
And update.sh normalises ownership at the END of a run as well as at the
start. The first call heals what a previous run left; the second heals what
this one made — `git checkout` rewrites the tree as the service account while
the old build is still serving.
── The panel that would not go away ─────────────────────────────────────────
The same 500 is why it kept coming back: every poll failed, the watcher
treats a failed request as "still restarting" (which it normally is), and the
overlay stayed up over a console nobody could then reach to find out why.
A restart is seconds. After two minutes of nothing the panel now says so and
offers a way out — for an operator only, behind the same flag as the step and
the log, so a customer on the 503 page is not shown a button suggesting they
can call the deployment off. A successful answer clears it again: one bad
minute must not leave a "something is wrong" notice sitting there for the
rest of the run.
Not an always-present close button. The deployment does not stop because
somebody dismissed a panel, and offering that at the wrong moment is a lie.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The console has been unreachable over the VPN, and the cause was the readiness
probe rather than the tunnel. The gateway binds to the WireGuard hub address
alone; the probe asked 127.0.0.1, where nothing has ever listened. It could only
fail, so VPN_READY stayed false, and on that basis the application withheld the
`DNS = 10.66.0.1` line from every client config it issued. A phone then built
the tunnel, asked its normal resolver for the console hostname, got the public
address and had its connection closed — indistinguishable from "this site does
not exist", which is exactly what it looked like.
Probing over TLS on the bare address would not have worked either: the site
matches on the console's hostname, so a request without SNI is offered no
certificate. The gateway now answers a plain-HTTP health port on the hub
address, which removes TLS, SNI and name resolution from the question and
answers only what is being asked — is this gateway listening, in this network
namespace, right now. Caddy refuses to start when the certificate is unreadable,
so a health port that answers still proves the whole file loaded.
Also here, all found while looking:
- Icons pushed their label onto a second line and rendered a size larger than
asked for. Tailwind's preflight makes an svg display:block, and `.size-4` and
`.size-5` have equal specificity, so stylesheet order decided — and it emits
size-4 first. Every icon written as 16px was silently 20px. Recorded as R18.
- Four error pages printed `errors.404.hint` in place of a sentence: the lang
files give those codes a null hint and Laravel returns the key for a null line.
- The Developer role had no label in either language, so the dropdown showed
`admin_settings.role_developer`.
- The secrets area held one key, which is not worth a password gate. It now
carries the credentials that actually stop the business when they expire —
DNS, monitoring, SMTP — and the test button appears only where a checker
exists, instead of reporting on Stripe whatever was being looked at.
- The update button said nothing about when a queued run would start or where a
running one had got to; both are now shown, and a failure names its step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An empty 404 is itself information: it says something terminates TLS here and
chose not to answer. The console and the websocket endpoint now close the
connection instead, which is what a hostname that serves nothing looks like.
The websocket endpoint answers a genuine upgrade — verified live, 101 — and
nothing else; a browser opening it gets a closed connection.
Reviewing that turned up two holes that mattered more than the thing being
reviewed.
The compose defaults published the application and Reverb on every interface.
Docker publishes ports ahead of UFW, so those backends were reachable from the
internet with the firewall closed — and reaching a backend directly skips the
proxy, and with it every hostname and address rule keeping the console private.
The defaults are loopback now, and an update rebinds an existing installation
that is behind a proxy actually running and holding 443. A development box
without one is left alone, because taking its port away would look like the
machine had broken.
And the console's own "open" switch was emitting 0.0.0.0/0 into the proxy's
allowlist. One click would have put the console on the public internet, which
is the exact state the owner's rule exists to prevent. Switching it off relaxes
the application's check; the proxy keeps its list, always.
The reference proxy config carries the arrangement: QUIC early data off, since
an address-based decision taken on 0-RTT can answer 425 and some browsers do
not retry — an intermittent lockout from the console is the worst kind — and
explicit handling for plain HTTP, whose automatic redirect otherwise announces
both hidden names to anyone who asks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dnsmasq image was pinned to 2.90-r0, which is not published. The gateway
therefore never started — correctly reported as not ready, so no client was
handed a resolver that was not there, but the tunnel console did not come up
either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The owner's phone shows a blank page on the VPN. That page is the reverse proxy
refusing the request, and it refuses because the phone is not on the tunnel: the
client config routes only the management subnet, so a request to the console's
public hostname leaves over the mobile network and arrives from a carrier
address. It is also why the peer reads "last contact: never" — WireGuard only
handshakes when it has traffic to send, and nothing is ever routed in.
The obvious fixes are both wrong. Adding the server's public address to
AllowedIPs routes the WireGuard endpoint into the tunnel it is trying to
establish. A public DNS record pointing at 10.66.0.1 publishes the internal
subnet to anyone enumerating the domain — the owner raised that himself, and he
was right.
So the console is served INSIDE the tunnel instead. A small resolver and a small
gateway share the hub container's network namespace, which is where wg0 lives,
and answer on 10.66.0.1 directly. A client therefore needs no route beyond the
subnet it already has, the host's docker bridge is never exposed, and the
request reaches the application with its real 10.66.0.x source — which is what
the console's own allowlist checks. The gateway reuses the certificate the
public Caddy already renews: a certificate is bound to the name, not to the
address serving it, so nothing new is issued and nothing internal reaches a
public zone.
Most of this commit is the arithmetic of not lying about it. Review found
fourteen ways the arrangement could report itself working while it was not:
enabling the compose profile after the services were started rather than before;
starting a gateway with empty certificate paths, which can only crash; treating
"container created" as "container running"; carrying a stale readiness marker
through a failed restart; reusing the previous hostname's certificate after a
rename; leaving the profile enabled when the hostname was cleared, in a script
that exits early precisely when nothing else would fix it; reading the renewal
timestamp as a user who cannot traverse Caddy's storage; and a `find | head`
that aborts the whole installer under pipefail the moment two issuers hold a
certificate for the same name.
One of them was mine and worth naming: to let the container read the
certificate, I had made the TLS private key world-readable. On a multi-user host
that hands the console's identity to every local account.
The client is told about the resolver only when both services are confirmed
running, because a config naming a resolver that does not exist takes the
device's entire name resolution with it for as long as the tunnel is up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github.* is the documented context alias and exists in 1.20; gitea.* came
later. The runner is pinned instead of :latest — the registration protocol
moves with the server, and a newer runner refuses to register against an older
Gitea.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three things that bit us on every deploy:
- artisan down now renders a branded page that says an update is running and
reloads itself, instead of Laravel's bare 503.
- A page left open across a deploy carries a Livewire snapshot and CSRF token
the new code rejects — the user got "419 Page Expired" and a dead interface.
A 419 from a /livewire/ request now reloads the page; the session is still
valid, so that is all it takes. Hooked at the fetch layer rather than through
Livewire's request hook, whose failure callback is not invoked for this case
in the installed version — verified against a real 419 in the browser, not a
simulated one.
- Vite no longer empties its output directory: wiping it is what left an open
page without its stylesheet mid-deploy, which is the "design is completely
broken" symptom. update.sh also rebuilds the caches with optimize:clear +
optimize instead of leaving them half-warm.
Plus CI: .gitea/workflows/tests.yml (Pest + asset build, and a tested- tag only
on a green main) and an opt-in act_runner service under the ci profile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cancel buttons were dead: wire-elements registers Livewire.on('closeModal'),
but the buttons used Alpine's $dispatch, which fires a DOM event. Livewire
bridges its own events TO the DOM, not back, so nothing ever received them.
Proven in the browser both ways — before: modal stays open after Abbrechen;
after: it closes.
Domain operation behind Zoraxy:
- trustProxies for FOR/PORT/PROTO so Laravel sees https (otherwise it builds
http:// URLs into an https page). X_FORWARDED_HOST is deliberately NOT
trusted — the console is gated on the request host, and trusting it would let
anyone reach /admin through a public domain by forging the header.
- APP_URL + VITE_REVERB_* point at the dev domains (wss via ws.dev).
- Vite dev server is now opt-in (VITE_AUTOSTART): over https the browser cannot
load assets from http://<ip>:5173. Built assets are the default, and the
entrypoint clears a stale public/hot, which otherwise 404s every asset.
Verified in the browser over https: login, styled pages, zero console errors,
zero unencrypted requests, /admin reachable only on admin.dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.
Verified end-to-end against a real Uptime Kuma 1.x:
create -> id, create again -> same id (idempotent), list, status, delete
and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.
Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
monitor legitimately reports 'pending' — confirmed in the e2e run); it records
the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Image installs wireguard-tools/iproute2; the provisioning worker brings up
wg0 before queue:work so LocalWireguardHub can manage peers.
- ConfirmRemoveHost deletes each run under its run:<uuid> lock so an in-flight
worker can't keep mutating the server or write to a deleted run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- RunRunner catches step-resolution errors (removed/renamed pipeline) and
fails the run terminally instead of looping forever every tick.
- queue-provisioning worker gains NET_ADMIN + tun + persistent wireguard
volume + published UDP port so LocalWireguardHub can manage wg0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Dedicated 'provisioning' queue connection (retry_after 2400) + worker
(--timeout=2100); AdvanceRunJob tries=1, timeout=2100 on that queue.
- Run lock TTL raised to 2100s so a long step can't be run concurrently by a
duplicate tick-dispatched job.
- StartHostOnboarding creates host+run in a transaction, dispatches after commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
env_file: .env exported the dev DB/cache config as real container env vars,
which overrode phpunit's forced test env — so the Pest suite ran RefreshDatabase
against the dev MariaDB and wiped it on every run. Laravel already reads .env
from the bind mount; only vite needs VITE_HMR_HOST/VITE_PORT at process level,
now injected explicitly. Tests now use sqlite :memory: and never touch dev data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo
Claude