45 Commits (87264ef1f7b566d085224fc381e2f6236da385cf)

Author	SHA1	Message	Date
nexxo	006ce3568b	Manage hostnames and watch their certificates from the console ... files.clupilot.com is why this exists. DNS pointed at the right machine, the env var was set, the release was deployed — and there was still no certificate, because /etc/caddy/Caddyfile is maintained by hand and nobody thought of it as a second, separate step. Nothing in the console would have said so. Three settings looked correct and the address did not answer. So the page holds two things side by side. The WISH — which names should be served — and the REALITY: whether the name has a certificate and for how much longer. The second is measured by opening a TLS connection and reading the expiry, not by reading configuration, because the configuration is exactly what looked right while the address was dead. verify_peer stays on: a certificate that fails validation is not a certificate for this question, and a display that called it valid would be the fake R19 records. Applying goes through the existing agent, not a new channel. The console writes a request, the path unit wakes the agent within a second, and the agent calls one fixed command line of the root-owned helper. What that helper is allowed to do is the careful part. It fetches the list ITSELF rather than being handed one, and the list is HOSTNAMES, never Caddy blocks — `php artisan clupilot:proxy-hosts` prints `<name> <purpose>` and nothing else. Each name is matched against a strict pattern before it is used, and the template around it lives in the helper, which the service account cannot touch. install-agent.sh already states the principle for the sudoers grant: a grant is only worth anything if the holder cannot change what it grants. A service account that could write proxy configuration would have everything the proxy can do — redirects anywhere, files from any directory. Purpose is a column rather than a habit. A console name gets the network lock, a public one does not, and a console name published without it looks exactly like a working page. Removing takes the name out of the list and NOT out of the running proxy. Two decisions in one click, and the second one takes a site off the air. There is deliberately no "renew" button. Caddy renews on its own at two thirds of the lifetime; what an operator actually needs is a second attempt after an issuance has failed, and that is a reload — which is what Apply does. Thirty days is treated as a problem rather than a warning: at ninety days' lifetime, a renewal should long since have run, so anything under it is not a tight certificate but a renewal that is not happening. The ACME contact falls back to the owner's address, because a contact nobody reads is the step before expired customer certificates. CONTRACT and HOST_STEP_NEEDS both move to 2, which is what tells a server carrying the older helper to run the installer again — caught by the guard test that compares the two halves. 2035 tests pass, assets build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 22:55:55 +02:00
nexxo	20d87c0474	Put the files hostname in the reference proxy configuration ... FILES_HOST tells Laravel which hostname to bind the routes to. It does not make the reverse proxy terminate TLS for that name or forward it anywhere, and the live server showed exactly that gap: DNS correct, the right machine, port 80 answering — and a TLS handshake that fails because no certificate exists for the name. The block carries two things the other public names do not have to think about. No console allowlist. The archive is fetched by a server in a RESCUE SYSTEM: a machine with no tunnel and a public address that is in nobody's allow list. The console's network lock here would be a lock against the only caller that needs it. The protection lives in the application instead — no valid one-time code, 404 — and the legal documents are public by intent. And no `http://… { abort }` either, unlike admin and ws below it. Those two hide that they exist; this one is meant to be known, and killing port 80 takes the path away from an ACME check on the day Caddy's own challenge handler is not the first thing to answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 22:34:15 +02:00
nexxo	ba4996f316	Keep the platform on .com and the customers on .cloud ... The owner caught me putting the archive on clupilot.cloud. That is the customer zone, and looking into it turned up the same confusion already sitting in the code — in the one place that matters most. RegisterHostDns says "fsn-01.node.clupilot.com" in its own docblock. The validation comment in Datacenters says it. ServicesTest writes it out verbatim. The step itself built the name from config('provisioning.dns.zone') — the CUSTOMER zone — so on this installation a host was actually called fsn-01.node.clupilot.cloud. Three places asserting one thing and the code doing another. OfficialDomains explains why that matters and is worth not weakening: two registrable domains by design, the company's for site, portal and console, the instance zone for customer workloads. A Nextcloud is third-party software that strangers sign into, and on the same registrable domain as the portal it shares cookie scope with it. A host name in that zone does not break the separation, but it puts it in question, and the next slip is more expensive. So there is now a platform_zone, derived from APP_URL when unset, and RegisterHostDns uses it. My own archiveUrl was broken for a second reason. It fell back to url() when APP_HOST is empty — and APP_HOST is empty on most installations, because empty means "the portal answers on any hostname" and that is the default. Called from the console, url() would have produced the CONSOLE hostname, and the line would have 404'd on a machine that is not allowed to reach the admin area at all. It takes the host from APP_URL now. The script no longer guesses its own name. It used reverse DNS, then the tunnel address, then a hard-coded clupilot.net — a third domain that appears nowhere else in this project and was simply invented. PrepareBaseSystem has the same invention. A guessed name does not stay guessed: it ends up in /etc/hostname, in /etc/hosts, in every log line and in every certificate request the machine ever makes. CluPilot knows the name because it just assigned it, so it passes --fqdn and the script refuses without it. The value now also survives the reboot in the arguments file, which it would not have. The ACME contact moved to .com for the same reason it was wrong: the operator does not live in the customer zone. Open, and NOT decided here: the owner also wants the host to get a public DNS record and a certificate on the .com name. RegisterHostDns deliberately writes host names only into the tunnel's dnsmasq, and says why — publishing them hands every scanner the internal subnet and roughly how many hosts sit behind it. Nothing in the current design needs a public certificate for a host's own name; Traefik serves customer domains, not this one. Reversing that is a security decision and belongs to the owner, not to this commit. 1992 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 21:04:08 +02:00
nexxo	f45f290f00	Hand over the token and let the console take it from here ... The last section carries three things, and none of them could sit anywhere else: the registration and key swap, Traefik's token plus the proof that a route table actually arrives, and the nftables lockdown that ran last in the old pipeline for a reason. `|| true` is gone from the pveum calls, and not by deleting it. `role add || true` applied the privilege list only on the run that first created the role, so a privilege added later reached new hosts and no existing one — which is exactly how Sys.Modify was missing everywhere, surfacing much later and somewhere else as a 403 on POST /cluster/backup that failed a paying customer at register_backup. So: add, and on failure modify, as two separate commands so a failure can be attributed. For the user and the ACL the tolerated `|| true` is replaced by looking: try, and if it fails, check whether the desired state holds anyway. A `|| true` hides "already there" and "went wrong" equally well. The privilege list is copied verbatim from config/provisioning.php and checked against it — 18 privileges, byte-identical, Sys.Modify included. It is granted on / because both endpoints that need it check / and nothing narrower satisfies them. The key swap follows the four steps this plan was amended to require. Register first and hold the answer, switch wg0 to the new private key, PROVE a fresh handshake, and only then let the old key go. If the handshake does not come, it puts the old key back and fails loudly. Ordering alone says when to discard, not whether the new key carries — and a host that locks itself out on the final step is the one failure nobody fixes remotely. Traefik's token is written here because here is where it exists, and the route table is then fetched directly with that token and that URL. Section 7 could not give this proof; it is not skipped, it is given where it can be. The lockdown comes dead last, with the full ruleset from SecureHostFirewall including the ICMP correction — ICMPv6 neighbour discovery and packet-too-big, IPv4 fragmentation-needed — because a bare policy drop takes IPv6 down within minutes and black-holes large transfers instead of failing them. The ruleset is validated with `nft -c` before it is applied, since a partially loaded ruleset means policy drop is in place and the exceptions are not. There is no automatic reopening: a firewall that reopens itself under failure is not a firewall, so the emergency release is a script for the provider's console. Verified without hardware: all seven files dash-clean; json_field reads full-tokenid, value and host_token without jq; the privilege list is identical to the config's; and the rendered ruleset carries policy drop, both ICMP families, 80/443, the DHCP rebind rule and the tunnel-only 22/8006. Step 2 unticked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 20:20:20 +02:00
nexxo	5d25018cc6	Build the golden template so nobody has to remember how ... All three Block A traps are CHECKED here rather than assumed, because each one has already cost a paid order and none of them shows a symptom until it is too late. Trap 2 is checked before anything is built: virt-filesystems is asked whether the image uses LVM and whether root is the last partition, and the section refuses if either is wrong. Debian's cloud image satisfies both by construction, which is exactly why it is the base — but "satisfies it by construction" is a claim about an artefact somebody else builds, and it gets verified rather than trusted. Traps 1 and 3 are checked AFTER the image is customised, because virt-customize reports success even when apt quietly did not install something. qemu-guest-agent has to be in /usr/bin or every provisioning run hangs in WaitForGuestAgent until it times out, and the compose file inside the image has to carry `user: www-data` or every occ call fails — including the acceptance check that decides whether a customer's instance is usable. The template is verified by its ATTRIBUTE, not its existence. VerifyVmTemplate checks only that 9000 is there, which a VM that merely happens to be numbered 9000 also passes; `template: 1` is passed only by a template. Docker comes from Docker's own repository, not Debian's. docker.io ships no compose plugin and Debian's docker-compose is the old Python one, which does not read this file at all. The compose file is deliberately customer-independent: no password, no name, no domain. Everything variable arrives in /opt/nextcloud/.env, written into the guest by cloud-init at clone time. Baking a password into an image copies it onto every host and into every instance, and leaves it there long after the customer has changed it. OVERWRITEPROTOCOL, OVERWRITEHOST and TRUSTED_PROXIES are set because TLS ends at Traefik on the host. Without them Nextcloud builds its own URLs with http://, the login loops, and WebDAV clients get handed an address that does not exist. Storage is discovered rather than named: local-zfs is what the PVE installer creates on ZFS, local-lvm on ext4, and `local` frequently cannot hold disks at all — an importdisk there fails only after the copy. Verified without hardware: dash-clean; the compose file parses, carries `user: www-data` on the app service, publishes 80, and every credential is a ${...} reference rather than a literal. Step 2 unticked, and it is the one the plan is most insistent about: the template must actually be cloned, started, and asked `occ status` as www-data. Without that clone it is not proven. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 20:16:50 +02:00
nexxo	a25e9c2fe6	Run Traefik as a service that fetches its own routes ... Binary plus systemd unit, no Docker on the hypervisor: nftables stays the only owner of the host's firewall rules and a Docker daemon would bring its own chain. The names were read out of SshTraefikWriter::render() rather than guessed, which is what the handoff asks for in as many words. The route endpoint emits entryPoints ["websecure"] and certResolver "letsencrypt", so the static config declares exactly those. Name them differently here and the routers point at nothing while Traefik still reports a clean start — a failure with no symptom at the place it happens. An ordering problem this plan had not resolved: the http provider needs the durable host token, and that token only exists after POST /host/register, two sections later. Rather than reorder the section keys — they are the contract with the platform plan — the static config is written twice. Here with an empty token, so the service stands and holds 80 and 443, and again in Task 9 with the real one. The proof does not disappear, it moves to where it can be given. That proof asks the endpoint directly with the same token and URL the config carries, instead of counting Traefik's routers. A fresh host has no customers, so its table is legitimately empty, and "zero routers" would mean both "fine" and "never fetched". Ports are checked separately from the service. "Running" and "listening" are two claims, and Traefik starts cleanly even when a typo means an entryPoint was never created. The binary is checksummed against the release's own checksums file, for the same reason the ISO is: what listens on 80 and 443 and holds every customer's certificate does not get taken off the network unverified. The version is discovered at runtime, because a pinned number becomes a 404 mid-takeover. acme.json is created at 600 before Traefik ever runs. It holds the private keys of every customer certificate, and Traefik refuses wider permissions — rightly. The unit runs with CAP_NET_BIND_SERVICE and nothing else, ProtectSystem=strict, NoNewPrivileges: this is the one process on the box reachable from the open internet. Verified without hardware: dash-clean, the generated config parses as YAML, and it carries web/websecure/traefik as entryPoints, letsencrypt as the resolver, the Bearer header on the http provider, web redirecting to websecure, and 640 on the config with 600 on acme.json and the token file. Step 2 unticked — it wants a fetched route set, and that is Task 9's to show. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 20:13:54 +02:00
nexxo	c345f21e7d	Join the tunnel, then tell the console everything so far ... The three interface states come over from ConfigureWireguard.php exactly as that file works them out, because the one line they replace got two of them wrong. `systemctl enable --now wg-quick@wg0 || wg-quick up wg0` brought the interface up via its fallback WITHOUT the systemd enablement, so the tunnel did not come back after a reboot; and on the next attempt both halves failed with "wg0 already exists", so it retried forever against a tunnel that was working. Enable before start, because enable is the half that survives a reboot, and it is repaired even when the interface is already up by other means. The configuration is compared before it is written. The file used to be rewritten unconditionally with nothing reloading it, so correcting a wrong hub key showed a new file and identical behaviour — the running interface still held the old peer. Knowing whether it CHANGED is what lets an unchanged, working tunnel be left alone and a corrected one actually be applied. A change means a restart rather than `wg syncconf`, because syncconf applies peers only and a corrected Address or AllowedIPs would silently do nothing. AllowedIPs is computed, not copied. Writing the host's own address there would let the tunnel handshake and leave every other participant unreachable — a fault that looks like a routing problem somewhere else entirely. The handshake target is derived from --api rather than taking its own argument. CluPilot's tunnel address is already in there, and a second value meaning the same thing is a second value that eventually disagrees with the first. The tunnel counts only when it has been PROVEN, never when the file is on disk. A stored tunnel address without proof made every later connection attempt useless and the recovery was hand-editing the database. Once it is proven, flush_reports runs and the whole history from the rescue system onwards reaches the console carrying its original timestamps. Verified without hardware: dash-clean; the subnet arithmetic is right on octet boundaries and off them (192.168.5.130/25 gives 192.168.5.128/25, 172.16.4.9/12 gives 172.16.0.0/12); the API host parser handles a bare address, a scheme, a port and a path; and wg0.conf renders with AllowedIPs on the network rather than the host. Step 2 unticked — it wants all five earlier sections showing up in the console with real timestamps, and that needs a hub. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 20:08:39 +02:00
nexxo	e9b2862331	Bridge the network, and take it back if the host goes quiet ... Reading ConfigureProxmox.php changed what this section does. The missing vmbr0 that killed the first paid order is a consequence of installing Proxmox ON TOP OF DEBIAN — "only the ISO installer writes that bridge into /etc/network/interfaces". This design installs from the ISO, so the bridge is normally already there. The section verifies it instead of building it: exists, has an address, and actually carries the default route. The old step checked only the first of those, threw the result away, and its comment claimed it recorded the absence. When there is no bridge, it builds one, and only then is this the dangerous section the plan calls it. Backup, five-minute rollback timer, switch, look, cancel — and the timer is a systemd unit rather than a backgrounded sleep, because a background job dies with its session and the session is exactly what breaks when the switch goes wrong. It is set BEFORE the change; setting it after would mean setting it at the moment the connection is already gone. The three provider shapes differ in one line, and that line decides whether the machine is still there afterwards. A routed single address needs pointopoint, because its gateway is outside its own subnet and without it the kernel has no route to reach it. Which shape applies is decided by asking `ip route get` about the gateway rather than by doing subnet arithmetic here — same logic the kernel will apply later. The reachability check is named for what it is: inside-out. A true outside-in proof would need a counterpart, and before the tunnel there is none. It is enough because a bridge that takes the host off the network takes both directions with it, and what it does not cover is precisely what the timer covers. One deviation from the plan, written into it: the nftables lockdown does NOT belong here. SecureHostFirewall ran LAST in the old pipeline, deliberately, so the tunnel had already carried every earlier step before SSH-to-the-world was closed. Here it would sit BEFORE wireguard_joined and shut port 22 with no handshake yet proven — the ordering that makes a host permanently unreachable. It moves to Task 9, where it used to be. The self-rollback stays here and covers the network change; the firewall keeps its manual emergency release, because a firewall that reopens itself under failure is not a firewall. The datacenter firewall comes over verbatim, including why `enable 1` alone is a trap: the management ipset is seeded from the public subnet, so enabling it blindly drops the tunnel address that every later step arrives on. Three settings in order, then read back — a pvesh set returning 0 is not the same as the value being in cluster.fw, and a write that lost quorum is exactly the case where every customer VM's rules stay inert while onboarding reports success. Verified without hardware: all four files dash-clean, and the generated interfaces file is correct for subnet, routed (with pointopoint) and dhcp. Step 2 unticked — the self-rollback has to be triggered on purpose on real hardware, and the plan is right that without that it is an assertion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 20:06:18 +02:00
nexxo	f214713113	Point Proxmox at the sources this Debian actually has ... PVE arrives with the image now, so this section accepts it rather than installs it — but it needs every line of knowledge from InstallProxmoxVe.php to do that, just pointed the other way. The kernel is the claim, not the package. An installed proxmox-ve running on a Debian kernel is exactly the half state RebootIntoPveKernel exists because of, so uname is what gets checked. The codename table now verifies a pairing instead of creating one. An image whose PVE major does not match its Debian base was built wrong, and no amount of fixing package sources heals a PVE compiled against a different libc — that machine gets reinstalled with the right image. The other half of the table, codename to PVE major, is written down here because the ISO route needs it and the original file only had the suite side. Removing the subscription repositories is the part that everything after this depends on. A Proxmox image ships pve-enterprise, and without a subscription it fails every apt-get update — after which no package install in any later section succeeds. Both spellings are handled, because PVE 8 used one-line .list files and PVE 9 moved to deb822, and ceph is in the list because the same trap is set there a second time. The proof is a clean apt-get update at the end, which is the only thing that tells removal apart from overwriting. One thing changed after the first draft: when pveversion cannot be parsed, the pairing check used to be skipped in silence. That is the fake that R19 records as worse than no check at all, because it stops the next person from looking. It now says "Paarung UNGEPRÜFT" in the report and in the log. Verified without hardware: dash-clean; bookworm maps to PVE 8, trixie to PVE 9, forky is refused; the pveversion parser reads 9 and 8 out of both real formats and yields nothing for garbage; and against a copy of /etc/apt/sources.list.d holding all five subscription files plus an unrelated one, exactly the five go and the unrelated one stays. Step 2 unticked — there is no Proxmox here to accept. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 20:00:01 +02:00
nexxo	0f563545fd	Write the image and survive the first reboot ... The official Proxmox ISO, an answer file baked into it, and QEMU running it against the physical disks from the rescue system. Options were looked up rather than recalled: prepare-iso takes --fetch-from iso, --answer-file and --on-first-boot, and it names its own output, so the finished ISO is found by glob afterwards instead of being handed an --output flag I would have invented. validate-answer runs BEFORE the ISO is baked. An answer file with a typo drops the installer into interactive mode, where it then hangs invisibly inside QEMU until the hour runs out — the failure would arrive as a timeout and say nothing about the cause. The codename table comes over verbatim from InstallProxmoxVe.php, which the platform plan deletes. It is used here to install the assistant into the RESCUE system, so it keys off the rescue system's own codename. An unknown one aborts; not "trixie is newest, so trixie", because the next Debian will be unknown too and by then nobody is watching. The ISO is checksummed against the mirror's SHA256SUMS. Booting an operating system that could have become anything in transit is not a risk worth taking, and a truncated download is enough to cause it — malice is not required. The reboot carry-across is the gap this plan was amended for, and [first-boot] takes exactly one executable. So that one file carries everything: the script, its lib/, the progress file and the arguments, as a base64 archive in its own body. That also avoids importing the fresh ZFS pool from the rescue system to copy files into it. Two bugs found by running it rather than reading it. The staging copy put the work directory inside the state directory, so cp refused to copy a directory into itself — and had it not refused, the gigabyte ISO would have been base64'd into the hook that gets baked into that same ISO. The fix is not a better exclude: the two directories have opposite lifetimes, so they now live in different places, and a guard refuses loudly if anyone points them at each other again. Structure beats a rule someone has to remember. Verified without hardware: both files pass sh and dash. The generated answer file parses as TOML with the expected disk-list, zfs.raid, zfs.arc-max, first-boot and network.filter, and a 64-character root password. The codename table accepts bookworm and trixie and refuses bullseye and forky. The shim comes out at 26 KB with a 3 MB dummy ISO sitting in the work directory, proving it stayed out. Run in a fresh debian:13-slim container it unpacks to /opt/clupilot/bootstrap and /var/lib/clupilot, restores the progress file with its original timestamp, has 700 on the directory and 600 on the arguments that hold the WireGuard key, skips rescue_checked as already done, and fails cleanly on the missing qemu. Step 2 stays unticked: a container is not a rescue system, and nothing here has yet written to a disk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 19:57:29 +02:00
nexxo	25bcab345d	Refuse a machine that cannot do the job ... Four checks, all of them before a single byte is written, because this is the only moment the script still has the option of not overwriting a disk. The expensive one is "is this actually a rescue system". It demands POSITIVE evidence rather than the absence of a counter-argument: a rescue system runs from RAM, so its root is tmpfs, an overlay or a ramdisk, and an installed system has a real partition there. A running pveversion is refused outright — that is a hypervisor with customers on it. Mounted partitions of real disks are refused too, because at that point the script cannot tell an empty machine from someone's data. There is deliberately no flag to skip the check. Such a flag gets used exactly once, on the evening it should not have been. A rescue system this does not recognise belongs in the runbook, not in a bypass. The disk floor is written as arithmetic rather than a number, because the repo has no threshold to borrow — plans live in the database, not in config/provisioning.php. Template ~20 GB, Proxmox and its swap and backups ~20 GB, one smallest customer ~50 GB, rounded up to 100. It is explicitly not capacity planning; HostCapacity and reserve_pct do that after registration. This only turns away the machine the whole thing cannot fit on. Network and clock come from one plain-HTTP HEAD, deliberately without TLS — otherwise a wrong clock would be checking itself and would report a certificate error instead of the time. Where `date -d` is missing (busybox), the clock is NOT checked and the report says so: a check that silently waves things through is worse than none, because it stops the next person from looking. That is the same lesson R19 records about ->timezone(config('app.timezone')). Verified here, in both directions where a direction existed. On this VM (root on /dev/sda1, 80 GB, /dev/kvm present): refuses with three findings at once, exits 1, writes nothing. In a debian:13-slim container (root on overlay, no /dev/kvm): the overlay root is accepted as rescue-like and the missing /dev/kvm is caught, which is the abort case the plan names. Against a receiver serving a Date header two years off: 63074141 seconds of drift, refused, with the exact `date -u -s` line to fix it. Step 2 stays unticked — none of these is a rescue system on a dedicated server. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 19:36:15 +02:00
nexxo	af284d7d26	Give the bootstrap a spine and a way to report ... Argument parsing, the state directory, and the half of the script that knows the way back to CluPilot. No sections are hooked up yet; main lists the nine it will get, in the order from spec §7, so what is missing stays visible. report() writes locally first and only then tries to send, because before wireguard_joined there is no path at all — the lines queue up and flush_reports delivers them later WITH THEIR OWN TIMESTAMPS. That last part is the whole point: a twenty-minute install that arrives in one batch looks like one second in the console, and nobody can see which section was slow. Every network operation is explicitly caught. The script runs under `set -e`, and a failing send is the normal case for the first five sections, not an error — a report that aborts the run would be the diagnosis that kills the patient. What is sent had no agreed shape. The section keys were the only agreement between the two plans; the envelope around them was not written down anywhere, so it is written down here, at the top of report.sh, for the platform side to read once. There is deliberately no "running" state: the console derives "open" from the absence of a report (spec §7), and a third state would be a second truth about the same thing. The sent-marker is a line count in its own file rather than a flag rewritten into progress.jsonl. That keeps the log append-only, so a power cut mid-write costs at most a partial last line instead of a rewritten file — and after the reboot in Task 3 that file is the only thing that remembers anything. Verified here, without hardware: both syntax checks pass; two reports against an unreachable CluPilot queue up without aborting and leave progress.sent at 0; a receiver brought up afterwards gets both in one batch, carrying 17:28:55 and 17:28:57 rather than the 17:29:22 they arrived at; a message containing quotes, a backslash, a tab and a newline survives as valid JSON; and section_done counts a `done` as done and a `failed` as not. Step 2 of this task is NOT ticked — none of that is a rescue system, and the plan is right to insist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-30 19:30:19 +02:00
nexxo	a4626d2569	Stop root workers breaking every page, and let the panel be closed ... ── The 500 ───────────────────────────────────────────────────────────────── "touch(): Utime failed: Operation not permitted", from BladeCompiler, on every request for the affected view. Blade compiles a view and then touch()es the compiled file to the source's mtime. touch() with an explicit time needs ownership of the file. A compose service without `user:` runs as ROOT — and queue, scheduler and reverb had none — so the first worker to render a view wrote a root-owned compiled file that the web process, as www-data, could never refresh again. The queue is what renders mails, which is why this surfaced now. queue, reverb and scheduler run as www-data. queue-provisioning stays root and says why: it brings wg0 up and runs `wg set` for every peer change, which needs NET_ADMIN on the running process. It renders no mail. And update.sh normalises ownership at the END of a run as well as at the start. The first call heals what a previous run left; the second heals what this one made — `git checkout` rewrites the tree as the service account while the old build is still serving. ── The panel that would not go away ───────────────────────────────────────── The same 500 is why it kept coming back: every poll failed, the watcher treats a failed request as "still restarting" (which it normally is), and the overlay stayed up over a console nobody could then reach to find out why. A restart is seconds. After two minutes of nothing the panel now says so and offers a way out — for an operator only, behind the same flag as the step and the log, so a customer on the 503 page is not shown a button suggesting they can call the deployment off. A successful answer clears it again: one bad minute must not leave a "something is wrong" notice sitting there for the rest of the run. Not an always-present close button. The deployment does not stop because somebody dismissed a panel, and offering that at the wrong moment is a lie. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-29 15:43:54 +02:00
nexxo	f8f0de4892	Let an update install host packages, without handing out root ... Asked for directly: "später wenn ich etwas lokal erweitere will ich mich nicht auf alle adminseiten einloggen müssen und extra rsync installieren". An update runs as the service account, so anything root-owned on the host is out of its reach and turns into "log into every server once and paste this". The obvious fix — sudo on deploy/install-agent.sh — is not a grant at all: the service account owns the checkout and can rewrite the very script it would be allowed to run as root. So the privileged part is written OUT of the checkout by install-agent.sh, to /usr/local/sbin/clupilot-host-step, root:root 0755, from a quoted here-document. The service account cannot influence a byte of it. sudoers names one exact command line including its argument, so a step added to the helper later is not covered by a grant written before it existed, and the helper refuses anything not on its own list as well. update.sh uses it only when rsync is actually missing, after the restart and never fatally: the archive is collected hours later, and an update that died over a package would be the bigger problem. A helper older than the steps the updater wants is reported through the existing one-time-setup hint — silently doing nothing is worse than an error. Existing servers still need `sudo bash deploy/install-agent.sh` once, because that is the run that puts the helper there. After it, they do not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-29 10:27:56 +02:00
nexxo	15b536f393	Mint the archive collection key from the console instead of by hand ... Setting a NAS up to collect the invoice archive meant three machines and a dozen commands: generate a keypair there, carry the public half to the server, write it into authorized_keys with a restriction, install rsync, and get every path right on the first try. Every one of those steps was a place to be told "permission denied" with no clue which of the three was wrong. It was, and several times over. One button now. The panel asks, the host does it — because the host is where all of it lives: the home directory, ssh-keygen, rrsync, and the archive itself. The panel is www-data in a container and owns none of that, so it uses the mailbox it already uses for updates. The private half is shown exactly once, alongside the finished rsync command, and is never written to the database: it exists to be copied into a NAS, and storing it "for convenience" would put a working credential in every backup of that database. rrsync does the restricting, not a pinned rsync option string. That string differs between rsync versions and fails silently — a refusal with no reason given, which is the shape of the afternoon this replaces. If rrsync is missing the agent refuses rather than issuing an unrestricted key while the panel says it is restricted. rsync now comes with install-agent.sh, which already runs as root once per machine. It has to be on the HOST: a NAS connects by ssh and sshd starts `rsync --server` here, so without it the pull fails with "command not found" from a NAS whose own setup is perfectly correct. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-29 10:08:32 +02:00
nexxo	33e5b15099	Restart the workers automatically after saving .env, instead of handing the operator back to the shell ... Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-29 02:26:34 +02:00
nexxo	4d4d1dd5d5	Stop the update agent dying on its own tag arithmetic ... Since 1.2.0 the agent has ended with a non-zero status on every tick, before writing anything. The console showed a check that never finished and a last-checked time frozen at the moment of the deployment, and the update button could not be reached at all — the one mechanism that would have delivered the fix. Two instances of the same trap, both mine, both under `set -Eeuo pipefail`: The tag count was `release_version_gt … && echo`. The LAST iteration is almost always a tag that is not newer, so the loop ended non-zero, pipefail carried it out of the pipeline, the command substitution inherited it, and set -e ended the agent. `if` rather than `&&` is the whole fix: an if whose condition is false still returns 0. The newest tag came from `git tag … | head -1`. head exits after one line, git takes SIGPIPE, pipefail does the rest. install-agent.sh has carried a written warning about exactly this since it was written; I read it and wrote it anyway. Both now live in deploy/lib/release.sh, because the reason neither was caught is that no PHP test can reach a bash pipeline. tests/Feature/ReleaseComparisonTest.php runs the real functions in a real throwaway repository under the agent's own set -Eeuo pipefail, and asserts the EXIT CODE as well as the answer — an answer nobody lives to read is not an answer. The SIGPIPE case needs four hundred tags to reproduce, because a short list finishes writing before head leaves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-29 00:00:17 +02:00
nexxo	6998a22527	Repair ownership by looking inside, not at the door ... The ownership repair added in 1.1.1 sampled the owner of each top-level directory and skipped the recursion when it already matched. node_modules was owned by www-data; node_modules/.vite-temp, left behind by an earlier root build, was not. So the repair walked past it, `npm run build` failed with EACCES, and the deployment stopped in maintenance mode — on the release whose whole point was that this could not happen. A directory's owner says nothing about its contents. It now walks each tree once with find and changes only the entries that are wrong, which is also cheaper than the detect-then-chown-everything it replaces. Verified against the shape that actually failed: a root-owned file inside a www-data-owned node_modules, repaired. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-28 23:53:25 +02:00
nexxo	00b9107ee3	Deploy as the application's user, and take back what root already took ... The 500 on the VPN config download was never in the VPN code. storage/logs/ laravel.log was owned by root, mode 644, since 27.07 04:16 — the moment a failed `artisan optimize` wrote its error there during a deployment. From then on the application could not append to its own log: Monolog threw on every attempt, and a throw while logging is a 500 on any page that logs, with nothing written down to say why. The error page said "Er wurde protokolliert". It was not. It surfaced on the config download because that is one of the few pages writing a log line on its way through — on success and on a wrong password alike. Pages that log nothing were unaffected, which is exactly why it looked like a VPN fault for days. `docker compose exec` is root unless told otherwise, and every deployment script relied on that default — including the agent's allowlist sync, which runs every minute. docker/entrypoint.sh had it right all along and drops to www-data for precisely these commands. Now so does everything else, help text included: telling an operator to run artisan as root is how the file ends up owned by root in the first place. update.sh also repairs what an earlier run left behind, before its first unprivileged step rather than after — a root-owned log file is not self-healing, the page that trips over it is nowhere near the deployment that caused it, and with in_app unprivileged a root-owned vendor/ would break the next composer step outright. tests/Feature/DeploymentRunsAsTheAppUserTest.php holds the line. It was checked against the previous commit and finds all nine places that were wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-28 22:52:09 +02:00
nexxo	21b7aaca42	Update only to a released version, never to whatever landed on main ... Asked for directly: a commit on main must change nothing on a server, and only a tag whose version is higher than the installed one may be installed. The agent asked two different questions before this. A pinned server compared tags; everything else counted commits on its branch — so a console following main offered an update for every push, verified or not, and "Jetzt aktualisieren" meant "take whatever is on the branch right now". Landing on main and being released are different events, and only the second is a decision somebody made. It is one question now, regardless of what the checkout is attached to: is there a tag whose version is higher than the deployed one? Compared with sort -V against the version in the deployment manifest — not the VERSION file, which an update moves before it migrates, so a run that failed in between would leave the checkout claiming a version it never finished installing. A run always targets that tag; the branch path is gone. A request that arrives with nothing to install is answered rather than obeyed, before anything announces a run. The console follows: the button is disabled with nothing released, says so in a line rather than leaving a grey button to be guessed at, names the version it would install rather than a count, and refuses the action server-side as well — a Livewire action is a public endpoint, and one place a rule may not live is only in the disabled attribute of a button. VERSION becomes 1.1.0, to be tagged as the first release this mechanism can see. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-28 22:32:44 +02:00
nexxo	07634c8a1d	Answer a check at once, start an update at once, and count down to neither ... Checking for updates was a request written into the same mailbox as a real update, collected by the same agent on the same systemd timer — so asking what was new was answered on the next tick, and the console, having nothing else to show, counted down to it. An operator who asked a question was told an update would start in 3:44. A systemd path unit now wakes the agent the moment the request file is written, for a check and for a run alike. PathChanged, not PathExists: PathExists re-arms as soon as the service goes inactive, so a request the agent could not consume — it holds a lock for the length of an update — would restart it in a tight loop. The timer stays as the fallback, and a drop-in disables systemd's start rate limit, because two triggers on one oneshot service can otherwise wedge the unit and leave neither of them able to run. The countdown is gone from both the settings card and the maintenance overlay. It could not be made honest: its target was recomputed on every poll, and where the interval the agent reports is shorter than the timer actually installed on that host it had already gone by, so the fallback moved it forward again each time. It ran backwards four seconds and snapped back to a full minute, forever — reported exactly that way. A queued run says it is starting; a queued check says nothing beyond its badge, because a check starts nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-28 22:25:07 +02:00
nexxo	f05547921d	Separate checking for updates from applying them, and show live progress	2026-07-28 20:23:06 +02:00
nexxo	bd779f00e5	Show times on the operator's clock, keep storing them in UTC ... The console announced an update for "spätestens um 15:21" while the clock on the wall said 17:21. Not a formatting slip in one string: fourteen views and four components printed database values straight out, so every absolute time in the product was two hours out all summer and one hour out all winter. Two of those views looked as though they had handled it. They called ->timezone(config('app.timezone')) which reads like "convert to local time" and, app.timezone being the STORAGE zone that must stay UTC, converts to UTC — a no-op wearing the costume of a fix, which is worse than no call at all because it stops the next person looking. That exact string is now banned by test. Worse than the labels were two FORMS. Maintenance windows and plan versions filled their datetime-local fields from UTC and parsed what came back as UTC. A datetime-local field carries no offset — it is the digits a person reads off their own clock — so an operator typing 21:00 scheduled a window for 23:00 their time, and nothing anywhere said so. App\Support\LocalTime now holds toField() and fromField() side by side, because getting one end right is not half a fix, it is a fresh bug. The mechanism is one Carbon macro, ->local(). It copies before converting: Illuminate\Support\Carbon is mutable, so without that, rendering a timestamp would rewrite the model attribute as a side effect and anything comparing or saving it afterwards would be an hour or two out. Both Carbon classes get the same body — Carbon keeps macros in one global table, so the second registration replaces the first for every Carbon class, and two different bodies meant the immutable version, safe for itself, silently became the mutable one's implementation too. My own test caught that. The existing tests had not caught any of this because they built their expected values with the same wrong call the views used — a test that recomputes the implementation asserts nothing. They now assert the wall clock, and the queued update additionally asserts that the UTC time is NOT shown. Recorded as R19 in CLAUDE.md and enforced by tests/Feature/DisplayTimezoneTest: no Blade or Livewire component may format an absolute time without ->local(), the UTC no-op is banned, storage stays UTC, both sides of the DST boundary are checked, ->local() must not mutate, and a form field must round-trip to the same instant. APP_DISPLAY_TIMEZONE, default Europe/Vienna. 615 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 17:32:21 +02:00
nexxo	b85c12e8b9	Give the readiness probe long enough for a restart to finish ... The gateway is restarted twice in a deployment — once by `up -d`, then again after the hub whose network namespace it lives in — and `docker compose exec` into a container that is still coming up fails outright rather than waiting. Twenty seconds did not cover that, so the first run of the new probe reported a gateway that answers as down, and the resolver stayed out of client configs for one more deployment. The health site also now binds the hub address instead of only matching on it. Nothing was published either way, but the file should do what its comment says. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 14:46:47 +02:00
nexxo	3699c7cdb9	Ask the tunnel gateway where it actually listens ... The console has been unreachable over the VPN, and the cause was the readiness probe rather than the tunnel. The gateway binds to the WireGuard hub address alone; the probe asked 127.0.0.1, where nothing has ever listened. It could only fail, so VPN_READY stayed false, and on that basis the application withheld the `DNS = 10.66.0.1` line from every client config it issued. A phone then built the tunnel, asked its normal resolver for the console hostname, got the public address and had its connection closed — indistinguishable from "this site does not exist", which is exactly what it looked like. Probing over TLS on the bare address would not have worked either: the site matches on the console's hostname, so a request without SNI is offered no certificate. The gateway now answers a plain-HTTP health port on the hub address, which removes TLS, SNI and name resolution from the question and answers only what is being asked — is this gateway listening, in this network namespace, right now. Caddy refuses to start when the certificate is unreadable, so a health port that answers still proves the whole file loaded. Also here, all found while looking: - Icons pushed their label onto a second line and rendered a size larger than asked for. Tailwind's preflight makes an svg display:block, and `.size-4` and `.size-5` have equal specificity, so stylesheet order decided — and it emits size-4 first. Every icon written as 16px was silently 20px. Recorded as R18. - Four error pages printed `errors.404.hint` in place of a sentence: the lang files give those codes a null hint and Laravel returns the key for a null line. - The Developer role had no label in either language, so the dropdown showed `admin_settings.role_developer`. - The secrets area held one key, which is not worth a password gate. It now carries the credentials that actually stop the business when they expire — DNS, monitoring, SMTP — and the test button appears only where a checker exists, instead of reporting on Stripe whatever was being looked at. - The update button said nothing about when a queued run would start or where a running one had got to; both are now shown, and a failure names its step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 14:43:23 +02:00
Claude	2bb1d7cc3b	Make the private hostnames look like nothing is there, and close the way past the proxy ... An empty 404 is itself information: it says something terminates TLS here and chose not to answer. The console and the websocket endpoint now close the connection instead, which is what a hostname that serves nothing looks like. The websocket endpoint answers a genuine upgrade — verified live, 101 — and nothing else; a browser opening it gets a closed connection. Reviewing that turned up two holes that mattered more than the thing being reviewed. The compose defaults published the application and Reverb on every interface. Docker publishes ports ahead of UFW, so those backends were reachable from the internet with the firewall closed — and reaching a backend directly skips the proxy, and with it every hostname and address rule keeping the console private. The defaults are loopback now, and an update rebinds an existing installation that is behind a proxy actually running and holding 443. A development box without one is left alone, because taking its port away would look like the machine had broken. And the console's own "open" switch was emitting 0.0.0.0/0 into the proxy's allowlist. One click would have put the console on the public internet, which is the exact state the owner's rule exists to prevent. Switching it off relaxes the application's check; the proxy keeps its list, always. The reference proxy config carries the arrangement: QUIC early data off, since an address-based decision taken on 0-RTT can answer 425 and some browsers do not retry — an intermittent lockout from the console is the worst kind — and explicit handling for plain HTTP, whose automatic redirect otherwise announces both hidden names to anyone who asks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 11:26:48 +02:00
Claude	c4649571df	Wait for the tunnel interface, and prove the gateway answers ... Two things reported success while the tunnel console was unreachable. The services were restarted the moment the hub container counted as started, but the hub brings wg0 up as part of its start command — so they were binding an address that did not exist yet. They now wait for the interface. And readiness was a container-state check. A container can be up and running while the process inside it listens in a network namespace that was torn down underneath it: nothing errors, the address simply refuses connections, and "running" reports everything fine. It asks the gateway now, and only a real answer counts — because the consequence of getting this wrong is a client config naming a resolver that is not there, which takes the device's whole name resolution with it for as long as the tunnel is up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 10:24:29 +02:00
Claude	d2d79b4f5f	Restart the tunnel services after the hub they live inside ... Both VPN services share the provisioning container's network namespace, and a process holds the namespace it started in. Restarting the hub — which every deploy does, to pick up new code — therefore leaves them listening inside a namespace that no longer exists. Nothing errors. The containers stay up, Caddy's log still reports it is serving on the tunnel address, and connections to that address are simply refused. It looks precisely like a gateway that never worked, which is how it presented on the live server after the first successful start. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 10:22:19 +02:00
Claude	457eeeeaef	Serve the console inside the tunnel, without publishing the internal network ... The owner's phone shows a blank page on the VPN. That page is the reverse proxy refusing the request, and it refuses because the phone is not on the tunnel: the client config routes only the management subnet, so a request to the console's public hostname leaves over the mobile network and arrives from a carrier address. It is also why the peer reads "last contact: never" — WireGuard only handshakes when it has traffic to send, and nothing is ever routed in. The obvious fixes are both wrong. Adding the server's public address to AllowedIPs routes the WireGuard endpoint into the tunnel it is trying to establish. A public DNS record pointing at 10.66.0.1 publishes the internal subnet to anyone enumerating the domain — the owner raised that himself, and he was right. So the console is served INSIDE the tunnel instead. A small resolver and a small gateway share the hub container's network namespace, which is where wg0 lives, and answer on 10.66.0.1 directly. A client therefore needs no route beyond the subnet it already has, the host's docker bridge is never exposed, and the request reaches the application with its real 10.66.0.x source — which is what the console's own allowlist checks. The gateway reuses the certificate the public Caddy already renews: a certificate is bound to the name, not to the address serving it, so nothing new is issued and nothing internal reaches a public zone. Most of this commit is the arithmetic of not lying about it. Review found fourteen ways the arrangement could report itself working while it was not: enabling the compose profile after the services were started rather than before; starting a gateway with empty certificate paths, which can only crash; treating "container created" as "container running"; carrying a stale readiness marker through a failed restart; reusing the previous hostname's certificate after a rename; leaving the profile enabled when the hostname was cleared, in a script that exits early precisely when nothing else would fix it; reading the renewal timestamp as a user who cannot traverse Caddy's storage; and a `find | head` that aborts the whole installer under pipefail the moment two issuers hold a certificate for the same name. One of them was mine and worth naming: to let the container read the certificate, I had made the TLS private key world-readable. On a multi-user host that hands the console's identity to every local account. The client is told about the resolver only when both services are confirmed running, because a config naming a resolver that does not exist takes the device's entire name resolution with it for as long as the tunnel is up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 10:14:33 +02:00
Claude	6082164f8b	Give the status page its own hostname, and name the right one for Stripe ... STATUS_HOST binds /status to a single hostname. Every other host redirects there rather than 404ing: a status page is the one address people keep in a bookmark and reach for when something is already wrong, so breaking old links to prove a point about separation would be exactly backwards. Unset, nothing changes. The /legal/status redirect goes through route() instead of a literal path. A relative redirect lands on whatever host the visitor is already on — which, once the page has a hostname of its own, is precisely where it no longer answers. The installer told the operator to point Stripe's webhook at the customer portal. Stripe posts server-to-server and never sees the portal; the endpoint is the api hostname, which the installer never even asked for. It asks now, and prints the address that actually receives the events. Getting this wrong does not fail loudly — payments simply stop being recorded. It also writes ADMIN_HOST_EXCLUSIVE=false on a fresh install. Switching the console onto its own hostname before the DNS for it exists makes the console unreachable under any other name, and that is not a thing to have on by default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 08:08:57 +02:00
Claude	bc8bbc56a5	Make the console's access list reach the proxy, price in euros, and answer errors ... The console was unreachable, and not for the reason it looked like. Two causes, both mine. A failed `artisan optimize` — the one that hit duplicate route names — left a broken route cache behind, so every request to the console host answered 500. Rebuilt. Underneath that: the reverse proxy has its own console allowlist, hard-coded, and it runs BEFORE the application. Everything added on the console's own access page was therefore ineffective, and when the owner's address changed they were turned away by the proxy before reaching the page that would have fixed it. The proxy now imports a fragment generated from the same list the console manages, regenerated by the host agent and reloaded when it changes. Getting that safe took most of the review. It refuses to rewrite an ambiguous Caddyfile rather than replacing some other site's matcher; it never falls back to a loopback-only list when the application cannot be reached, because that list validates cleanly and locks out every remote operator; it retries a reload that failed instead of assuming it worked; and an installation that upgrades without rerunning the installer is told, because otherwise the whole mechanism is invisibly absent. Prices are entered in euros. The form asked for cents, so €799 was typed as 79900 and one slipped digit was a factor of ten on an invoice. Conversion happens in one place, on the string rather than through a float — (float) '79.90' × 100 is 7989.999… and casting truncates to 7989, one cent short on exactly the prices people charge — and it refuses an amount the column cannot hold instead of failing at the database. Every error code has a page now, in the site's own language and typeface, self-contained so it still renders when the asset manifest is the thing that broke. There was only a bare white 404. The VPN list stops being a seven-column table nobody could fit: names broke across two lines and so did the headings. These are attributes of one access, not quantities compared down a column, so each access is a row — identity on one line, measurements in mono on the next. The status page says what it measures rather than what it promises. "New orders are delivered without failures" read as a marketing claim on a page whose only job is to be believed, and said nothing about the last 24 hours. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 06:51:05 +02:00
Claude	276f926d57	Add the update button, and give it a host-side agent that can actually update ... The panel had no way to update the installation, and could not have one on its own: it runs as www-data inside a container, and the update restarts that container. A button that shelled out would kill its own web server mid-response, and would need the application to hold host-level credentials — a far worse thing to own than an out-of-date checkout. So the button is a mailbox. The panel writes a request into the checkout; a systemd timer on the host, running as the service account, consumes it, runs deploy/update.sh and writes back what happened. The same timer answers the question the panel cannot answer alone — is there anything to update — which needs a git fetch, and therefore credentials the application does not have. The parts that were wrong before review, all of which would have shipped as a button that looks fine and does nothing: - The agent was only installed by install.sh, which the installed base never runs again. Every existing server would have shown the button permanently disabled. It has its own root entry point now, install.sh delegates to it, and update.sh says so when the unit is missing. - On a release-pinned server the checkout is detached, so it followed "origin/HEAD" and then ran an update that deliberately stays on the pinned tag: updates advertised, nothing applied. It now compares TAGS in release mode and passes the target release through. - On a branch other than main it advertised that branch's commits and then deployed main's, because update.sh defaults to main. - A request written while the agent was down was executed whenever the agent next started, days later. - A single status file meant the routine five-minute check overwrote a failed update with "idle" before anyone saw it. - An agent that had been stopped left the button enabled forever, because a status file written once counted as an agent. - The agent's error messages were German strings rendered into an English interface; it reports codes now and the panel translates them. Also: the add-address button in the console-access panel was as tall as the field plus its hint, because the hint was a sibling inside the flex row. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 05:33:17 +02:00
nexxo	875fd11982	fix(deploy): an answer file with a space in a value killed the install ... The file is sourced, so ADMIN_NAME=Boban Blaskovic is read as a command and dies with "Blaskovic: command not found" — naming neither the file nor the variable — and set -e ends the install there. A display name with a space is entirely ordinary; the example just happened to use "Administrator", one word, so nobody hit it. Found while installing on the live server. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-26 18:49:58 +02:00
nexxo	35307e64cc	fix(deploy): the installer could never finish, twice over ... Found by running it on a fresh Debian 12 with systemd, no docker, no git and no users — which is the first time it has ever been run anywhere. `optional_env` ended in `[[ -n "$2" ]] && set_env …`. With an empty value the function returns 1, and under `set -e` that ends the install, silently, with a half-written root-owned .env. Absent is the NORMAL case for those variables — they are the optional ones — so this killed every installation that did not happen to supply a Hetzner token. An `if` returns 0 and does not. Then it waited for `php -v` before migrating. PHP answers in seconds; the entrypoint is still running `composer install` on a fresh checkout, which takes minutes. So `artisan migrate` ran against a checkout with no vendor/ and died on a missing autoload.php. It now waits for vendor/autoload.php, exactly as the dependent containers already do, and says where to look if it never appears. After both: exit 0, 59 migrations applied, the plan catalogue seeded and consistent, the Owner account created with its role, and the portal answering 200 while the console redirects to login and the unconfigured Stripe webhook fails closed with 400. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-26 17:42:26 +02:00
nexxo	e9240c1324	feat(deploy): releases you can pin to, and a version that tells the truth ... VERSION at the repository root is the release number and the only place it is written down — a file rather than `git describe`, because a source archive, a shallow CI checkout and a container built without .git have no tags to describe. Cutting a release is editing that file and merging it; CI creates the annotated v<VERSION> tag once green, only on the commit that raised the number, and never moves it. Servers are pinned to these. Two modes, and the difference is the point. RELEASE=v1.0.0 pins a server to an immutable tag, checked out detached: it moves when someone decides it moves. BRANCH=main follows the edge, where CI tags a commit only after it is pushed. The mode is remembered, so re-running the updater on a pinned box does not quietly walk it back onto main. Going backwards is refused, by commit ancestry rather than by comparing version strings — a tag can be cut from anywhere, and only ancestry says whether history is going back. The schema has already moved forward by then, and the migrations needed to reverse it are not in the older checkout at all. What the console reports comes from a manifest written atomically after every step succeeded, never from live git: git says what the files are, not whether the deployment came up. So a failed update keeps reporting the version that is actually serving — including its version number, not the newer one the checkout has already moved to. A deployment pinned to the tag reads "1.0.0 (abc1234)"; anything else reads "1.0.0-dev (abc1234) · main", because every commit after the tag still carries VERSION=1.0.0 and is not that release. Reporting it as one is how a bug gets filed against the wrong code. Codex reviewed the design before it was written and the result six times after. Its findings, all real: the version had to come from the manifest too, not just the commit; branch names need JSON escaping or the manifest silently vanishes; a `case` glob does not anchor and accepted 1x.2y.3garbage; pinning to the commit a server already sits on skipped recording the mode, and then skipped detaching; a manifest that failed to write would never be repaired; an unparseable timestamp would 500 every admin page; and an empty tag list read as a connectivity failure. 466 tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-26 15:21:38 +02:00
nexxo	c119e8d96e	fix(deploy): say which branches exist instead of failing inside git ... The installer clones BRANCH, which defaults to main — and the repository has no main. Git's own "Remote branch not found" reads like a bad token or an unreachable server, and the branch is the one thing here that is routinely wrong. It now asks first and, when the branch is missing, prints the ones that do exist and how to pass one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-26 14:48:49 +02:00
nexxo	4f3e11ae5e	docs(deploy): the installer's Stripe steps match what phase 5 actually needs ... It still named two webhook events; the application handles six, and the four new ones are the billing cycle itself — renewals, failed payments, status changes and endings. It also said nothing about stripe:sync-catalogue, without which Stripe has no prices and nothing can be sold at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-26 14:46:25 +02:00
nexxo	8117630a65	feat(ops): update page, automatic 419 recovery, CI workflow ... Three things that bit us on every deploy: - artisan down now renders a branded page that says an update is running and reloads itself, instead of Laravel's bare 503. - A page left open across a deploy carries a Livewire snapshot and CSRF token the new code rejects — the user got "419 Page Expired" and a dead interface. A 419 from a /livewire/ request now reloads the page; the session is still valid, so that is all it takes. Hooked at the fetch layer rather than through Livewire's request hook, whose failure callback is not invoked for this case in the installed version — verified against a real 419 in the browser, not a simulated one. - Vite no longer empties its output directory: wiping it is what left an open page without its stylesheet mid-deploy, which is the "design is completely broken" symptom. update.sh also rebuilds the caches with optimize:clear + optimize instead of leaving them half-warm. Plus CI: .gitea/workflows/tests.yml (Pest + asset build, and a tested- tag only on a green main) and an opt-in act_runner service under the ci profile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 00:58:27 +02:00
nexxo	ac86efd9c3	feat(deploy): unattended install from a single answers file ... --env-file makes the whole run non-interactive, which is what a bare server needs: install git, clone, run, done. The file also carries the operational secrets (Hetzner DNS, Stripe, SMTP, the Proxmox key path); those are optional, so a first install can happen before the Stripe account exists and the features stay dark until the values are filled in. The installer warns when the file is readable by more than its owner, and clupilot.env is gitignored — it holds every secret an installation will need. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 00:40:33 +02:00
nexxo	2415dde288	fix(deploy,traffic): clear caches before restarts, hide the token, widen the lock ... - update.sh restarted the long-lived services before clearing the caches, so they booted from the old cached config and kept it for the life of the process — the release's configuration never reached the workers. - install.sh put the Gitea token in the clone URL, which is visible in the process list to every local user while the clone runs. It goes through a short-lived askpass helper now. - The collector's uniqueness lock expired at exactly the collection interval, so a delayed queue could run two collectors on the same baseline and count the same delta twice — throttling a customer for traffic they never used. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 23:42:12 +02:00
nexxo	25c4c5df5a	fix(deploy): restart the worker after the hub key, diff against what was deployed ... - The provisioning worker boots before install.sh writes CLUPILOT_WG_HUB_PUBKEY and holds the empty value for the life of its process, so every host onboarded afterwards would have got an empty PublicKey in its wg0.conf. - A failed update left the checkout at the target, so the rerun compared a commit with itself and skipped exactly the dependency and image steps that had not finished. The comparison base is now the last commit actually deployed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 23:38:32 +02:00
nexxo	4681b135db	feat(deploy): run under a dedicated account instead of root ... The installer still needs root for apt and the firewall, but it now creates a clupilot service account that owns the checkout and runs Docker, maps the container user to it (HOST_UID/GID), and update.sh refuses to run as root — root-owned files in the checkout are files the application cannot write. The account has no password login: docker group membership is root-equivalent on the host, so it is reachable only via sudo -u or an SSH key. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 23:25:59 +02:00
nexxo	0219e3c987	fix(deploy): recreate the container, keep the root password secret, forget failures ... - update.sh recreated the app container only at the very end, so an update that changes the PHP runtime would have installed dependencies and migrated inside the old one. - install.sh randomised DB_PASSWORD but left DB_ROOT_PASSWORD at the value committed in .env.example — the same root credential on every installation, reachable from any container on the compose network. - Settings cached the fallback after a database failure, so one blip could leave a hidden site publicly visible until someone cleared the cache. Only a successful read is cached now, proven by a test that drops the table and brings it back. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 23:25:06 +02:00
nexxo	773ef4bd5f	fix(deploy): migrate behind maintenance mode, resume a failed update, install deps ... Codex was right on all three, and the first was a design error on my part: the checkout is bind-mounted into every container, so 'git merge' swaps the running code instantly — 'migrations before traffic' was not achievable by ordering. The updater now goes down → merge → dependencies → migrate → assets → restart → up, trading a short deliberate outage for never serving code whose schema does not exist yet. If a step fails the site STAYS down: coming back up with new code on an old schema is worse than staying dark until someone looks. It also records the last successfully deployed commit. A run that died halfway left the checkout ahead, so the next run said 'already up to date' and skipped the rest forever. And it installs dependencies when a lockfile moved: vendor/ and node_modules/ live in the bind mount and shadow the image, so rebuilding the image never updated them — migrations could run against stale packages. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 23:15:15 +02:00
nexxo	834abcec40	feat(deploy): installer and updater for a fresh server ... install.sh sets up a bare Debian/Ubuntu server end to end: Docker, git, WireGuard, clone, generated secrets, stack, migrations, hub keypair, the Owner account and a closed firewall. Re-runnable: it keeps an existing .env and never regenerates the hub key, which would disconnect every onboarded host. update.sh pulls and applies. Migrations run before the new containers take traffic, the image is rebuilt only when its definition changed, and the queue workers are restarted — they are long-running processes that otherwise keep the old classes in memory, which cost us an hour during development. clupilot:create-admin creates or promotes an Owner, so re-running the installer fixes a lost role instead of failing on a taken address. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 23:13:24 +02:00