10 Commits (db125781ed6894acfc929a049cd792e0b8b916c2)
| Author | SHA1 | Message | Date |
|---|---|---|---|
nexxo
|
db125781ed |
Release v1.3.88 — die Vorlage baut sich selbst
tests / pest (push) Failing after 9m17s
Details
tests / assets (push) Successful in 22s
Details
tests / release (push) Has been skipped
Details
Der letzte Handgriff in der Host-Übernahme fällt weg. VerifyVmTemplate meldete
bisher nur, dass eine Vorlage fehlt, weil niemand entschieden hatte, was in die
goldene Vorlage gehört. Entschieden ist es längst und steht in
deploy/bootstrap/lib/template.sh — der neue Schritt BuildVmTemplate lädt genau
diese Datei auf den Host und führt sie dort aus, statt ihre Prüfungen ein
zweites Mal in PHP zu haben.
Er läuft abgekoppelt und wird abgefragt: Abbild laden und drei
virt-customize-Läufe brauchen zehn bis zwanzig Minuten, ein einzelner
SSH-Aufruf liefe gegen den Befehlszeitablauf von 2000 s. "Läuft noch" heißt
dabei, dass der Prozess lebt (kill -0 gegen die hinterlegte PID) — in der
Statusdatei steht "running" auch dann noch, wenn niemand mehr da ist, der sie
ändert.
Fünf Fehler, die dabei aufgefallen sind und Geld gekostet hätten:
- qm importdisk hängte die Platte unter ${storage}:vm-9000-disk-0 ein. Der Name
gilt nur bei Block-Ablagen; auf einer Verzeichnis-Ablage heißt sie
local:9000/vm-9000-disk-0.qcow2 — also genau auf dem per Debian aufgesetzten
Proxmox, um das es hier geht. Jetzt qm set --import-from, und Proxmox
benennt selbst.
- growpart war nie installiert. GrowGuestFilesystem ruft es auf, und es lief
bisher, weil Debians Cloud-Abbild es zufällig mitbringt. Fiele es heraus,
läge jedes gekaufte Kontingent über einem Dateisystem, das nie gewachsen ist.
Jetzt ausdrücklich eingebaut und als vierte Falle nachgewiesen.
- local nimmt ab Werk keine Platten an. Ohne das stirbt nicht nur der Bau,
RegisterCapacity meldet danach Kapazität 0: ein Host, der fertig aussieht und
nie einen Kunden tragen kann. ensure_image_storage greift nur ein, wenn keine
Ablage Platten annimmt, hängt images an die vorhandene Liste an statt sie zu
ersetzen, und schreibt über pvesm set statt in die pmxcfs-Datei.
- Ein abgebrochener Download blieb unter dem Zielnamen liegen und wäre beim
nächsten Lauf ungeprüft weiterbenutzt worden. Jetzt .part, umbenannt erst
nach geprüfter Summe.
- VerifyVmTemplate und VmTemplateCheck fragten nur, ob VMID 9000 existiert. Ein
abgebrochener Bau hinterlässt eine gewöhnliche VM mit dieser Nummer, und
beide sagten dazu "passt" — der Fehler kam beim ersten bezahlten Klon zurück.
Jetzt template: 1.
isTemplate() stellt zwei Anfragen, weil die falsche Antwort hier etwas
zerstört: false heißt "Vorlage fehlt", und der Bau fängt mit qm destroy --purge
an. Proxmox beantwortet die Konfiguration einer nicht vorhandenen VM mit 500 —
demselben Code wie einen Knoten in Not. Die VM-Liste klärt deshalb die
Abwesenheit, alles darunter wirft und landet im Wiederholungs-Zweig.
Aufgeben beendet erst die Prozessgruppe, dann räumt es auf, und gebaut wird nur
die Fehlliste: create_proxmox_template räumt eine VMID weg, bevor es sie
anlegt, also hätte "alles Verlangte" eine gesunde zweite Vorlage auf dem Weg
zerstört.
Geprüft: 2267 Tests grün, Pint sauber, sh -n über alle drei Shell-Dateien, die
storage.cfg-Auswertung gegen eine echte Beispieldatei durchgespielt, und jeder
Befehl, den der Schritt absetzt, geht durch sh -n — keine andere Prüfung führt
diese Shell je aus. Drei Codex-Runden (R15), alle Befunde behoben.
Nicht geprüft: nichts davon lief je gegen echte Hardware. Die erste Übernahme
auf einem Proxmox-Host ist die Abnahme.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
|
nexxo
|
ba4996f316 |
Keep the platform on .com and the customers on .cloud
The owner caught me putting the archive on clupilot.cloud. That is the customer
zone, and looking into it turned up the same confusion already sitting in the
code — in the one place that matters most.
RegisterHostDns says "fsn-01.node.clupilot.com" in its own docblock. The
validation comment in Datacenters says it. ServicesTest writes it out verbatim.
The step itself built the name from config('provisioning.dns.zone') — the
CUSTOMER zone — so on this installation a host was actually called
fsn-01.node.clupilot.cloud. Three places asserting one thing and the code doing
another.
OfficialDomains explains why that matters and is worth not weakening: two
registrable domains by design, the company's for site, portal and console, the
instance zone for customer workloads. A Nextcloud is third-party software that
strangers sign into, and on the same registrable domain as the portal it shares
cookie scope with it. A host name in that zone does not break the separation,
but it puts it in question, and the next slip is more expensive.
So there is now a platform_zone, derived from APP_URL when unset, and
RegisterHostDns uses it.
My own archiveUrl was broken for a second reason. It fell back to url() when
APP_HOST is empty — and APP_HOST is empty on most installations, because empty
means "the portal answers on any hostname" and that is the default. Called from
the console, url() would have produced the CONSOLE hostname, and the line would
have 404'd on a machine that is not allowed to reach the admin area at all. It
takes the host from APP_URL now.
The script no longer guesses its own name. It used reverse DNS, then the tunnel
address, then a hard-coded clupilot.net — a third domain that appears nowhere
else in this project and was simply invented. PrepareBaseSystem has the same
invention. A guessed name does not stay guessed: it ends up in /etc/hostname, in
/etc/hosts, in every log line and in every certificate request the machine ever
makes. CluPilot knows the name because it just assigned it, so it passes --fqdn
and the script refuses without it. The value now also survives the reboot in the
arguments file, which it would not have.
The ACME contact moved to .com for the same reason it was wrong: the operator
does not live in the customer zone.
Open, and NOT decided here: the owner also wants the host to get a public DNS
record and a certificate on the .com name. RegisterHostDns deliberately writes
host names only into the tunnel's dnsmasq, and says why — publishing them hands
every scanner the internal subnet and roughly how many hosts sit behind it.
Nothing in the current design needs a public certificate for a host's own name;
Traefik serves customer domains, not this one. Reversing that is a security
decision and belongs to the owner, not to this commit.
1992 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
|
nexxo
|
f45f290f00 |
Hand over the token and let the console take it from here
The last section carries three things, and none of them could sit anywhere else: the registration and key swap, Traefik's token plus the proof that a route table actually arrives, and the nftables lockdown that ran last in the old pipeline for a reason. `|| true` is gone from the pveum calls, and not by deleting it. `role add || true` applied the privilege list only on the run that first created the role, so a privilege added later reached new hosts and no existing one — which is exactly how Sys.Modify was missing everywhere, surfacing much later and somewhere else as a 403 on POST /cluster/backup that failed a paying customer at register_backup. So: add, and on failure modify, as two separate commands so a failure can be attributed. For the user and the ACL the tolerated `|| true` is replaced by looking: try, and if it fails, check whether the desired state holds anyway. A `|| true` hides "already there" and "went wrong" equally well. The privilege list is copied verbatim from config/provisioning.php and checked against it — 18 privileges, byte-identical, Sys.Modify included. It is granted on / because both endpoints that need it check / and nothing narrower satisfies them. The key swap follows the four steps this plan was amended to require. Register first and hold the answer, switch wg0 to the new private key, PROVE a fresh handshake, and only then let the old key go. If the handshake does not come, it puts the old key back and fails loudly. Ordering alone says when to discard, not whether the new key carries — and a host that locks itself out on the final step is the one failure nobody fixes remotely. Traefik's token is written here because here is where it exists, and the route table is then fetched directly with that token and that URL. Section 7 could not give this proof; it is not skipped, it is given where it can be. The lockdown comes dead last, with the full ruleset from SecureHostFirewall including the ICMP correction — ICMPv6 neighbour discovery and packet-too-big, IPv4 fragmentation-needed — because a bare policy drop takes IPv6 down within minutes and black-holes large transfers instead of failing them. The ruleset is validated with `nft -c` before it is applied, since a partially loaded ruleset means policy drop is in place and the exceptions are not. There is no automatic reopening: a firewall that reopens itself under failure is not a firewall, so the emergency release is a script for the provider's console. Verified without hardware: all seven files dash-clean; json_field reads full-tokenid, value and host_token without jq; the privilege list is identical to the config's; and the rendered ruleset carries policy drop, both ICMP families, 80/443, the DHCP rebind rule and the tunnel-only 22/8006. Step 2 unticked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
5d25018cc6 |
Build the golden template so nobody has to remember how
All three Block A traps are CHECKED here rather than assumed, because each one
has already cost a paid order and none of them shows a symptom until it is too
late.
Trap 2 is checked before anything is built: virt-filesystems is asked whether
the image uses LVM and whether root is the last partition, and the section
refuses if either is wrong. Debian's cloud image satisfies both by construction,
which is exactly why it is the base — but "satisfies it by construction" is a
claim about an artefact somebody else builds, and it gets verified rather than
trusted.
Traps 1 and 3 are checked AFTER the image is customised, because virt-customize
reports success even when apt quietly did not install something. qemu-guest-agent
has to be in /usr/bin or every provisioning run hangs in WaitForGuestAgent until
it times out, and the compose file inside the image has to carry `user:
www-data` or every occ call fails — including the acceptance check that decides
whether a customer's instance is usable.
The template is verified by its ATTRIBUTE, not its existence. VerifyVmTemplate
checks only that 9000 is there, which a VM that merely happens to be numbered
9000 also passes; `template: 1` is passed only by a template.
Docker comes from Docker's own repository, not Debian's. docker.io ships no
compose plugin and Debian's docker-compose is the old Python one, which does not
read this file at all.
The compose file is deliberately customer-independent: no password, no name, no
domain. Everything variable arrives in /opt/nextcloud/.env, written into the
guest by cloud-init at clone time. Baking a password into an image copies it
onto every host and into every instance, and leaves it there long after the
customer has changed it.
OVERWRITEPROTOCOL, OVERWRITEHOST and TRUSTED_PROXIES are set because TLS ends at
Traefik on the host. Without them Nextcloud builds its own URLs with http://,
the login loops, and WebDAV clients get handed an address that does not exist.
Storage is discovered rather than named: local-zfs is what the PVE installer
creates on ZFS, local-lvm on ext4, and `local` frequently cannot hold disks at
all — an importdisk there fails only after the copy.
Verified without hardware: dash-clean; the compose file parses, carries `user:
www-data` on the app service, publishes 80, and every credential is a ${...}
reference rather than a literal. Step 2 unticked, and it is the one the plan is
most insistent about: the template must actually be cloned, started, and asked
`occ status` as www-data. Without that clone it is not proven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
|
nexxo
|
a25e9c2fe6 |
Run Traefik as a service that fetches its own routes
Binary plus systemd unit, no Docker on the hypervisor: nftables stays the only owner of the host's firewall rules and a Docker daemon would bring its own chain. The names were read out of SshTraefikWriter::render() rather than guessed, which is what the handoff asks for in as many words. The route endpoint emits entryPoints ["websecure"] and certResolver "letsencrypt", so the static config declares exactly those. Name them differently here and the routers point at nothing while Traefik still reports a clean start — a failure with no symptom at the place it happens. An ordering problem this plan had not resolved: the http provider needs the durable host token, and that token only exists after POST /host/register, two sections later. Rather than reorder the section keys — they are the contract with the platform plan — the static config is written twice. Here with an empty token, so the service stands and holds 80 and 443, and again in Task 9 with the real one. The proof does not disappear, it moves to where it can be given. That proof asks the endpoint directly with the same token and URL the config carries, instead of counting Traefik's routers. A fresh host has no customers, so its table is legitimately empty, and "zero routers" would mean both "fine" and "never fetched". Ports are checked separately from the service. "Running" and "listening" are two claims, and Traefik starts cleanly even when a typo means an entryPoint was never created. The binary is checksummed against the release's own checksums file, for the same reason the ISO is: what listens on 80 and 443 and holds every customer's certificate does not get taken off the network unverified. The version is discovered at runtime, because a pinned number becomes a 404 mid-takeover. acme.json is created at 600 before Traefik ever runs. It holds the private keys of every customer certificate, and Traefik refuses wider permissions — rightly. The unit runs with CAP_NET_BIND_SERVICE and nothing else, ProtectSystem=strict, NoNewPrivileges: this is the one process on the box reachable from the open internet. Verified without hardware: dash-clean, the generated config parses as YAML, and it carries web/websecure/traefik as entryPoints, letsencrypt as the resolver, the Bearer header on the http provider, web redirecting to websecure, and 640 on the config with 600 on acme.json and the token file. Step 2 unticked — it wants a fetched route set, and that is Task 9's to show. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
c345f21e7d |
Join the tunnel, then tell the console everything so far
The three interface states come over from ConfigureWireguard.php exactly as that file works them out, because the one line they replace got two of them wrong. `systemctl enable --now wg-quick@wg0 || wg-quick up wg0` brought the interface up via its fallback WITHOUT the systemd enablement, so the tunnel did not come back after a reboot; and on the next attempt both halves failed with "wg0 already exists", so it retried forever against a tunnel that was working. Enable before start, because enable is the half that survives a reboot, and it is repaired even when the interface is already up by other means. The configuration is compared before it is written. The file used to be rewritten unconditionally with nothing reloading it, so correcting a wrong hub key showed a new file and identical behaviour — the running interface still held the old peer. Knowing whether it CHANGED is what lets an unchanged, working tunnel be left alone and a corrected one actually be applied. A change means a restart rather than `wg syncconf`, because syncconf applies peers only and a corrected Address or AllowedIPs would silently do nothing. AllowedIPs is computed, not copied. Writing the host's own address there would let the tunnel handshake and leave every other participant unreachable — a fault that looks like a routing problem somewhere else entirely. The handshake target is derived from --api rather than taking its own argument. CluPilot's tunnel address is already in there, and a second value meaning the same thing is a second value that eventually disagrees with the first. The tunnel counts only when it has been PROVEN, never when the file is on disk. A stored tunnel address without proof made every later connection attempt useless and the recovery was hand-editing the database. Once it is proven, flush_reports runs and the whole history from the rescue system onwards reaches the console carrying its original timestamps. Verified without hardware: dash-clean; the subnet arithmetic is right on octet boundaries and off them (192.168.5.130/25 gives 192.168.5.128/25, 172.16.4.9/12 gives 172.16.0.0/12); the API host parser handles a bare address, a scheme, a port and a path; and wg0.conf renders with AllowedIPs on the network rather than the host. Step 2 unticked — it wants all five earlier sections showing up in the console with real timestamps, and that needs a hub. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
e9b2862331 |
Bridge the network, and take it back if the host goes quiet
Reading ConfigureProxmox.php changed what this section does. The missing vmbr0 that killed the first paid order is a consequence of installing Proxmox ON TOP OF DEBIAN — "only the ISO installer writes that bridge into /etc/network/interfaces". This design installs from the ISO, so the bridge is normally already there. The section verifies it instead of building it: exists, has an address, and actually carries the default route. The old step checked only the first of those, threw the result away, and its comment claimed it recorded the absence. When there is no bridge, it builds one, and only then is this the dangerous section the plan calls it. Backup, five-minute rollback timer, switch, look, cancel — and the timer is a systemd unit rather than a backgrounded sleep, because a background job dies with its session and the session is exactly what breaks when the switch goes wrong. It is set BEFORE the change; setting it after would mean setting it at the moment the connection is already gone. The three provider shapes differ in one line, and that line decides whether the machine is still there afterwards. A routed single address needs pointopoint, because its gateway is outside its own subnet and without it the kernel has no route to reach it. Which shape applies is decided by asking `ip route get` about the gateway rather than by doing subnet arithmetic here — same logic the kernel will apply later. The reachability check is named for what it is: inside-out. A true outside-in proof would need a counterpart, and before the tunnel there is none. It is enough because a bridge that takes the host off the network takes both directions with it, and what it does not cover is precisely what the timer covers. One deviation from the plan, written into it: the nftables lockdown does NOT belong here. SecureHostFirewall ran LAST in the old pipeline, deliberately, so the tunnel had already carried every earlier step before SSH-to-the-world was closed. Here it would sit BEFORE wireguard_joined and shut port 22 with no handshake yet proven — the ordering that makes a host permanently unreachable. It moves to Task 9, where it used to be. The self-rollback stays here and covers the network change; the firewall keeps its manual emergency release, because a firewall that reopens itself under failure is not a firewall. The datacenter firewall comes over verbatim, including why `enable 1` alone is a trap: the management ipset is seeded from the public subnet, so enabling it blindly drops the tunnel address that every later step arrives on. Three settings in order, then read back — a pvesh set returning 0 is not the same as the value being in cluster.fw, and a write that lost quorum is exactly the case where every customer VM's rules stay inert while onboarding reports success. Verified without hardware: all four files dash-clean, and the generated interfaces file is correct for subnet, routed (with pointopoint) and dhcp. Step 2 unticked — the self-rollback has to be triggered on purpose on real hardware, and the plan is right that without that it is an assertion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
f214713113 |
Point Proxmox at the sources this Debian actually has
PVE arrives with the image now, so this section accepts it rather than installs it — but it needs every line of knowledge from InstallProxmoxVe.php to do that, just pointed the other way. The kernel is the claim, not the package. An installed proxmox-ve running on a Debian kernel is exactly the half state RebootIntoPveKernel exists because of, so uname is what gets checked. The codename table now verifies a pairing instead of creating one. An image whose PVE major does not match its Debian base was built wrong, and no amount of fixing package sources heals a PVE compiled against a different libc — that machine gets reinstalled with the right image. The other half of the table, codename to PVE major, is written down here because the ISO route needs it and the original file only had the suite side. Removing the subscription repositories is the part that everything after this depends on. A Proxmox image ships pve-enterprise, and without a subscription it fails every apt-get update — after which no package install in any later section succeeds. Both spellings are handled, because PVE 8 used one-line .list files and PVE 9 moved to deb822, and ceph is in the list because the same trap is set there a second time. The proof is a clean apt-get update at the end, which is the only thing that tells removal apart from overwriting. One thing changed after the first draft: when pveversion cannot be parsed, the pairing check used to be skipped in silence. That is the fake that R19 records as worse than no check at all, because it stops the next person from looking. It now says "Paarung UNGEPRÜFT" in the report and in the log. Verified without hardware: dash-clean; bookworm maps to PVE 8, trixie to PVE 9, forky is refused; the pveversion parser reads 9 and 8 out of both real formats and yields nothing for garbage; and against a copy of /etc/apt/sources.list.d holding all five subscription files plus an unrelated one, exactly the five go and the unrelated one stays. Step 2 unticked — there is no Proxmox here to accept. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
0f563545fd |
Write the image and survive the first reboot
The official Proxmox ISO, an answer file baked into it, and QEMU running it against the physical disks from the rescue system. Options were looked up rather than recalled: prepare-iso takes --fetch-from iso, --answer-file and --on-first-boot, and it names its own output, so the finished ISO is found by glob afterwards instead of being handed an --output flag I would have invented. validate-answer runs BEFORE the ISO is baked. An answer file with a typo drops the installer into interactive mode, where it then hangs invisibly inside QEMU until the hour runs out — the failure would arrive as a timeout and say nothing about the cause. The codename table comes over verbatim from InstallProxmoxVe.php, which the platform plan deletes. It is used here to install the assistant into the RESCUE system, so it keys off the rescue system's own codename. An unknown one aborts; not "trixie is newest, so trixie", because the next Debian will be unknown too and by then nobody is watching. The ISO is checksummed against the mirror's SHA256SUMS. Booting an operating system that could have become anything in transit is not a risk worth taking, and a truncated download is enough to cause it — malice is not required. The reboot carry-across is the gap this plan was amended for, and [first-boot] takes exactly one executable. So that one file carries everything: the script, its lib/, the progress file and the arguments, as a base64 archive in its own body. That also avoids importing the fresh ZFS pool from the rescue system to copy files into it. Two bugs found by running it rather than reading it. The staging copy put the work directory inside the state directory, so cp refused to copy a directory into itself — and had it not refused, the gigabyte ISO would have been base64'd into the hook that gets baked into that same ISO. The fix is not a better exclude: the two directories have opposite lifetimes, so they now live in different places, and a guard refuses loudly if anyone points them at each other again. Structure beats a rule someone has to remember. Verified without hardware: both files pass sh and dash. The generated answer file parses as TOML with the expected disk-list, zfs.raid, zfs.arc-max, first-boot and network.filter, and a 64-character root password. The codename table accepts bookworm and trixie and refuses bullseye and forky. The shim comes out at 26 KB with a 3 MB dummy ISO sitting in the work directory, proving it stayed out. Run in a fresh debian:13-slim container it unpacks to /opt/clupilot/bootstrap and /var/lib/clupilot, restores the progress file with its original timestamp, has 700 on the directory and 600 on the arguments that hold the WireGuard key, skips rescue_checked as already done, and fails cleanly on the missing qemu. Step 2 stays unticked: a container is not a rescue system, and nothing here has yet written to a disk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
af284d7d26 |
Give the bootstrap a spine and a way to report
Argument parsing, the state directory, and the half of the script that knows the way back to CluPilot. No sections are hooked up yet; main lists the nine it will get, in the order from spec §7, so what is missing stays visible. report() writes locally first and only then tries to send, because before wireguard_joined there is no path at all — the lines queue up and flush_reports delivers them later WITH THEIR OWN TIMESTAMPS. That last part is the whole point: a twenty-minute install that arrives in one batch looks like one second in the console, and nobody can see which section was slow. Every network operation is explicitly caught. The script runs under `set -e`, and a failing send is the normal case for the first five sections, not an error — a report that aborts the run would be the diagnosis that kills the patient. What is sent had no agreed shape. The section keys were the only agreement between the two plans; the envelope around them was not written down anywhere, so it is written down here, at the top of report.sh, for the platform side to read once. There is deliberately no "running" state: the console derives "open" from the absence of a report (spec §7), and a third state would be a second truth about the same thing. The sent-marker is a line count in its own file rather than a flag rewritten into progress.jsonl. That keeps the log append-only, so a power cut mid-write costs at most a partial last line instead of a rewritten file — and after the reboot in Task 3 that file is the only thing that remembers anything. Verified here, without hardware: both syntax checks pass; two reports against an unreachable CluPilot queue up without aborting and leave progress.sent at 0; a receiver brought up afterwards gets both in one batch, carrying 17:28:55 and 17:28:57 rather than the 17:29:22 they arrived at; a message containing quotes, a backslash, a tab and a newline survives as valid JSON; and section_done counts a `done` as done and a `failed` as not. Step 2 of this task is NOT ticked — none of that is a rescue system, and the plan is right to insist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |