The last section carries three things, and none of them could sit anywhere
else: the registration and key swap, Traefik's token plus the proof that a route
table actually arrives, and the nftables lockdown that ran last in the old
pipeline for a reason.
`|| true` is gone from the pveum calls, and not by deleting it. `role add ||
true` applied the privilege list only on the run that first created the role, so
a privilege added later reached new hosts and no existing one — which is exactly
how Sys.Modify was missing everywhere, surfacing much later and somewhere else
as a 403 on POST /cluster/backup that failed a paying customer at
register_backup. So: add, and on failure modify, as two separate commands so a
failure can be attributed. For the user and the ACL the tolerated `|| true` is
replaced by looking: try, and if it fails, check whether the desired state holds
anyway. A `|| true` hides "already there" and "went wrong" equally well.
The privilege list is copied verbatim from config/provisioning.php and checked
against it — 18 privileges, byte-identical, Sys.Modify included. It is granted on
/ because both endpoints that need it check / and nothing narrower satisfies
them.
The key swap follows the four steps this plan was amended to require. Register
first and hold the answer, switch wg0 to the new private key, PROVE a fresh
handshake, and only then let the old key go. If the handshake does not come, it
puts the old key back and fails loudly. Ordering alone says when to discard, not
whether the new key carries — and a host that locks itself out on the final step
is the one failure nobody fixes remotely.
Traefik's token is written here because here is where it exists, and the route
table is then fetched directly with that token and that URL. Section 7 could not
give this proof; it is not skipped, it is given where it can be.
The lockdown comes dead last, with the full ruleset from SecureHostFirewall
including the ICMP correction — ICMPv6 neighbour discovery and packet-too-big,
IPv4 fragmentation-needed — because a bare policy drop takes IPv6 down within
minutes and black-holes large transfers instead of failing them. The ruleset is
validated with `nft -c` before it is applied, since a partially loaded ruleset
means policy drop is in place and the exceptions are not. There is no automatic
reopening: a firewall that reopens itself under failure is not a firewall, so the
emergency release is a script for the provider's console.
Verified without hardware: all seven files dash-clean; json_field reads
full-tokenid, value and host_token without jq; the privilege list is identical to
the config's; and the rendered ruleset carries policy drop, both ICMP families,
80/443, the DHCP rebind rule and the tunnel-only 22/8006. Step 2 unticked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three interface states come over from ConfigureWireguard.php exactly as
that file works them out, because the one line they replace got two of them
wrong. `systemctl enable --now wg-quick@wg0 || wg-quick up wg0` brought the
interface up via its fallback WITHOUT the systemd enablement, so the tunnel did
not come back after a reboot; and on the next attempt both halves failed with
"wg0 already exists", so it retried forever against a tunnel that was working.
Enable before start, because enable is the half that survives a reboot, and it
is repaired even when the interface is already up by other means.
The configuration is compared before it is written. The file used to be
rewritten unconditionally with nothing reloading it, so correcting a wrong hub
key showed a new file and identical behaviour — the running interface still
held the old peer. Knowing whether it CHANGED is what lets an unchanged, working
tunnel be left alone and a corrected one actually be applied. A change means a
restart rather than `wg syncconf`, because syncconf applies peers only and a
corrected Address or AllowedIPs would silently do nothing.
AllowedIPs is computed, not copied. Writing the host's own address there would
let the tunnel handshake and leave every other participant unreachable — a fault
that looks like a routing problem somewhere else entirely.
The handshake target is derived from --api rather than taking its own argument.
CluPilot's tunnel address is already in there, and a second value meaning the
same thing is a second value that eventually disagrees with the first.
The tunnel counts only when it has been PROVEN, never when the file is on disk.
A stored tunnel address without proof made every later connection attempt
useless and the recovery was hand-editing the database. Once it is proven,
flush_reports runs and the whole history from the rescue system onwards reaches
the console carrying its original timestamps.
Verified without hardware: dash-clean; the subnet arithmetic is right on octet
boundaries and off them (192.168.5.130/25 gives 192.168.5.128/25, 172.16.4.9/12
gives 172.16.0.0/12); the API host parser handles a bare address, a scheme, a
port and a path; and wg0.conf renders with AllowedIPs on the network rather than
the host. Step 2 unticked — it wants all five earlier sections showing up in the
console with real timestamps, and that needs a hub.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reading ConfigureProxmox.php changed what this section does. The missing vmbr0
that killed the first paid order is a consequence of installing Proxmox ON TOP
OF DEBIAN — "only the ISO installer writes that bridge into
/etc/network/interfaces". This design installs from the ISO, so the bridge is
normally already there. The section verifies it instead of building it: exists,
has an address, and actually carries the default route. The old step checked
only the first of those, threw the result away, and its comment claimed it
recorded the absence.
When there is no bridge, it builds one, and only then is this the dangerous
section the plan calls it. Backup, five-minute rollback timer, switch, look,
cancel — and the timer is a systemd unit rather than a backgrounded sleep,
because a background job dies with its session and the session is exactly what
breaks when the switch goes wrong. It is set BEFORE the change; setting it after
would mean setting it at the moment the connection is already gone.
The three provider shapes differ in one line, and that line decides whether the
machine is still there afterwards. A routed single address needs pointopoint,
because its gateway is outside its own subnet and without it the kernel has no
route to reach it. Which shape applies is decided by asking `ip route get` about
the gateway rather than by doing subnet arithmetic here — same logic the kernel
will apply later.
The reachability check is named for what it is: inside-out. A true outside-in
proof would need a counterpart, and before the tunnel there is none. It is
enough because a bridge that takes the host off the network takes both
directions with it, and what it does not cover is precisely what the timer
covers.
One deviation from the plan, written into it: the nftables lockdown does NOT
belong here. SecureHostFirewall ran LAST in the old pipeline, deliberately, so
the tunnel had already carried every earlier step before SSH-to-the-world was
closed. Here it would sit BEFORE wireguard_joined and shut port 22 with no
handshake yet proven — the ordering that makes a host permanently unreachable.
It moves to Task 9, where it used to be. The self-rollback stays here and covers
the network change; the firewall keeps its manual emergency release, because a
firewall that reopens itself under failure is not a firewall.
The datacenter firewall comes over verbatim, including why `enable 1` alone is a
trap: the management ipset is seeded from the public subnet, so enabling it
blindly drops the tunnel address that every later step arrives on. Three
settings in order, then read back — a pvesh set returning 0 is not the same as
the value being in cluster.fw, and a write that lost quorum is exactly the case
where every customer VM's rules stay inert while onboarding reports success.
Verified without hardware: all four files dash-clean, and the generated
interfaces file is correct for subnet, routed (with pointopoint) and dhcp.
Step 2 unticked — the self-rollback has to be triggered on purpose on real
hardware, and the plan is right that without that it is an assertion.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nexxo