Nine defects in the host pipeline, every one of which either stopped an
install or let a host reach `active` while it could not do its job.
The install-stoppers:
- InstallProxmoxVe hard-coded `bookworm`. A server ordered today boots
Debian 13 trixie, so it added the PVE 8 repo and keyring to a trixie
base. The codename now comes from /etc/os-release, the suite and keyring
are looked up per release, and an unknown one fails naming what it found
instead of guessing. Keys land in /usr/share/keyrings with Signed-By
rather than in the deprecated trusted.gpg.d, where they would vouch for
every other repository on the machine too.
- keyLogin() dialled wg_ip whenever it was filled, but wg_ip is persisted
inside the allocation lock BEFORE the handshake is ever proven. Attempt
1 reserved the address and failed at the handshake; every later attempt,
including a console Retry, died connecting to a tunnel that did not work
— before reaching the only code that could repair wg0.conf. Recovery
meant nulling hosts.wg_ip by hand. SSH now uses the tunnel only once the
`wg_peer` breadcrumb proves a handshake succeeded, which also rescues
the same shape in RebootIntoPveKernel, ConfigureProxmox,
CreateAutomationToken and SecureHostFirewall. Not a hole: 22 on the
public IP is open until SecureHostFirewall runs, that step runs last,
and by then the tunnel is necessarily proven. And nothing validated the
hub key/endpoint, so renderConfig() would emit `PublicKey = ` — now
refused before anything is installed, allocated or written.
- `systemctl enable --now wg-quick@wg0 || wg-quick up wg0` brought the
interface up without the enablement, so the tunnel did not come back
after the step-6 reboot; on the next attempt both halves failed with
"wg0 already exists" and the step retried forever on a working tunnel.
The three states are handled separately now, and a changed wg0.conf is
actually applied instead of only written.
- RebootIntoPveKernel removed the Debian kernel with `|| true`, ignored
update-grub's exit status, and rebooted. A /boot that filled up during
the full-upgrade could leave a machine only the provider's console can
reach. The pve kernel image, its initramfs and update-grub's exit status
are all proven first; anything missing fails loudly and does not reboot.
- ConfigureProxmox ran `ip link show vmbr0` and threw the result away
(its comment claimed it recorded the absence; it recorded nothing), then
repeated a rm -f, and always advanced. Proxmox on top of Debian does not
create vmbr0, so onboarding reported `active` with no bridge. It now
fails with the default route in the message and deliberately does not
build the bridge over the primary NIC from a remote shell.
The things that were quietly inert:
- Proxmox applies a guest's firewall rules only while the firewall is
enabled at datacenter level, and it ships disabled — so applyFirewall()'s
"80/443 only" rules were inert on every customer VM. Enabling it blindly
is the opposite trap (input policy defaults to DROP, node firewall
defaults to on, and its management ipset is seeded from the PUBLIC
subnet, not the tunnel), so: policy ACCEPT and the node firewall off
first, master switch last, read back to confirm. nftables stays the one
owner of the host's input policy.
- role_privs lacked Sys.Modify, which POST /cluster/backup requires
(pve-manager's PVE/API2/Backup.pm), so RegisterBackup 403'd and failed
EVERY customer run. And `pveum role add … || true` only ever applied the
privilege list on the run that created the role — it converges now, above
the token short-circuit so a replay reaches it.
- The nftables ruleset dropped all ICMP and ICMPv6. On a real host that
breaks IPv6 outright once the neighbour cache expires and black-holes
large transfers through PMTUD. Both families accept the types they need,
policy drop stays, and a DHCP client reply is allowed so a rebind cannot
lose the IPv4 address.
- api_token_ref used the `encrypted` cast, i.e. APP_KEY, against
SecretVault's own written rule. One rotation would have made every host's
Proxmox token undecryptable at once. Moved onto SecretCipher with a
migration that leaves any value it cannot read exactly as it is and says
so, rather than destroying a credential that exists nowhere else.
- Every plan version points at template_vmid 9000 and nothing created or
verified it, so the first paid order died in CloneVirtualMachine after
the customer had paid. VerifyVmTemplate reports that during onboarding
instead. It does not build a template: what goes into the golden image is
a product decision.
Three tests that proved nothing, fixed: `ran(…emergency-open-firewall.sh)`
only ever matched the `chmod 700` above it, because putFile() is not
recorded — the script's contents are now asserted, including that nothing
in it reopens the firewall on a timer or in the background. The port policy
was tested one-sidedly, so a mutation adding `tcp dport { 8006 } accept`
passed the suite; the ruleset is now read as a list of what it opens to an
unrestricted source. And the end-to-end test says in writing that it proves
sequencing only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- ConfigureWireguard and RemoveWireguardPeer mutate the hub too, and were not
taking the lock, so a sync could still read the interface around them and
write stale state. Both now hold wireguard:hub, which is what makes the
read/mutation guarantee actually hold.
- The duplicate-key check ran before the allocation lock, so two concurrent
creations with the same key both passed it and the loser hit the unique index
as an unhandled 500. The check moved inside the lock, with the index kept as
a caught backstop for any writer that does not take it — reproduced in a test
by inserting a colliding row from within the allocation call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keeps the wg_peer resource gated on a verified handshake (idempotency) while
ensuring PurgeHost can always remove the hub peer, even on terminal failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard checks setup command results and only records the wg_peer
resource after the handshake verifies, so a transient setup failure re-runs
the full setup instead of getting stuck on the idempotent path.
- Host removal now deactivates immediately and queues PurgeHost, which deletes
runs under the runner lock (waiting out a long step) — no 15s LockTimeout error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hub_ip is now defined in config (default: subnet .1), so a custom
CLUPILOT_WG_SUBNET no longer pings the wrong handshake target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- StepResult::poll — polling steps (reboot) wait without consuming the retry
budget; the step owns its deadline. Reboot maxDuration > deadline.
- Failed runs move a Host subject to 'error' via ProvisioningSubject hook
(no host stuck 'onboarding').
- ConfigureWireguard allocates + reserves the wg_ip under a global lock;
unique index on hosts.wg_ip as a backstop against duplicate addresses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CreateAutomationToken now bootstraps the pveum role/user/token over the
authenticated SSH session (a fresh host has no API token yet); ProxmoxClient
is read-only in A.
- ConfigureWireguard re-verifies the handshake on the idempotent replay path,
never advancing over a dead tunnel.
- PhpseclibRemoteShell treats a missing SSH exit status as failure (255).
- connectWithKey verifies the pinned host key fingerprint on later logins.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo