The owner caught me putting the archive on clupilot.cloud. That is the customer
zone, and looking into it turned up the same confusion already sitting in the
code — in the one place that matters most.
RegisterHostDns says "fsn-01.node.clupilot.com" in its own docblock. The
validation comment in Datacenters says it. ServicesTest writes it out verbatim.
The step itself built the name from config('provisioning.dns.zone') — the
CUSTOMER zone — so on this installation a host was actually called
fsn-01.node.clupilot.cloud. Three places asserting one thing and the code doing
another.
OfficialDomains explains why that matters and is worth not weakening: two
registrable domains by design, the company's for site, portal and console, the
instance zone for customer workloads. A Nextcloud is third-party software that
strangers sign into, and on the same registrable domain as the portal it shares
cookie scope with it. A host name in that zone does not break the separation,
but it puts it in question, and the next slip is more expensive.
So there is now a platform_zone, derived from APP_URL when unset, and
RegisterHostDns uses it.
My own archiveUrl was broken for a second reason. It fell back to url() when
APP_HOST is empty — and APP_HOST is empty on most installations, because empty
means "the portal answers on any hostname" and that is the default. Called from
the console, url() would have produced the CONSOLE hostname, and the line would
have 404'd on a machine that is not allowed to reach the admin area at all. It
takes the host from APP_URL now.
The script no longer guesses its own name. It used reverse DNS, then the tunnel
address, then a hard-coded clupilot.net — a third domain that appears nowhere
else in this project and was simply invented. PrepareBaseSystem has the same
invention. A guessed name does not stay guessed: it ends up in /etc/hostname, in
/etc/hosts, in every log line and in every certificate request the machine ever
makes. CluPilot knows the name because it just assigned it, so it passes --fqdn
and the script refuses without it. The value now also survives the reboot in the
arguments file, which it would not have.
The ACME contact moved to .com for the same reason it was wrong: the operator
does not live in the customer zone.
Open, and NOT decided here: the owner also wants the host to get a public DNS
record and a certificate on the .com name. RegisterHostDns deliberately writes
host names only into the tunnel's dnsmasq, and says why — publishing them hands
every scanner the internal subnet and roughly how many hosts sit behind it.
Nothing in the current design needs a public certificate for a host's own name;
Traefik serves customer domains, not this one. Reversing that is a security
decision and belongs to the owner, not to this commit.
1992 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine defects in the host pipeline, every one of which either stopped an
install or let a host reach `active` while it could not do its job.
The install-stoppers:
- InstallProxmoxVe hard-coded `bookworm`. A server ordered today boots
Debian 13 trixie, so it added the PVE 8 repo and keyring to a trixie
base. The codename now comes from /etc/os-release, the suite and keyring
are looked up per release, and an unknown one fails naming what it found
instead of guessing. Keys land in /usr/share/keyrings with Signed-By
rather than in the deprecated trusted.gpg.d, where they would vouch for
every other repository on the machine too.
- keyLogin() dialled wg_ip whenever it was filled, but wg_ip is persisted
inside the allocation lock BEFORE the handshake is ever proven. Attempt
1 reserved the address and failed at the handshake; every later attempt,
including a console Retry, died connecting to a tunnel that did not work
— before reaching the only code that could repair wg0.conf. Recovery
meant nulling hosts.wg_ip by hand. SSH now uses the tunnel only once the
`wg_peer` breadcrumb proves a handshake succeeded, which also rescues
the same shape in RebootIntoPveKernel, ConfigureProxmox,
CreateAutomationToken and SecureHostFirewall. Not a hole: 22 on the
public IP is open until SecureHostFirewall runs, that step runs last,
and by then the tunnel is necessarily proven. And nothing validated the
hub key/endpoint, so renderConfig() would emit `PublicKey = ` — now
refused before anything is installed, allocated or written.
- `systemctl enable --now wg-quick@wg0 || wg-quick up wg0` brought the
interface up without the enablement, so the tunnel did not come back
after the step-6 reboot; on the next attempt both halves failed with
"wg0 already exists" and the step retried forever on a working tunnel.
The three states are handled separately now, and a changed wg0.conf is
actually applied instead of only written.
- RebootIntoPveKernel removed the Debian kernel with `|| true`, ignored
update-grub's exit status, and rebooted. A /boot that filled up during
the full-upgrade could leave a machine only the provider's console can
reach. The pve kernel image, its initramfs and update-grub's exit status
are all proven first; anything missing fails loudly and does not reboot.
- ConfigureProxmox ran `ip link show vmbr0` and threw the result away
(its comment claimed it recorded the absence; it recorded nothing), then
repeated a rm -f, and always advanced. Proxmox on top of Debian does not
create vmbr0, so onboarding reported `active` with no bridge. It now
fails with the default route in the message and deliberately does not
build the bridge over the primary NIC from a remote shell.
The things that were quietly inert:
- Proxmox applies a guest's firewall rules only while the firewall is
enabled at datacenter level, and it ships disabled — so applyFirewall()'s
"80/443 only" rules were inert on every customer VM. Enabling it blindly
is the opposite trap (input policy defaults to DROP, node firewall
defaults to on, and its management ipset is seeded from the PUBLIC
subnet, not the tunnel), so: policy ACCEPT and the node firewall off
first, master switch last, read back to confirm. nftables stays the one
owner of the host's input policy.
- role_privs lacked Sys.Modify, which POST /cluster/backup requires
(pve-manager's PVE/API2/Backup.pm), so RegisterBackup 403'd and failed
EVERY customer run. And `pveum role add … || true` only ever applied the
privilege list on the run that created the role — it converges now, above
the token short-circuit so a replay reaches it.
- The nftables ruleset dropped all ICMP and ICMPv6. On a real host that
breaks IPv6 outright once the neighbour cache expires and black-holes
large transfers through PMTUD. Both families accept the types they need,
policy drop stays, and a DHCP client reply is allowed so a rebind cannot
lose the IPv4 address.
- api_token_ref used the `encrypted` cast, i.e. APP_KEY, against
SecretVault's own written rule. One rotation would have made every host's
Proxmox token undecryptable at once. Moved onto SecretCipher with a
migration that leaves any value it cannot read exactly as it is and says
so, rather than destroying a credential that exists nowhere else.
- Every plan version points at template_vmid 9000 and nothing created or
verified it, so the first paid order died in CloneVirtualMachine after
the customer had paid. VerifyVmTemplate reports that during onboarding
instead. It does not build a template: what goes into the golden image is
a product decision.
Three tests that proved nothing, fixed: `ran(…emergency-open-firewall.sh)`
only ever matched the `chmod 700` above it, because putFile() is not
recorded — the script's contents are now asserted, including that nothing
in it reopens the firewall on a timer or in the background. The port policy
was tested one-sidedly, so a mutation adding `tcp dport { 8006 } accept`
passed the suite; the ruleset is now read as a list of what it opens to an
unrestricted source. And the end-to-end test says in writing that it proves
sequencing only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SSH private key CluPilot deploys to every host is now stored through
SecretVault (ssh.private_key), preferred over CLUPILOT_SSH_PRIVATE_KEY(_PATH)
at all three real consumers — SshTraefikWriter and HostStep::keyLogin.
kuma.password/kuma.totp stay in .env: they authenticate a separate Python
container at boot, and nothing in this app reads them, so the vault would
have nothing to wire them to.
A new /admin/infrastructure page, backed by App\Support\ProvisioningSettings,
makes the non-secret deployment settings that used to force a shell —
DNS zone, WireGuard hub endpoint and public key, Traefik's dynamic-config
path, the SSH public key, and the monitoring bridge URL — visible and
editable from the console, each wired to every real consumer. WG subnet/hub
IP and Kuma's own connection details stay in .env: the compose file and a
separate container read those before this application ever does.
The failure may be a lost response to a request that did create the record.
Advancing straight away stored no id, so PurgeHost could never remove it and the
host's management address stayed published. The upsert is idempotent, so a retry
recovers the id; only after the attempts run out does the onboarding continue
without a name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two legacy codes can normalise to the same label (eu_west and eu-west), and
separate counters then handed both of them eu-west-01 — a unique-constraint
failure inside the reservation, which blocks onboarding rather than being a DNS
problem the step can shrug off. Lock, counter and the in-use check all key off
the label now.
(The helper was also called label(), which HostStep already declares as the
step's display name — renamed.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
alpha_dash accepted eu_west, -edge and edge-, none of which are valid DNS
labels — every host in such a datacenter would have failed registration and,
because DNS is non-fatal, silently ended up without a name. The console now
requires a proper label, and the step normalises anything created before that
rule so existing rows still get a usable name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex was right that my never-reuse claim did not hold:
- The lock was released with only a candidate in hand, so two concurrent
onboardings in one datacenter could pick the same name and overwrite each
other's record. The name is now persisted on the host while still locked.
- The number was derived from the hosts that happen to exist, so removing the
highest one handed that number straight back out — a cached name would then
resolve to a different machine. The counter is stored per datacenter, floored
by what is actually in use so a wiped settings store cannot reissue live names.
- PurgeHost deleted the row that carried the record id, leaving the machine's
management address published with nothing left to clean it up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fsn-01.node.clupilot.com, numbered per datacenter and never reused — removing a
host must not renumber its neighbours onto its name, so the number is stored
rather than derived.
The record points at the host's WireGuard address, not its public IP: the name
exists so an operator can reach a host by name over the VPN, and publishing a
Proxmox host's public address would hand every scanner a target, which is the
one thing this network design avoids.
DNS is convenience, not a prerequisite: a failure logs an event and lets the
onboarding finish rather than stranding a host that is otherwise ready.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard and RemoveWireguardPeer mutate the hub too, and were not
taking the lock, so a sync could still read the interface around them and
write stale state. Both now hold wireguard:hub, which is what makes the
read/mutation guarantee actually hold.
- The duplicate-key check ran before the allocation lock, so two concurrent
creations with the same key both passed it and the loser hit the unique index
as an unhandled 500. The check moved inside the lock, with the index kept as
a caught backstop for any writer that does not take it — reproduced in a test
by inserting a colliding row from within the allocation call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keeps the wg_peer resource gated on a verified handshake (idempotency) while
ensuring PurgeHost can always remove the hub peer, even on terminal failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard checks setup command results and only records the wg_peer
resource after the handshake verifies, so a transient setup failure re-runs
the full setup instead of getting stuck on the idempotent path.
- Host removal now deactivates immediately and queues PurgeHost, which deletes
runs under the runner lock (waiting out a long step) — no 15s LockTimeout error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Removing a host now removes its WireGuard hub peer so a freed wg_ip can't
route to the removed server via a stale peer.
- RebootIntoPveKernel clears reboot_issued/deadline when it times out, so the
manual retry action actually re-issues a reboot instead of failing instantly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hub_ip is now defined in config (default: subnet .1), so a custom
CLUPILOT_WG_SUBNET no longer pings the wrong handshake target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- RunRunner returns early for a waiting run whose next_attempt_at is in the
future, so stale/duplicate jobs can't bypass backoff.
- RegisterCapacity takes the largest VM-capable datastore instead of summing
overlapping pools (local + local-lvm), preventing placement overcommit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- StepResult::poll — polling steps (reboot) wait without consuming the retry
budget; the step owns its deadline. Reboot maxDuration > deadline.
- Failed runs move a Host subject to 'error' via ProvisioningSubject hook
(no host stuck 'onboarding').
- ConfigureWireguard allocates + reserves the wg_ip under a global lock;
unique index on hosts.wg_ip as a backstop against duplicate addresses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CreateAutomationToken now bootstraps the pveum role/user/token over the
authenticated SSH session (a fresh host has no API token yet); ProxmoxClient
is read-only in A.
- ConfigureWireguard re-verifies the handshake on the idempotent replay path,
never advancing over a dead tunnel.
- PhpseclibRemoteShell treats a missing SSH exit status as failure (255).
- connectWithKey verifies the pinned host key fingerprint on later logins.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo