Nine defects in the host pipeline, every one of which either stopped an
install or let a host reach `active` while it could not do its job.
The install-stoppers:
- InstallProxmoxVe hard-coded `bookworm`. A server ordered today boots
Debian 13 trixie, so it added the PVE 8 repo and keyring to a trixie
base. The codename now comes from /etc/os-release, the suite and keyring
are looked up per release, and an unknown one fails naming what it found
instead of guessing. Keys land in /usr/share/keyrings with Signed-By
rather than in the deprecated trusted.gpg.d, where they would vouch for
every other repository on the machine too.
- keyLogin() dialled wg_ip whenever it was filled, but wg_ip is persisted
inside the allocation lock BEFORE the handshake is ever proven. Attempt
1 reserved the address and failed at the handshake; every later attempt,
including a console Retry, died connecting to a tunnel that did not work
— before reaching the only code that could repair wg0.conf. Recovery
meant nulling hosts.wg_ip by hand. SSH now uses the tunnel only once the
`wg_peer` breadcrumb proves a handshake succeeded, which also rescues
the same shape in RebootIntoPveKernel, ConfigureProxmox,
CreateAutomationToken and SecureHostFirewall. Not a hole: 22 on the
public IP is open until SecureHostFirewall runs, that step runs last,
and by then the tunnel is necessarily proven. And nothing validated the
hub key/endpoint, so renderConfig() would emit `PublicKey = ` — now
refused before anything is installed, allocated or written.
- `systemctl enable --now wg-quick@wg0 || wg-quick up wg0` brought the
interface up without the enablement, so the tunnel did not come back
after the step-6 reboot; on the next attempt both halves failed with
"wg0 already exists" and the step retried forever on a working tunnel.
The three states are handled separately now, and a changed wg0.conf is
actually applied instead of only written.
- RebootIntoPveKernel removed the Debian kernel with `|| true`, ignored
update-grub's exit status, and rebooted. A /boot that filled up during
the full-upgrade could leave a machine only the provider's console can
reach. The pve kernel image, its initramfs and update-grub's exit status
are all proven first; anything missing fails loudly and does not reboot.
- ConfigureProxmox ran `ip link show vmbr0` and threw the result away
(its comment claimed it recorded the absence; it recorded nothing), then
repeated a rm -f, and always advanced. Proxmox on top of Debian does not
create vmbr0, so onboarding reported `active` with no bridge. It now
fails with the default route in the message and deliberately does not
build the bridge over the primary NIC from a remote shell.
The things that were quietly inert:
- Proxmox applies a guest's firewall rules only while the firewall is
enabled at datacenter level, and it ships disabled — so applyFirewall()'s
"80/443 only" rules were inert on every customer VM. Enabling it blindly
is the opposite trap (input policy defaults to DROP, node firewall
defaults to on, and its management ipset is seeded from the PUBLIC
subnet, not the tunnel), so: policy ACCEPT and the node firewall off
first, master switch last, read back to confirm. nftables stays the one
owner of the host's input policy.
- role_privs lacked Sys.Modify, which POST /cluster/backup requires
(pve-manager's PVE/API2/Backup.pm), so RegisterBackup 403'd and failed
EVERY customer run. And `pveum role add … || true` only ever applied the
privilege list on the run that created the role — it converges now, above
the token short-circuit so a replay reaches it.
- The nftables ruleset dropped all ICMP and ICMPv6. On a real host that
breaks IPv6 outright once the neighbour cache expires and black-holes
large transfers through PMTUD. Both families accept the types they need,
policy drop stays, and a DHCP client reply is allowed so a rebind cannot
lose the IPv4 address.
- api_token_ref used the `encrypted` cast, i.e. APP_KEY, against
SecretVault's own written rule. One rotation would have made every host's
Proxmox token undecryptable at once. Moved onto SecretCipher with a
migration that leaves any value it cannot read exactly as it is and says
so, rather than destroying a credential that exists nowhere else.
- Every plan version points at template_vmid 9000 and nothing created or
verified it, so the first paid order died in CloneVirtualMachine after
the customer had paid. VerifyVmTemplate reports that during onboarding
instead. It does not build a template: what goes into the golden image is
a product decision.
Three tests that proved nothing, fixed: `ran(…emergency-open-firewall.sh)`
only ever matched the `chmod 700` above it, because putFile() is not
recorded — the script's contents are now asserted, including that nothing
in it reopens the firewall on a timer or in the background. The port policy
was tested one-sidedly, so a mutation adding `tcp dport { 8006 } accept`
passed the suite; the ruleset is now read as a list of what it opens to an
unrestricted source. And the end-to-end test says in writing that it proves
sequencing only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A paid order that no host has room for used to FAIL at the reservation
step. The customer has paid by then, so that turned a sale into an
incident and left somebody holding money against nothing. The estate is
finite and booked thick, so "no host has room right now" is an ordinary
Tuesday — it means a machine has to be bought, which takes days.
So the order waits instead. poll(), not retry(): the wait is measured in
days and must not eat the run's attempt budget. It only fails after
PARK_DAYS (14) — long enough to buy, rack and onboard a server over a
weekend, short enough that a forgotten order surfaces instead of waiting
for ever.
Then say so, in the same words on both sides of the payment:
- The price sheet marks each package "wird sofort ausgeliefert" or
"Bereitstellung in 2-3 Werktagen", read from free capacity rather
than written into the page.
- The customer's own overview says their cloud is being prepared while
it is parked, instead of a stepper standing still for two days under
"wird eingerichtet" — which reads as broken.
- The console front page shows the same per-package answer, so an
operator can see what visitors are being promised.
And a capacity page for the decision that follows: who is waiting, what
the roomiest host still has, what the queue needs ("3x Start, 1x
Business" - the LARGEST parked package sets the minimum machine, because
an instance lives on one host), and what such a machine costs today from
the provider's live list. It orders nothing: buying a server is a
contract, and a bug in a capacity calculation must not sign one.
The operator can also say where each parked order goes. A pin is
honoured or it waits — never quietly redirected — because placing it
elsewhere answers a different question than the one they answered, and
they would find out from the finished instance.
Two things this shook out:
- `current_step` is an INDEX into the pipeline, not the step's key, and
it is cast to integer. `where('current_step', 'reserve_resources')`
reads as correct and matches step 0 of EVERY pipeline instead. Both
lookups go through HostCapacity::parkedRuns() now, which resolves the
index and filters by pipeline.
- The auction feed sends `hdd_hr` as a list, not a string. Casting it
printed "Array" into the console and raised a warning that took the
page down with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A verified custom domain was reported as the customer's address by
Instance::address(), by the portal and by the credentials mail while
nothing on the platform routed it: the Traefik router's rule was
hard-coded to {subdomain}.{zone}. The address a customer was handed
answered nothing, and a withdrawn domain stayed in Nextcloud's
trusted_domains forever, because the only thing that ever wrote either
was the initial provisioning run.
- TraefikWriter::write() takes a LIST of hostnames under one stable
router name — the platform address always, the verified domain as
well when there is one. One file per instance, so a withdrawal is a
rewrite rather than a second thing somebody has to remember.
- ConfigureDnsAndTls records the custom domain's certificate instead of
waiting on it: it depends on an A record in the customer's own zone,
which may never appear, and must never fail a run. The platform
address keeps its 840s deadline. instances.domain_cert_ok tells
"proven" apart from "answering", and the portal now says which.
- ConfigureNextcloud deletes trusted_domains 2 when there is no verified
domain, so a withdrawn one stops being trusted.
- New `address` pipeline (those two steps) plus ReapplyInstanceAddress,
which starts one against the order and refuses to start a second while
any run is in flight. The route is rewritten when the hostname list
differs from what the router carries — not on route_written, which
would short-circuit exactly the case a re-apply exists for.
- Triggered where the address changes: verification flipping either way
in clupilot:verify-domains, the customer's own domain page, and
CustomDomainAccess::deactivate() on a package downgrade.
- A maintenance run no longer condemns its subject: an address run that
failed used to mark the order failed and release the live instance
with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three console pages were fiction. The front page claimed 42 customers, 39
instances, four hosts named pve-fsn-1..3 and €7,842 a month, over a twelve-month
growth curve; the instance list held seven invented machines; the revenue page
reported churn and a trend for a business with no recorded history. All of it
was hard-coded. It read like a running company and measured nothing.
They now read the database. Two figures are gone rather than approximated —
the revenue trend, which needs a monthly history nobody records, and churn,
which needs a base the data cannot supply. ARR stays, labelled as the
projection it is. The green "all systems normal" badge is computed from the
notice list instead of asserted, and the notices themselves come from failed
runs, hosts reporting errors or gone quiet, and monitoring that is down.
Host load is the one number that had to agree with something else: placement
counts the VM disk allocation, ignores a failed instance that never got a VM,
and subtracts the host's reserve. A dashboard doing its own arithmetic would
show a host as comfortable while orders were already being refused on it, so it
uses the host's own accounting — with the filter moved into a scope both share,
and the sum preloaded so listing hosts stays one query.
The instance list drops the Nextcloud version column: that version is not
recorded anywhere, and a column filled with a plausible number is worse than no
column. Statuses the lifecycle writes but nobody had translated no longer
render as "admin.status.failed".
The price sheet also gains the included traffic, which the catalogue has always
carried and the page simply never printed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fsn-01.node.clupilot.com, numbered per datacenter and never reused — removing a
host must not renumber its neighbours onto its name, so the number is stored
rather than derived.
The record points at the host's WireGuard address, not its public IP: the name
exists so an operator can reach a host by name over the VPN, and publishing a
Proxmox host's public address would hand every scanner a target, which is the
one thing this network design avoids.
DNS is convenience, not a prerequisite: a failure logs an event and lets the
onboarding finish rather than stranding a host that is otherwise ready.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- applyFirewall sets policy_in=DROP so only 80/443 are exposed.
- Capacity/placement account for disk_gb (the real VM allocation), not the
smaller Nextcloud user quota — no systematic overcommit.
- CloudReady sent synchronously again; the step retries on mail failure with the
password preserved (no lost credential in a failed queue job).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Accept checkout.session.async_payment_succeeded (async methods) and dedupe on
the checkout session id, not the event id.
- committedGb still counts a failed instance while its VM exists (has vmid);
releases quota only when no VM was created — no overcommit vs no leak.
- CloudReady is queued (durable). Credential delivery is documented at-least-once
(a rare duplicate welcome email beats a lost credential; exactly-once = v1.1 outbox).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- StepResult::poll — polling steps (reboot) wait without consuming the retry
budget; the step owns its deadline. Reboot maxDuration > deadline.
- Failed runs move a Host subject to 'error' via ProvisioningSubject hook
(no host stuck 'onboarding').
- ConfigureWireguard allocates + reserves the wg_ip under a global lock;
unique index on hosts.wg_ip as a backstop against duplicate addresses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo
Claude