The owner caught me putting the archive on clupilot.cloud. That is the customer
zone, and looking into it turned up the same confusion already sitting in the
code — in the one place that matters most.
RegisterHostDns says "fsn-01.node.clupilot.com" in its own docblock. The
validation comment in Datacenters says it. ServicesTest writes it out verbatim.
The step itself built the name from config('provisioning.dns.zone') — the
CUSTOMER zone — so on this installation a host was actually called
fsn-01.node.clupilot.cloud. Three places asserting one thing and the code doing
another.
OfficialDomains explains why that matters and is worth not weakening: two
registrable domains by design, the company's for site, portal and console, the
instance zone for customer workloads. A Nextcloud is third-party software that
strangers sign into, and on the same registrable domain as the portal it shares
cookie scope with it. A host name in that zone does not break the separation,
but it puts it in question, and the next slip is more expensive.
So there is now a platform_zone, derived from APP_URL when unset, and
RegisterHostDns uses it.
My own archiveUrl was broken for a second reason. It fell back to url() when
APP_HOST is empty — and APP_HOST is empty on most installations, because empty
means "the portal answers on any hostname" and that is the default. Called from
the console, url() would have produced the CONSOLE hostname, and the line would
have 404'd on a machine that is not allowed to reach the admin area at all. It
takes the host from APP_URL now.
The script no longer guesses its own name. It used reverse DNS, then the tunnel
address, then a hard-coded clupilot.net — a third domain that appears nowhere
else in this project and was simply invented. PrepareBaseSystem has the same
invention. A guessed name does not stay guessed: it ends up in /etc/hostname, in
/etc/hosts, in every log line and in every certificate request the machine ever
makes. CluPilot knows the name because it just assigned it, so it passes --fqdn
and the script refuses without it. The value now also survives the reboot in the
arguments file, which it would not have.
The ACME contact moved to .com for the same reason it was wrong: the operator
does not live in the customer zone.
Open, and NOT decided here: the owner also wants the host to get a public DNS
record and a certificate on the .com name. RegisterHostDns deliberately writes
host names only into the tunnel's dnsmasq, and says why — publishing them hands
every scanner the internal subnet and roughly how many hosts sit behind it.
Nothing in the current design needs a public certificate for a host's own name;
Traefik serves customer domains, not this one. Reversing that is a security
decision and belongs to the owner, not to this commit.
1992 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The last section carries three things, and none of them could sit anywhere
else: the registration and key swap, Traefik's token plus the proof that a route
table actually arrives, and the nftables lockdown that ran last in the old
pipeline for a reason.
`|| true` is gone from the pveum calls, and not by deleting it. `role add ||
true` applied the privilege list only on the run that first created the role, so
a privilege added later reached new hosts and no existing one — which is exactly
how Sys.Modify was missing everywhere, surfacing much later and somewhere else
as a 403 on POST /cluster/backup that failed a paying customer at
register_backup. So: add, and on failure modify, as two separate commands so a
failure can be attributed. For the user and the ACL the tolerated `|| true` is
replaced by looking: try, and if it fails, check whether the desired state holds
anyway. A `|| true` hides "already there" and "went wrong" equally well.
The privilege list is copied verbatim from config/provisioning.php and checked
against it — 18 privileges, byte-identical, Sys.Modify included. It is granted on
/ because both endpoints that need it check / and nothing narrower satisfies
them.
The key swap follows the four steps this plan was amended to require. Register
first and hold the answer, switch wg0 to the new private key, PROVE a fresh
handshake, and only then let the old key go. If the handshake does not come, it
puts the old key back and fails loudly. Ordering alone says when to discard, not
whether the new key carries — and a host that locks itself out on the final step
is the one failure nobody fixes remotely.
Traefik's token is written here because here is where it exists, and the route
table is then fetched directly with that token and that URL. Section 7 could not
give this proof; it is not skipped, it is given where it can be.
The lockdown comes dead last, with the full ruleset from SecureHostFirewall
including the ICMP correction — ICMPv6 neighbour discovery and packet-too-big,
IPv4 fragmentation-needed — because a bare policy drop takes IPv6 down within
minutes and black-holes large transfers instead of failing them. The ruleset is
validated with `nft -c` before it is applied, since a partially loaded ruleset
means policy drop is in place and the exceptions are not. There is no automatic
reopening: a firewall that reopens itself under failure is not a firewall, so the
emergency release is a script for the provider's console.
Verified without hardware: all seven files dash-clean; json_field reads
full-tokenid, value and host_token without jq; the privilege list is identical to
the config's; and the rendered ruleset carries policy drop, both ICMP families,
80/443, the DHCP rebind rule and the tunnel-only 22/8006. Step 2 unticked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All three Block A traps are CHECKED here rather than assumed, because each one
has already cost a paid order and none of them shows a symptom until it is too
late.
Trap 2 is checked before anything is built: virt-filesystems is asked whether
the image uses LVM and whether root is the last partition, and the section
refuses if either is wrong. Debian's cloud image satisfies both by construction,
which is exactly why it is the base — but "satisfies it by construction" is a
claim about an artefact somebody else builds, and it gets verified rather than
trusted.
Traps 1 and 3 are checked AFTER the image is customised, because virt-customize
reports success even when apt quietly did not install something. qemu-guest-agent
has to be in /usr/bin or every provisioning run hangs in WaitForGuestAgent until
it times out, and the compose file inside the image has to carry `user:
www-data` or every occ call fails — including the acceptance check that decides
whether a customer's instance is usable.
The template is verified by its ATTRIBUTE, not its existence. VerifyVmTemplate
checks only that 9000 is there, which a VM that merely happens to be numbered
9000 also passes; `template: 1` is passed only by a template.
Docker comes from Docker's own repository, not Debian's. docker.io ships no
compose plugin and Debian's docker-compose is the old Python one, which does not
read this file at all.
The compose file is deliberately customer-independent: no password, no name, no
domain. Everything variable arrives in /opt/nextcloud/.env, written into the
guest by cloud-init at clone time. Baking a password into an image copies it
onto every host and into every instance, and leaves it there long after the
customer has changed it.
OVERWRITEPROTOCOL, OVERWRITEHOST and TRUSTED_PROXIES are set because TLS ends at
Traefik on the host. Without them Nextcloud builds its own URLs with http://,
the login loops, and WebDAV clients get handed an address that does not exist.
Storage is discovered rather than named: local-zfs is what the PVE installer
creates on ZFS, local-lvm on ext4, and `local` frequently cannot hold disks at
all — an importdisk there fails only after the copy.
Verified without hardware: dash-clean; the compose file parses, carries `user:
www-data` on the app service, publishes 80, and every credential is a ${...}
reference rather than a literal. Step 2 unticked, and it is the one the plan is
most insistent about: the template must actually be cloned, started, and asked
`occ status` as www-data. Without that clone it is not proven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Binary plus systemd unit, no Docker on the hypervisor: nftables stays the only
owner of the host's firewall rules and a Docker daemon would bring its own
chain.
The names were read out of SshTraefikWriter::render() rather than guessed, which
is what the handoff asks for in as many words. The route endpoint emits
entryPoints ["websecure"] and certResolver "letsencrypt", so the static config
declares exactly those. Name them differently here and the routers point at
nothing while Traefik still reports a clean start — a failure with no symptom
at the place it happens.
An ordering problem this plan had not resolved: the http provider needs the
durable host token, and that token only exists after POST /host/register, two
sections later. Rather than reorder the section keys — they are the contract
with the platform plan — the static config is written twice. Here with an empty
token, so the service stands and holds 80 and 443, and again in Task 9 with the
real one. The proof does not disappear, it moves to where it can be given.
That proof asks the endpoint directly with the same token and URL the config
carries, instead of counting Traefik's routers. A fresh host has no customers,
so its table is legitimately empty, and "zero routers" would mean both "fine"
and "never fetched".
Ports are checked separately from the service. "Running" and "listening" are two
claims, and Traefik starts cleanly even when a typo means an entryPoint was
never created.
The binary is checksummed against the release's own checksums file, for the same
reason the ISO is: what listens on 80 and 443 and holds every customer's
certificate does not get taken off the network unverified. The version is
discovered at runtime, because a pinned number becomes a 404 mid-takeover.
acme.json is created at 600 before Traefik ever runs. It holds the private keys
of every customer certificate, and Traefik refuses wider permissions — rightly.
The unit runs with CAP_NET_BIND_SERVICE and nothing else, ProtectSystem=strict,
NoNewPrivileges: this is the one process on the box reachable from the open
internet.
Verified without hardware: dash-clean, the generated config parses as YAML, and
it carries web/websecure/traefik as entryPoints, letsencrypt as the resolver,
the Bearer header on the http provider, web redirecting to websecure, and 640 on
the config with 600 on acme.json and the token file. Step 2 unticked — it wants a
fetched route set, and that is Task 9's to show.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three interface states come over from ConfigureWireguard.php exactly as
that file works them out, because the one line they replace got two of them
wrong. `systemctl enable --now wg-quick@wg0 || wg-quick up wg0` brought the
interface up via its fallback WITHOUT the systemd enablement, so the tunnel did
not come back after a reboot; and on the next attempt both halves failed with
"wg0 already exists", so it retried forever against a tunnel that was working.
Enable before start, because enable is the half that survives a reboot, and it
is repaired even when the interface is already up by other means.
The configuration is compared before it is written. The file used to be
rewritten unconditionally with nothing reloading it, so correcting a wrong hub
key showed a new file and identical behaviour — the running interface still
held the old peer. Knowing whether it CHANGED is what lets an unchanged, working
tunnel be left alone and a corrected one actually be applied. A change means a
restart rather than `wg syncconf`, because syncconf applies peers only and a
corrected Address or AllowedIPs would silently do nothing.
AllowedIPs is computed, not copied. Writing the host's own address there would
let the tunnel handshake and leave every other participant unreachable — a fault
that looks like a routing problem somewhere else entirely.
The handshake target is derived from --api rather than taking its own argument.
CluPilot's tunnel address is already in there, and a second value meaning the
same thing is a second value that eventually disagrees with the first.
The tunnel counts only when it has been PROVEN, never when the file is on disk.
A stored tunnel address without proof made every later connection attempt
useless and the recovery was hand-editing the database. Once it is proven,
flush_reports runs and the whole history from the rescue system onwards reaches
the console carrying its original timestamps.
Verified without hardware: dash-clean; the subnet arithmetic is right on octet
boundaries and off them (192.168.5.130/25 gives 192.168.5.128/25, 172.16.4.9/12
gives 172.16.0.0/12); the API host parser handles a bare address, a scheme, a
port and a path; and wg0.conf renders with AllowedIPs on the network rather than
the host. Step 2 unticked — it wants all five earlier sections showing up in the
console with real timestamps, and that needs a hub.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reading ConfigureProxmox.php changed what this section does. The missing vmbr0
that killed the first paid order is a consequence of installing Proxmox ON TOP
OF DEBIAN — "only the ISO installer writes that bridge into
/etc/network/interfaces". This design installs from the ISO, so the bridge is
normally already there. The section verifies it instead of building it: exists,
has an address, and actually carries the default route. The old step checked
only the first of those, threw the result away, and its comment claimed it
recorded the absence.
When there is no bridge, it builds one, and only then is this the dangerous
section the plan calls it. Backup, five-minute rollback timer, switch, look,
cancel — and the timer is a systemd unit rather than a backgrounded sleep,
because a background job dies with its session and the session is exactly what
breaks when the switch goes wrong. It is set BEFORE the change; setting it after
would mean setting it at the moment the connection is already gone.
The three provider shapes differ in one line, and that line decides whether the
machine is still there afterwards. A routed single address needs pointopoint,
because its gateway is outside its own subnet and without it the kernel has no
route to reach it. Which shape applies is decided by asking `ip route get` about
the gateway rather than by doing subnet arithmetic here — same logic the kernel
will apply later.
The reachability check is named for what it is: inside-out. A true outside-in
proof would need a counterpart, and before the tunnel there is none. It is
enough because a bridge that takes the host off the network takes both
directions with it, and what it does not cover is precisely what the timer
covers.
One deviation from the plan, written into it: the nftables lockdown does NOT
belong here. SecureHostFirewall ran LAST in the old pipeline, deliberately, so
the tunnel had already carried every earlier step before SSH-to-the-world was
closed. Here it would sit BEFORE wireguard_joined and shut port 22 with no
handshake yet proven — the ordering that makes a host permanently unreachable.
It moves to Task 9, where it used to be. The self-rollback stays here and covers
the network change; the firewall keeps its manual emergency release, because a
firewall that reopens itself under failure is not a firewall.
The datacenter firewall comes over verbatim, including why `enable 1` alone is a
trap: the management ipset is seeded from the public subnet, so enabling it
blindly drops the tunnel address that every later step arrives on. Three
settings in order, then read back — a pvesh set returning 0 is not the same as
the value being in cluster.fw, and a write that lost quorum is exactly the case
where every customer VM's rules stay inert while onboarding reports success.
Verified without hardware: all four files dash-clean, and the generated
interfaces file is correct for subnet, routed (with pointopoint) and dhcp.
Step 2 unticked — the self-rollback has to be triggered on purpose on real
hardware, and the plan is right that without that it is an assertion.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PVE arrives with the image now, so this section accepts it rather than installs
it — but it needs every line of knowledge from InstallProxmoxVe.php to do that,
just pointed the other way.
The kernel is the claim, not the package. An installed proxmox-ve running on a
Debian kernel is exactly the half state RebootIntoPveKernel exists because of,
so uname is what gets checked.
The codename table now verifies a pairing instead of creating one. An image
whose PVE major does not match its Debian base was built wrong, and no amount
of fixing package sources heals a PVE compiled against a different libc — that
machine gets reinstalled with the right image. The other half of the table,
codename to PVE major, is written down here because the ISO route needs it and
the original file only had the suite side.
Removing the subscription repositories is the part that everything after this
depends on. A Proxmox image ships pve-enterprise, and without a subscription it
fails every apt-get update — after which no package install in any later
section succeeds. Both spellings are handled, because PVE 8 used one-line .list
files and PVE 9 moved to deb822, and ceph is in the list because the same trap
is set there a second time. The proof is a clean apt-get update at the end,
which is the only thing that tells removal apart from overwriting.
One thing changed after the first draft: when pveversion cannot be parsed, the
pairing check used to be skipped in silence. That is the fake that R19 records
as worse than no check at all, because it stops the next person from looking.
It now says "Paarung UNGEPRÜFT" in the report and in the log.
Verified without hardware: dash-clean; bookworm maps to PVE 8, trixie to PVE 9,
forky is refused; the pveversion parser reads 9 and 8 out of both real formats
and yields nothing for garbage; and against a copy of /etc/apt/sources.list.d
holding all five subscription files plus an unrelated one, exactly the five go
and the unrelated one stays. Step 2 unticked — there is no Proxmox here to
accept.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The official Proxmox ISO, an answer file baked into it, and QEMU running it
against the physical disks from the rescue system. Options were looked up
rather than recalled: prepare-iso takes --fetch-from iso, --answer-file and
--on-first-boot, and it names its own output, so the finished ISO is found by
glob afterwards instead of being handed an --output flag I would have invented.
validate-answer runs BEFORE the ISO is baked. An answer file with a typo drops
the installer into interactive mode, where it then hangs invisibly inside QEMU
until the hour runs out — the failure would arrive as a timeout and say nothing
about the cause.
The codename table comes over verbatim from InstallProxmoxVe.php, which the
platform plan deletes. It is used here to install the assistant into the RESCUE
system, so it keys off the rescue system's own codename. An unknown one aborts;
not "trixie is newest, so trixie", because the next Debian will be unknown too
and by then nobody is watching.
The ISO is checksummed against the mirror's SHA256SUMS. Booting an operating
system that could have become anything in transit is not a risk worth taking,
and a truncated download is enough to cause it — malice is not required.
The reboot carry-across is the gap this plan was amended for, and [first-boot]
takes exactly one executable. So that one file carries everything: the script,
its lib/, the progress file and the arguments, as a base64 archive in its own
body. That also avoids importing the fresh ZFS pool from the rescue system to
copy files into it.
Two bugs found by running it rather than reading it. The staging copy put the
work directory inside the state directory, so cp refused to copy a directory
into itself — and had it not refused, the gigabyte ISO would have been base64'd
into the hook that gets baked into that same ISO. The fix is not a better
exclude: the two directories have opposite lifetimes, so they now live in
different places, and a guard refuses loudly if anyone points them at each other
again. Structure beats a rule someone has to remember.
Verified without hardware: both files pass sh and dash. The generated answer
file parses as TOML with the expected disk-list, zfs.raid, zfs.arc-max,
first-boot and network.filter, and a 64-character root password. The codename
table accepts bookworm and trixie and refuses bullseye and forky. The shim comes
out at 26 KB with a 3 MB dummy ISO sitting in the work directory, proving it
stayed out. Run in a fresh debian:13-slim container it unpacks to
/opt/clupilot/bootstrap and /var/lib/clupilot, restores the progress file with
its original timestamp, has 700 on the directory and 600 on the arguments that
hold the WireGuard key, skips rescue_checked as already done, and fails cleanly
on the missing qemu. Step 2 stays unticked: a container is not a rescue system,
and nothing here has yet written to a disk.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four checks, all of them before a single byte is written, because this is the
only moment the script still has the option of not overwriting a disk.
The expensive one is "is this actually a rescue system". It demands POSITIVE
evidence rather than the absence of a counter-argument: a rescue system runs
from RAM, so its root is tmpfs, an overlay or a ramdisk, and an installed
system has a real partition there. A running pveversion is refused outright —
that is a hypervisor with customers on it. Mounted partitions of real disks are
refused too, because at that point the script cannot tell an empty machine from
someone's data.
There is deliberately no flag to skip the check. Such a flag gets used exactly
once, on the evening it should not have been. A rescue system this does not
recognise belongs in the runbook, not in a bypass.
The disk floor is written as arithmetic rather than a number, because the repo
has no threshold to borrow — plans live in the database, not in
config/provisioning.php. Template ~20 GB, Proxmox and its swap and backups
~20 GB, one smallest customer ~50 GB, rounded up to 100. It is explicitly not
capacity planning; HostCapacity and reserve_pct do that after registration.
This only turns away the machine the whole thing cannot fit on.
Network and clock come from one plain-HTTP HEAD, deliberately without TLS —
otherwise a wrong clock would be checking itself and would report a certificate
error instead of the time. Where `date -d` is missing (busybox), the clock is
NOT checked and the report says so: a check that silently waves things through
is worse than none, because it stops the next person from looking. That is the
same lesson R19 records about ->timezone(config('app.timezone')).
Verified here, in both directions where a direction existed. On this VM (root
on /dev/sda1, 80 GB, /dev/kvm present): refuses with three findings at once,
exits 1, writes nothing. In a debian:13-slim container (root on overlay, no
/dev/kvm): the overlay root is accepted as rescue-like and the missing
/dev/kvm is caught, which is the abort case the plan names. Against a receiver
serving a Date header two years off: 63074141 seconds of drift, refused, with
the exact `date -u -s` line to fix it. Step 2 stays unticked — none of these is
a rescue system on a dedicated server.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Argument parsing, the state directory, and the half of the script that knows
the way back to CluPilot. No sections are hooked up yet; main lists the nine it
will get, in the order from spec §7, so what is missing stays visible.
report() writes locally first and only then tries to send, because before
wireguard_joined there is no path at all — the lines queue up and flush_reports
delivers them later WITH THEIR OWN TIMESTAMPS. That last part is the whole
point: a twenty-minute install that arrives in one batch looks like one second
in the console, and nobody can see which section was slow.
Every network operation is explicitly caught. The script runs under `set -e`,
and a failing send is the normal case for the first five sections, not an
error — a report that aborts the run would be the diagnosis that kills the
patient.
What is sent had no agreed shape. The section keys were the only agreement
between the two plans; the envelope around them was not written down anywhere,
so it is written down here, at the top of report.sh, for the platform side to
read once. There is deliberately no "running" state: the console derives "open"
from the absence of a report (spec §7), and a third state would be a second
truth about the same thing.
The sent-marker is a line count in its own file rather than a flag rewritten
into progress.jsonl. That keeps the log append-only, so a power cut mid-write
costs at most a partial last line instead of a rewritten file — and after the
reboot in Task 3 that file is the only thing that remembers anything.
Verified here, without hardware: both syntax checks pass; two reports against
an unreachable CluPilot queue up without aborting and leave progress.sent at 0;
a receiver brought up afterwards gets both in one batch, carrying 17:28:55 and
17:28:57 rather than the 17:29:22 they arrived at; a message containing quotes,
a backslash, a tab and a newline survives as valid JSON; and section_done
counts a `done` as done and a `failed` as not. Step 2 of this task is NOT
ticked — none of that is a rescue system, and the plan is right to insist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nexxo