The owner caught me putting the archive on clupilot.cloud. That is the customer
zone, and looking into it turned up the same confusion already sitting in the
code — in the one place that matters most.
RegisterHostDns says "fsn-01.node.clupilot.com" in its own docblock. The
validation comment in Datacenters says it. ServicesTest writes it out verbatim.
The step itself built the name from config('provisioning.dns.zone') — the
CUSTOMER zone — so on this installation a host was actually called
fsn-01.node.clupilot.cloud. Three places asserting one thing and the code doing
another.
OfficialDomains explains why that matters and is worth not weakening: two
registrable domains by design, the company's for site, portal and console, the
instance zone for customer workloads. A Nextcloud is third-party software that
strangers sign into, and on the same registrable domain as the portal it shares
cookie scope with it. A host name in that zone does not break the separation,
but it puts it in question, and the next slip is more expensive.
So there is now a platform_zone, derived from APP_URL when unset, and
RegisterHostDns uses it.
My own archiveUrl was broken for a second reason. It fell back to url() when
APP_HOST is empty — and APP_HOST is empty on most installations, because empty
means "the portal answers on any hostname" and that is the default. Called from
the console, url() would have produced the CONSOLE hostname, and the line would
have 404'd on a machine that is not allowed to reach the admin area at all. It
takes the host from APP_URL now.
The script no longer guesses its own name. It used reverse DNS, then the tunnel
address, then a hard-coded clupilot.net — a third domain that appears nowhere
else in this project and was simply invented. PrepareBaseSystem has the same
invention. A guessed name does not stay guessed: it ends up in /etc/hostname, in
/etc/hosts, in every log line and in every certificate request the machine ever
makes. CluPilot knows the name because it just assigned it, so it passes --fqdn
and the script refuses without it. The value now also survives the reboot in the
arguments file, which it would not have.
The ACME contact moved to .com for the same reason it was wrong: the operator
does not live in the customer zone.
Open, and NOT decided here: the owner also wants the host to get a public DNS
record and a certificate on the .com name. RegisterHostDns deliberately writes
host names only into the tunnel's dnsmasq, and says why — publishing them hands
every scanner the internal subnet and roughly how many hosts sit behind it.
Nothing in the current design needs a public certificate for a host's own name;
Traefik serves customer domains, not this one. Reversing that is a security
decision and belongs to the owner, not to this commit.
1992 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The capacity park could not survive its own first poll. ReserveResources
polls every 120 s and declared a maxDuration of 60, and RunRunner measures
a step's budget from a started_at it deliberately does not reset on a poll
— so every re-entry was ruled timed out before the body ran, a timeout
consumes an attempt, and five of them failed a paid order in about six
minutes. The fourteen-day promise, the console's capacity queue and the
"go and buy a server" workflow were unreachable code. maxDuration is now
the whole park plus one poll interval, and a test drives a park through
the runner rather than calling execute() directly, which is why this was
invisible.
Every guest occ call ran as root. `docker compose exec` defaults to root
and the Nextcloud image's console.php exits 1 unless the caller owns
config/config.php, so all five call sites were failing on every instance.
There is now one builder, App\Support\NextcloudOcc, and a test refuses a
second: nothing in app/ may spell the invocation out by hand. deploy/
update.sh had learned this for our own container and nobody carried it
across.
RunAcceptanceChecks was terminal on the first no. certReachable() answers
false for a connect timeout as readily as for a missing certificate, so
one bad second ended a finished, certified Nextcloud as a failed order
with the instance released. The probes retry now and the run still fails
for good with the probe's own reason once the budget is spent; the two
facts a retry cannot change stay terminal.
The in-flight guard asked whether anything was running, not whether it
would do the work. App\Provisioning\WorkInFlight asks the second question,
reading the step lists the runner executes: a domain proven during a
restart is routed, and a storage pack booked during an address run is
delivered instead of charged monthly and forgotten.
The address pipeline ran its steps in the wrong order for the direction it
exists for. Nextcloud before the router in `address` and `plan-change`, as
in `customer`, because a run that got the certificate and then failed left
the customer's own domain serving an untrusted-domain error under a valid
certificate while the portal called it live. Safe in both directions; the
reasoning is written above the pipeline.
A stale guest_ip could never be corrected. ConfigureDnsAndTls compared the
hostnames but never the backend, and nothing re-read the address after the
build. It now records and compares `routed_backend`, and ConfigureNetwork
joins the `restart` pipeline — a cold boot is what moves a DHCP lease — and
asks for the address to be re-applied when the guest has actually moved.
Also: HetznerDnsClient::upsertRecord read only the first page of a zone
that pages at 100, so past a hundred records it created a second A record
for the same name and the cloud was up about half the time; the console's
retry left a live customer's order in `provisioning` for ever for any
maintenance pipeline; and a revived run's first pass burned an attempt on a
timeout that had already happened, because the clause written for it used
`??=`.
Seven test sites wrote config('provisioning.plans.*'), a key that has not
existed since the catalogue moved into the database. Every write was a
no-op, so each test proved something other than its name: the missing
template, the two snapshot regressions and the grandfathered-price scenario
are now actually constructed, ValidateOrder's reasons are asserted rather
than only its type, and the end-to-end run checks that the quota step
delivered something.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine defects in the host pipeline, every one of which either stopped an
install or let a host reach `active` while it could not do its job.
The install-stoppers:
- InstallProxmoxVe hard-coded `bookworm`. A server ordered today boots
Debian 13 trixie, so it added the PVE 8 repo and keyring to a trixie
base. The codename now comes from /etc/os-release, the suite and keyring
are looked up per release, and an unknown one fails naming what it found
instead of guessing. Keys land in /usr/share/keyrings with Signed-By
rather than in the deprecated trusted.gpg.d, where they would vouch for
every other repository on the machine too.
- keyLogin() dialled wg_ip whenever it was filled, but wg_ip is persisted
inside the allocation lock BEFORE the handshake is ever proven. Attempt
1 reserved the address and failed at the handshake; every later attempt,
including a console Retry, died connecting to a tunnel that did not work
— before reaching the only code that could repair wg0.conf. Recovery
meant nulling hosts.wg_ip by hand. SSH now uses the tunnel only once the
`wg_peer` breadcrumb proves a handshake succeeded, which also rescues
the same shape in RebootIntoPveKernel, ConfigureProxmox,
CreateAutomationToken and SecureHostFirewall. Not a hole: 22 on the
public IP is open until SecureHostFirewall runs, that step runs last,
and by then the tunnel is necessarily proven. And nothing validated the
hub key/endpoint, so renderConfig() would emit `PublicKey = ` — now
refused before anything is installed, allocated or written.
- `systemctl enable --now wg-quick@wg0 || wg-quick up wg0` brought the
interface up without the enablement, so the tunnel did not come back
after the step-6 reboot; on the next attempt both halves failed with
"wg0 already exists" and the step retried forever on a working tunnel.
The three states are handled separately now, and a changed wg0.conf is
actually applied instead of only written.
- RebootIntoPveKernel removed the Debian kernel with `|| true`, ignored
update-grub's exit status, and rebooted. A /boot that filled up during
the full-upgrade could leave a machine only the provider's console can
reach. The pve kernel image, its initramfs and update-grub's exit status
are all proven first; anything missing fails loudly and does not reboot.
- ConfigureProxmox ran `ip link show vmbr0` and threw the result away
(its comment claimed it recorded the absence; it recorded nothing), then
repeated a rm -f, and always advanced. Proxmox on top of Debian does not
create vmbr0, so onboarding reported `active` with no bridge. It now
fails with the default route in the message and deliberately does not
build the bridge over the primary NIC from a remote shell.
The things that were quietly inert:
- Proxmox applies a guest's firewall rules only while the firewall is
enabled at datacenter level, and it ships disabled — so applyFirewall()'s
"80/443 only" rules were inert on every customer VM. Enabling it blindly
is the opposite trap (input policy defaults to DROP, node firewall
defaults to on, and its management ipset is seeded from the PUBLIC
subnet, not the tunnel), so: policy ACCEPT and the node firewall off
first, master switch last, read back to confirm. nftables stays the one
owner of the host's input policy.
- role_privs lacked Sys.Modify, which POST /cluster/backup requires
(pve-manager's PVE/API2/Backup.pm), so RegisterBackup 403'd and failed
EVERY customer run. And `pveum role add … || true` only ever applied the
privilege list on the run that created the role — it converges now, above
the token short-circuit so a replay reaches it.
- The nftables ruleset dropped all ICMP and ICMPv6. On a real host that
breaks IPv6 outright once the neighbour cache expires and black-holes
large transfers through PMTUD. Both families accept the types they need,
policy drop stays, and a DHCP client reply is allowed so a rebind cannot
lose the IPv4 address.
- api_token_ref used the `encrypted` cast, i.e. APP_KEY, against
SecretVault's own written rule. One rotation would have made every host's
Proxmox token undecryptable at once. Moved onto SecretCipher with a
migration that leaves any value it cannot read exactly as it is and says
so, rather than destroying a credential that exists nowhere else.
- Every plan version points at template_vmid 9000 and nothing created or
verified it, so the first paid order died in CloneVirtualMachine after
the customer had paid. VerifyVmTemplate reports that during onboarding
instead. It does not build a template: what goes into the golden image is
a product decision.
Three tests that proved nothing, fixed: `ran(…emergency-open-firewall.sh)`
only ever matched the `chmod 700` above it, because putFile() is not
recorded — the script's contents are now asserted, including that nothing
in it reopens the firewall on a timer or in the background. The port policy
was tested one-sidedly, so a mutation adding `tcp dport { 8006 } accept`
passed the suite; the ruleset is now read as a list of what it opens to an
unrestricted source. And the end-to-end test says in writing that it proves
sequencing only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A booked storage pack was a row in `subscription_addons` and nothing else:
priced, frozen, charged every month, and delivered nowhere. Nextcloud's
quota came from `instances.quota_gb`, which is the package alone; the disk
was sized for the package alone; and on the one occasion a disk did grow —
a plan upgrade — the guest never saw it, because nothing stretched the
partition or the filesystem over the new space.
One authority for the allowance. StorageAllowance adds the package and the
booked packs, and everything that needs the figure asks it: the quota step,
the downgrade check, the portal, the console, the repair command. It is
DERIVED rather than stored — a column would have to be rewritten by booking,
cancelling, granting, a grant expiring and a plan change, and the day one of
those paths forgot it would be silently wrong in whichever direction costs
somebody money. `quota_applied_gb` keeps its own meaning: what the guest was
last actually told, which is how a machine with the allowance enforced is
told from one where the figure has only ever been a row in our database.
Buying it delivers it. BookAddon asks ApplyStorageAllowance for a run on
booking AND on cancellation; the new `storage` pipeline grows the virtual
disk to the allowance plus the package's own overhead (read off the package
— 20/40/50/100 GB on the four catalogue packages — never a ratio invented
here), then GrowGuestFilesystem makes the guest see it, then the quota is
applied. No cold boot is involved: the data disk is scsi0 and Proxmox's
resize on a running guest is a qemu block_resize, so the capacity reaches
the guest while it runs. The new step rescans, growparts and grows the
filesystem with the tool its type actually needs — ext2/3/4, xfs, btrfs —
and fails loudly on one it does not know rather than handing it to resize2fs
and hoping. Idempotent end to end: NOCHANGE from growpart is not a failure,
and every tool here exits 0 when there is nothing left to grow.
The same step now runs in the plan-change pipeline, where an upgrade used to
stop one step short of the customer.
A downgrade blocked by data gets a way out. The block stays — it is correct
— but the check now measures against the target package PLUS the packs the
customer already owns, and reports the numbers behind the refusal: what is
stored, what would be allowed, how much has to go, and how many packs would
cover it instead. The portal offers both routes: book exactly those packs
(confirmed in a modal, R23), or delete data and have the fill level measured
on demand rather than waiting for the nightly sampler. That reading is taken
by DiskUsageProbe, which CollectInstanceTraffic now uses too, so there is one
notion of "how full is it" and not two.
A plan change keeps booked packs. They were paid for separately and have
nothing to do with which package the customer is on; the new package's disk
and quota are sized with them included.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A paid order that no host has room for used to FAIL at the reservation
step. The customer has paid by then, so that turned a sale into an
incident and left somebody holding money against nothing. The estate is
finite and booked thick, so "no host has room right now" is an ordinary
Tuesday — it means a machine has to be bought, which takes days.
So the order waits instead. poll(), not retry(): the wait is measured in
days and must not eat the run's attempt budget. It only fails after
PARK_DAYS (14) — long enough to buy, rack and onboard a server over a
weekend, short enough that a forgotten order surfaces instead of waiting
for ever.
Then say so, in the same words on both sides of the payment:
- The price sheet marks each package "wird sofort ausgeliefert" or
"Bereitstellung in 2-3 Werktagen", read from free capacity rather
than written into the page.
- The customer's own overview says their cloud is being prepared while
it is parked, instead of a stepper standing still for two days under
"wird eingerichtet" — which reads as broken.
- The console front page shows the same per-package answer, so an
operator can see what visitors are being promised.
And a capacity page for the decision that follows: who is waiting, what
the roomiest host still has, what the queue needs ("3x Start, 1x
Business" - the LARGEST parked package sets the minimum machine, because
an instance lives on one host), and what such a machine costs today from
the provider's live list. It orders nothing: buying a server is a
contract, and a bug in a capacity calculation must not sign one.
The operator can also say where each parked order goes. A pin is
honoured or it waits — never quietly redirected — because placing it
elsewhere answers a different question than the one they answered, and
they would find out from the finished instance.
Two things this shook out:
- `current_step` is an INDEX into the pipeline, not the step's key, and
it is cast to integer. `where('current_step', 'reserve_resources')`
reads as correct and matches step 0 of EVERY pipeline instead. Both
lookups go through HostCapacity::parkedRuns() now, which resolves the
index and filters by pipeline.
- The auction feed sends `hdd_hr` as a list, not a string. Casting it
printed "Array" into the console and raised a warning that took the
page down with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three places where the product said one thing and did another.
**Nothing could restart a virtual machine.** ProxmoxClient had startVm and no
way to stop anything, so `restart_required_since` — set by every plan change
that grows a running guest — could only ever be cleared by a resize step that
happened to find the machine stopped, which nothing in the product could
arrange. A paid upgrade's cores and RAM could stay unreached for the life of the
contract, and the cloud page's "Neu starten" was a toast.
There is now a `restart` pipeline: shutdown, start, wait for the agent, confirm.
A shutdown and a start rather than a reboot, because only a cold boot makes qemu
read the VM definition again — a reset would take the machine round and bring it
back exactly as small as it was. The shutdown is a REQUEST, bounded at ten
minutes, and there is no escalation: ProxmoxClient deliberately does not expose
Proxmox's forceStop, because the guest is a Nextcloud and cutting power to a
database mid-write to apply a CPU change trades an inconvenience for a restore
from backup. A guest that ignores ACPI for ten minutes fails the run instead, and
a person decides what happens to it.
`restart_required_since` is cleared by the last step, from what the RUNNING guest
reports (`cpus`/`maxmem`) — not by the button, and not by the run merely
finishing. A machine that comes back smaller than it was sold fails the run.
Started by App\Actions\RestartInstance, which re-checks authorisation itself
rather than trusting a hidden button: an operator holding the new
`instances.restart` may restart anything, a portal user their own machine and
nothing else. Refused while another run is in flight against the same order.
Confirmed in a modal on both sides (R23), addressed by uuid (R11).
**Existing instances had no storage quota.** ApplyStorageQuota reaches new and
changed machines; everything built before it kept the whole disk whatever it had
paid for, and nothing recorded which was which. `instances.quota_applied_gb` is
now what the guest was actually told, written by the step only after the occ call
succeeds — so "sold" and "enforced" are separate facts and can be compared.
`clupilot:apply-quotas` sweeps the estate: one `quota` run per instance through
the same step the pipelines use, skipping what it should and saying why, a no-op
once an allowance is enforced, and inert under --dry-run. Deliberately NOT
scheduled — a nightly sweep would quietly cover for the pipeline step the day it
stopped working, which is how the original hole survived this long.
**A Traefik router was never torn down.** TraefikWriter::remove() had no caller
anywhere. What was missing was the moment: ConfirmCancelPackage wrote a date into
`service_ends_at` and nothing ever went back to it, so every route this platform
ever wrote was written for good — pointing at a guest address the host is free to
reassign. `clupilot:end-due-services` (hourly) keeps that appointment through
App\Actions\EndInstanceService. "Ended" means cancellation_scheduled AND
`service_ends_at` passed, never the day a cancellation is scheduled: that
customer has paid to the end of the term and is working in it. The DNS record
goes with the route — it is in our zone, pointing at a host that serves other
people, which is the shape of every subdomain takeover there has ever been. The
customer's own domain is in their zone and is not ours to touch; the virtual
machine is left alone, because the cancellation flow promises a data export first
and destroying disks is not this change's decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The platform books thick: a package reserves its whole disk_gb — the sold
quota plus the machine's own overhead — on ONE host at placement. That is the
right choice for a small operator, because nothing can quietly overfill. It
has one consequence nobody was being told about: the catalogue goes on
offering a package no host has room for, and the first anyone hears of it is a
PAID order failing at the reservation step.
On the machine this was found on the numbers are stark. 2 x 512 GB NVMe in a
mirror is about 400 GB usable; Start reserves 120, Team 540, Business 1050,
Enterprise 2100. Three of the four packages on sale cannot be placed at all,
and the fourth fits three times.
So the console says so now, per package, with the shortfall — "Team needs 540
GB, the roomiest host has 260" — because "buy a server" is not the decision.
"Buy a server big enough for what I am selling" is.
Two things it deliberately does not do. It does not warn on an installation
with no host at all: that is a machine nobody has onboarded yet, it has its
own notice, and without the guard a fresh console greeted its operator with
one warning per package before they had done anything. And it does not take
the overview down when the catalogue cannot be read — a capacity warning does
not get to break the page somebody opens when something is already wrong.
The largest FREE HOST, never the sum: an instance goes on one machine, so two
hosts with 300 GB each cannot take a 540 GB package between them.
And when placement does fail, the log now names the shortfall instead of only
'no_capacity'.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PlanChange could say what a move would cost and whether it was allowed, and
that was all it could do. Billing::purchase() wrote an upgrade order and
nothing ever consumed it: same snapshot, same machine, same quota. A customer
could pay for a bigger package and receive nothing.
ApplyPlanChange is now the single place a change lands — it moves the contract
onto the target's current version, writes one register row, settles the custom
domain, and starts a run that resizes the machine. Applying the same order
twice is a no-op, enforced by a unique event key rather than by a check two
callers could both pass.
Two things a plan change must not do, and now does not. A disk is never shrunk
— Proxmox cannot, so the QUOTA shrinks instead, which is what was sold anyway.
And a live machine is never rebooted as a side effect: cores and RAM are
written, and where they need a restart the instance says so where an operator
and the customer can both see it.
The storage allowance also joins the build pipeline. ApplyStorageQuota was
written for plan changes and ran only there, so a brand-new customer still got
no quota at all — quota_gb reached the instance row and stopped, and every
package delivered the whole disk. The test guarding that pipeline only counted
its steps, which is how a list missing the one step that makes a package's
storage real stayed green for its whole life. It now names the step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A verified custom domain was reported as the customer's address by
Instance::address(), by the portal and by the credentials mail while
nothing on the platform routed it: the Traefik router's rule was
hard-coded to {subdomain}.{zone}. The address a customer was handed
answered nothing, and a withdrawn domain stayed in Nextcloud's
trusted_domains forever, because the only thing that ever wrote either
was the initial provisioning run.
- TraefikWriter::write() takes a LIST of hostnames under one stable
router name — the platform address always, the verified domain as
well when there is one. One file per instance, so a withdrawal is a
rewrite rather than a second thing somebody has to remember.
- ConfigureDnsAndTls records the custom domain's certificate instead of
waiting on it: it depends on an A record in the customer's own zone,
which may never appear, and must never fail a run. The platform
address keeps its 840s deadline. instances.domain_cert_ok tells
"proven" apart from "answering", and the portal now says which.
- ConfigureNextcloud deletes trusted_domains 2 when there is no verified
domain, so a withdrawn one stops being trusted.
- New `address` pipeline (those two steps) plus ReapplyInstanceAddress,
which starts one against the order and refuses to start a second while
any run is in flight. The route is rewritten when the hostname list
differs from what the router carries — not on route_written, which
would short-circuit exactly the case a re-apply exists for.
- Triggered where the address changes: verification flipping either way
in clupilot:verify-domains, the customer's own domain page, and
CustomDomainAccess::deactivate() on a package downgrade.
- A maintenance run no longer condemns its subject: an address run that
failed used to mark the order failed and release the live instance
with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things, all from trying to actually use the incident page.
── Where the later entries go ───────────────────────────────────────────────
The form only ever creates the first one, and nothing said so — so it looked
as though the stage had to be chosen up front and there was only a dropdown
for the impact. The field now says what it becomes ("wird untersucht") and
where the rest are added: on the incident itself, once it exists.
── Deleting ────────────────────────────────────────────────────────────────
Asked for in order to test the thing at all, and right. A whole incident can
now be removed, with its entries, behind a confirmation in the product's own
dialog (R23).
Deliberately not the same as editing an entry, which there is still no way to
do: an update is a statement made at a time, and rewriting one afterwards is
what the record exists to prevent. Removing the whole incident is a different
act — it is how a test entry, or one published against the wrong service, is
taken back, and it leaves nothing half-standing behind. A test holds that
distinction so the delete button does not quietly become an exception to it.
── The hosts, which nothing was watching ────────────────────────────────────
`hosts.last_seen_at` drove the health dot in the console and was written
exactly once, at onboarding — RegisterCapacity stamped it and that was the
only write in the codebase. So every host read "offline" thirty minutes after
it was added, permanently, and the dot meant nothing. It is also why host
reachability could not appear on the status page: it was never being measured.
PingHosts asks every host's API the cheapest question it answers, once a
minute, and stamps the timestamp on success. Failure writes nothing on
purpose: absence IS the signal, and a second column counting failures would be
a second version of the same fact.
The status page has a fifth component for it. Counts only — never a host name,
never an address; the page is world-readable and the estate is not public
information. A host between five and thirty minutes silent reads as degraded,
not as healthy: "stale" is one of three answers healthState() gives and it is
not the good one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things.
── The update screen, still opening twice ───────────────────────────────────
Reported again on 1.3.9, and the cause was not the one fixed in 1.3.8. The
agent consumes the request file BEFORE it resolves the release — deliberately,
because update.sh may kill the shell and a request left in place would loop —
and writes `state: running` only once it has decided to go ahead. In between,
the request is gone and the status does not say running yet, so the endpoint
honestly answers "nothing is running". The watcher read that as "the run has
finished" and reloaded the page: overlay on the click, gone a poll later, 503
after it.
The overlay now closes only once the server has BOTH confirmed a run and then
stopped reporting it. Before the confirmation, silence means the agent has not
got there yet. Bounded at twenty polls so a request the agent refuses does not
leave the console covered forever.
── Custom domains, proven and re-proven ─────────────────────────────────────
`custom_domain` was a free-text field and everything downstream believed it:
the proxy served it, the certificate was issued for it, Nextcloud trusted it.
Anyone who pointed any hostname at the platform got somebody else's files
under their own name.
The proof is a TXT record at _clupilot-challenge.<domain> holding a token only
this instance has. Nothing is served until it has been read. Every reader now
goes through Instance::address(), which is the one place the decision is
made — `custom_domain ?: subdomain` was the hole, written out four times.
It is re-read every night at 03:40, because a token checked once can be taken
straight back out and a domain that later lapses keeps resolving here. Three
consecutive misses before a live domain is withdrawn: one failed lookup is a
nameserver having a bad minute, and withdrawing takes a working Nextcloud off
its own address. The domain and its token stay on the row so the customer can
put the record back rather than start over.
Changing the domain mints a NEW token. Reusing it would let somebody who once
verified example.com claim any other domain later without touching its DNS —
the old record is still sitting there and only the value is compared.
The domain is changeable at any time, and removable. Fixing it once set was
considered and rejected: adding or moving an address is a proxy entry, a
certificate and one line in trusted_domains.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SSH private key CluPilot deploys to every host is now stored through
SecretVault (ssh.private_key), preferred over CLUPILOT_SSH_PRIVATE_KEY(_PATH)
at all three real consumers — SshTraefikWriter and HostStep::keyLogin.
kuma.password/kuma.totp stay in .env: they authenticate a separate Python
container at boot, and nothing in this app reads them, so the vault would
have nothing to wire them to.
A new /admin/infrastructure page, backed by App\Support\ProvisioningSettings,
makes the non-secret deployment settings that used to force a shell —
DNS zone, WireGuard hub endpoint and public key, Traefik's dynamic-config
path, the SSH public key, and the monitoring bridge URL — visible and
editable from the console, each wired to every real consumer. WG subnet/hub
IP and Kuma's own connection details stay in .env: the compose file and a
separate container read those before this application ever does.
The password no longer travels through third-party mail servers or sits
forever in an inbox: it moves onto the instance (encrypted, same as
nc_admin_ref) and shows on the dashboard until the customer clicks
"Notiert", which deletes it for good.
Availability is now counted from our own monitoring answers. Uptime Kuma knows
the figure, but the REST bridge in front of it only answers "healthy, yes or
no", and waiting for the bridge to grow an endpoint would have left the panel
without the number indefinitely. The sync already asks on a schedule; counting
the answers gives a percentage that is ours and explainable.
An unanswered check is not a failed one — the counter only moves when a real
answer arrived, so our monitoring going down does not become the customer's
outage on the one figure they are asked to trust. An instance nobody has ever
checked shows no availability rather than 100 %.
The downgrade path existed nowhere. Going down is not the mirror of going up:
it can ask an instance to hold more than the target plan allows. Smaller plans
are now listed even when they cannot be taken, with the obstacle in the
customer's own numbers — "you have 31 users, this package allows 10" — because
a greyed-out button that does not name what is in the way is the thing people
ring about. Storage is checked against the last real reading rather than the
contractual allowance, or anyone who once bought a large plan could never leave
it. The limit is re-checked in the action: a rule enforced only in markup is
not enforced.
And the smaller corrections from this round:
- The cloud tab carried a hardcoded "B" as the instance's initial, and the PHP
and MariaDB versions, which tell a tenant nothing they can act on.
- "EU — Serverstandort im Angebot festgehalten" under a label already reading
"Serverstandort" was two sentences saying nothing. It is the country.
- Support promised a reply "binnen 4 Std." with nothing behind it. What is true
everywhere else is an answer on the same working day.
- The actions column appeared even when no row in the table had an action.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The panel could not show the storage ring or the transfer trend because nothing
measured them. instance_traffic keeps one row per billing period — right for an
allowance, useless as a series — and storage consumption was never sampled at
all, so the card could only ever state what was sold rather than what is used.
instance_metrics holds one row per instance per day, written by the traffic
collector on the visit it already makes. A second scheduler entry against the
same Proxmox API would have doubled the load and the failure modes for no gain.
Disk usage comes from `df` inside the guest rather than from Proxmox's own disk
figure, which reports the allocated image: that would show every customer a full
500 GB from their first day.
The rule the whole thing is built on: a reading that could not be taken is not a
reading of zero. The disk columns are nullable and left untouched when the guest
agent does not answer, so yesterday's figure stands instead of the ring dropping
to empty — which would tell a customer their data had vanished. Missing days
stay missing in the series rather than being filled with zeroes, which would
draw an outage that never happened. Both are covered by tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things the owner asked for, and one the review found underneath them.
The console leaves /admin. It has its own route file now, registered before
routes/web.php because once it has a hostname to itself it sits at the ROOT of
that host — where `/` and `/settings` also exist for the portal, and the first
matching route wins. Where it mounts is one decision in one place: `/` on the
console hostname where it has one to itself, `/admin` on any host otherwise.
Names stay admin.* in both modes, so nothing that builds a URL has to know.
Exclusivity is its own switch, not a consequence of having a hostname. The
first attempt made "this host is the console" follow from ADMIN_HOSTS, and a
development machine lists its own IP there so the console works without DNS —
which took the public site and the portal off that machine entirely. The
switch also registers the console on every listed hostname, canonical last, so
the alternates that exist as recovery paths keep working.
The status page moves out of /legal, where it sat beside the imprint and the
terms. Nothing about the current health of the platform is a legal document. It
is a real page now: portal, instances, provisioning and backups, each derived
from records, aggregate only, and a component with no signal reports "unknown"
rather than "operational".
That last rule is what exposed the real bug. monitoring_targets.status was
written once — 'up', at provisioning — and nothing ever updated it. Both the
console's notices and the new public page read it, so an outage would have been
published as healthy indefinitely. There is a sync job now, on a five-minute
schedule, and a checked_at column so a verdict can go stale instead of standing
forever.
The monitoring contract had to grow a third state for that to be honest.
isHealthy() answers true when no monitoring is configured and false when the
monitor is unreachable; recording that boolean would have published either a
fleet-wide all-clear nobody measured or a fleet-wide outage that was really one
broken monitor. health() returns null for both, the recorder leaves the old
verdict to go stale, and isHealthy() keeps its forgiving semantics for the one
caller that wants them — the provisioning acceptance check, which must not fail
a delivery because monitoring is not set up.
Backups are counted from the instances that need protecting rather than from
the backup rows that exist, so an instance with no schedule at all cannot be
missing from the arithmetic that declares the estate protected.
Also: the update panel polls itself, offers the button only when there is
something to install, and opens the log while a run is in progress.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pipeline re-resolved config('provisioning.plans') by order.plan, so the
subscription snapshot protected a customer's price but not their machine:
shrinking a plan resized an existing customer's VM on its next run. Nothing
created a subscription either, so closing this meant opening the contract at
purchase and pointing provisioning at it.
- OpenSubscription freezes the catalogue onto a subscription when a checkout
is paid; StartCustomerProvisioning calls it inside the order transaction.
- CustomerStep::plan() reads the frozen snapshot. ValidateOrder and
ReserveResources fail closed with no_subscription rather than falling back
to the catalogue, which is the bug itself.
- template_vmid joins the snapshot so a re-clone cannot pick up a blueprint
published after the sale. Deliberately outside FROZEN: it is how we build
the machine, not a term the customer is owed, and a dead template must be
replaceable without cancelling a contract.
- TrafficMeter reads the allowance off the contract too — cutting a plan's
traffic was otherwise enough to start throttling someone who bought more.
- The migration backfills contracts for orders that already bought something,
reconstructed from what was actually delivered where an instance exists,
and adopts an existing order-less contract instead of opening a second.
Orders paid in a currency the catalogue cannot price get none, matching the
checkout path.
price_cents stays the catalogue's NET price, which is what PlanChange
prorates against — not Order::amount_cents, which holds Stripe's GROSS total.
Reconciling the two belongs to the proof register and Stripe (phases 4/5).
Also pins STRIPE_WEBHOOK_SECRET blank in phpunit.xml: the operator's real
secret was reaching the suite from .env and rejecting every unsigned test
payload, which is why 7 webhook tests failed before any of this.
Verified in the browser: with team traffic cut from 3000 to 500 GB in the
catalogue, the customer's portal still shows 3 TB.
373 tests green. Codex review clean after three rounds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two from Codex:
- The job accepted any stored instance with a host. A closed one's VMID can have
been reused on the same machine, and the reset would then land on another
customer's VM. It now requires a live instance, and the action is not offered
for anything else.
- An unreachable Proxmox or guest agent throws rather than returning an exit
code, so nothing was ever written to the handoff and the modal polled forever.
The throw is caught, and a failed() handler covers anything that still escapes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Impersonation borrows the customer's portal session; this is access to their
installation, which is a different thing and now a different button.
Nextcloud has no passwordless admin jump, so this does the only honest thing it
offers: it resets OUR managed admin account inside that installation and hands
the credentials over once. The customer's own accounts are untouched, and the
next request sets a new password again.
- New capability instances.adminlogin, Owner and Admin only — stronger than
impersonation, because it hands over control rather than a session.
- The operator's own password is required every time, rate-limited: taking over
a customer's installation is not something an unattended browser should manage
on its own.
- The reset runs on the provisioning worker (it owns the tunnel and the Proxmox
credentials); the console polls a handoff token, so the credentials never
enter a Livewire snapshot.
- A silent instance is reported as such instead of appearing to succeed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Once a key is free, someone can legitimately create a new access with it. The
forced revocation looked the key up and would have deleted that person's access
and pulled their key off the hub. It now acts only on an adoption artifact — a
system peer with no owner and no creator — and leaves anything else alone.
Re-issuing also threw when VPN_CONFIG_KEY was missing, which is exactly the
situation the console tells people to fix by re-issuing. It now hands the new
config over once instead of storing it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex found that a sync landing between the key swap and its removal would see
the old key as unknown. Writing the test showed something worse than the
duplicate he predicted: the adoption cannot even happen — the address still
belongs to the live access — so the insert violated the unique index and took
the ENTIRE reconciliation down with it. One stale key would have stopped every
peer's state from updating.
The sync now recognises that case and queues the removal instead of adopting,
and a key with no row of its own is revoked with force: it can only ever be
removed, never kept, so the removal cannot be talked out of it by a row that
should not exist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A registration finishing while the purge waited on the run locks wrote
dns_record_id after the purge had already loaded the host, so it skipped the
deletion and then deleted the row holding the only id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The failure may be a lost response to a request that did create the record.
Advancing straight away stored no id, so PurgeHost could never remove it and the
host's management address stayed published. The upsert is idempotent, so a retry
recovers the id; only after the attempts run out does the onboarding continue
without a name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two legacy codes can normalise to the same label (eu_west and eu-west), and
separate counters then handed both of them eu-west-01 — a unique-constraint
failure inside the reservation, which blocks onboarding rather than being a DNS
problem the step can shrug off. Lock, counter and the in-use check all key off
the label now.
(The helper was also called label(), which HostStep already declares as the
step's display name — renamed.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
alpha_dash accepted eu_west, -edge and edge-, none of which are valid DNS
labels — every host in such a datacenter would have failed registration and,
because DNS is non-fatal, silently ended up without a name. The console now
requires a proper label, and the step normalises anything created before that
rule so existing rows still get a usable name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Swallowing it and carrying on deleted the row that held the only reference to
that record, publishing the machine's management address for good. The purge now
fails instead — and is retried, since it re-reads the host and its other steps
are idempotent — so a broken DNS provider becomes a visible failed job rather
than a silent leak.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex was right that my never-reuse claim did not hold:
- The lock was released with only a candidate in hand, so two concurrent
onboardings in one datacenter could pick the same name and overwrite each
other's record. The name is now persisted on the host while still locked.
- The number was derived from the hosts that happen to exist, so removing the
highest one handed that number straight back out — a cached name would then
resolve to a different machine. The counter is stored per datacenter, floored
by what is actually in use so a wiped settings store cannot reissue live names.
- PurgeHost deleted the row that carried the record id, leaving the machine's
management address published with nothing left to clean it up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fsn-01.node.clupilot.com, numbered per datacenter and never reused — removing a
host must not renumber its neighbours onto its name, so the number is stored
rather than derived.
The record points at the host's WireGuard address, not its public IP: the name
exists so an operator can reach a host by name over the VPN, and publishing a
Proxmox host's public address would hand every scanner a target, which is the
one thing this network design avoids.
DNS is convenience, not a prerequisite: a failure logs an event and lets the
onboarding finish rather than stranding a host that is otherwise ready.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Defaulting missing netin/netout to zero reset the baseline, so the next real
sample added the entire cumulative counter again — a customer throttled for
traffic that was counted twice. Such a sample is skipped and logged now.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- update.sh restarted the long-lived services before clearing the caches, so
they booted from the old cached config and kept it for the life of the
process — the release's configuration never reached the workers.
- install.sh put the Gitea token in the clone URL, which is visible in the
process list to every local user while the clone runs. It goes through a
short-lived askpass helper now.
- The collector's uniqueness lock expired at exactly the collection interval,
so a delayed queue could run two collectors on the same baseline and count
the same delta twice — throttling a customer for traffic they never used.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tying the release to the creation of the new month's row meant a single failed
call was never retried: the row exists from then on, and the current period
looks unthrottled, so nothing could detect the stale NIC limit. A customer would
have stayed slow for a whole month they had paid for. Checked on every sample
now, with a test that makes the first release fail.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two from Codex:
- A throttled instance kept its NIC limit into the new month: the new period
row starts unthrottled, so neither branch of enforce() fired and nothing ever
took the limit off. A customer would have stayed slow through a month they
had already paid for.
- PublicSiteGate exempted livewire/* wholesale. That endpoint is shared by the
console and the portal, so a signed-in customer could keep driving portal
components — billing included — while the portal was supposed to be offline.
Operators pass the normal check anyway, which is all the console needed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Customers can now see what they have used and what is left, and the service
slows down rather than stopping when the allowance is gone — a slow Nextcloud
gets a top-up, a dead one gets a cancellation.
- instance_traffic keeps one row per instance per month. Proxmox counters are
cumulative since the VM last started, so usage is the difference between two
samples, and a counter that went backwards means a restart, not a refund.
- Outbound is what counts: inbound is free at our providers and egress is what
Hetzner's 20 TB per server applies to.
- CollectInstanceTraffic samples every 15 minutes, warns once per threshold
(80/95 %) rather than on every run, and at 100 % limits the VM's NIC via
Proxmox — released again as soon as the customer tops up or the month rolls
over.
- The dashboard gets a full-width band (it throttles the service, so it is not
a tile among tiles) with the top-up offer right next to the warning.
- Bytes are formatted in SI units now: allowances are computed in SI, so
dividing by 1024 made a 1000 GB plan read as "931 GB".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex spotted the migration labelling sync-adopted peers as staff with no
owner, which breaks the invariant the same migration introduces. The live sync
had the identical hole — it never set kind at all, so every adopted peer landed
on the 'staff' default. Both now use system for a peer nobody is behind, and
promote it to host once the link is known.
.env.example carried ADMIN_HOSTS twice; phpdotenv keeps the first, so a fresh
checkout would have got the list without localhost and 404'd its own console.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The FK only nulled host_id, so the adopted peer stayed enabled-but-absent: shown
as "wird angewendet" forever, and re-enabling it would have put the deleted
host's key and address back on the hub. PurgeHost now tombstones that row, which
hands it to the same revocation machinery as everything else.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- A host's wg_pubkey may not be in vpn_peers yet, so the duplicate check passed
and an operator could create an access with it. ApplyVpnPeer would then run
wg set with a freshly allocated address, rewriting that host's allowed-ip and
cutting CluPilot's management tunnel to a live machine. Creation now rejects
keys already held by a host, checked under the allocation lock.
- A sync landing between addPeer() and the step storing wg_pubkey adopted the
peer as "unknown", and a later sync filled in host_id but left the name, so
the page showed that host as unknown forever. The name is now filled in when
the link becomes known — only for sync-adopted rows, so an access an operator
named keeps its name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard and RemoveWireguardPeer mutate the hub too, and were not
taking the lock, so a sync could still read the interface around them and
write stale state. Both now hold wireguard:hub, which is what makes the
read/mutation guarantee actually hold.
- The duplicate-key check ran before the allocation lock, so two concurrent
creations with the same key both passed it and the loser hit the unique index
as an unhandled 500. The check moved inside the lock, with the index kept as
a caught backstop for any writer that does not take it — reproduced in a test
by inserting a colliding row from within the allocation call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A sync could read the hub snapshot just before ApplyVpnPeer removed a peer and
purged its tombstone, then adopt what it had seen as a fresh enabled access —
restoring a revoked one. Both jobs now hold Cache::lock('wireguard:hub') around
read-decide-mutate, so a reconciliation can never straddle a removal. Retry
removals are dispatched after the lock is released, because the sync queue
driver runs them inline and they take the same lock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
If an add job was retried after the access had been revoked and its tombstone
purged, it found no row and trusted its own enabled=true payload — re-adding a
key that no longer had a row anyone could revoke it with. A missing row can only
mean the access is gone, so the captured payload may now remove but never
enable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two lifecycle holes Codex found in the new VPN management:
1. Delete removed the row before the hub knew. If the removal job was delayed
or failed, SyncVpnPeers saw a peer it did not recognise and adopted it back
as a live access — silently restoring what an operator had just revoked, and
with no row left to revoke it again. Deletion is now a soft-delete
tombstone: the row survives until the hub confirms the peer is gone, the
sync retries the removal instead of adopting, and the tombstone is purged
only once the peer is really absent. It also keeps the key and the tunnel
address reserved meanwhile, so neither is handed out twice.
2. ApplyVpnPeer applied the state captured at dispatch. A retried block job
could therefore undo a later unblock, and nothing would repair it — the sync
only observes state, it never re-applies intent. The job now resolves the
current desired state from the row and treats its payload as a fallback for
the case where the row is already gone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The console can now create, block and remove VPN accesses, and shows who is
actually connected — traffic counters, source endpoint and last handshake.
Design notes:
- The web container has no wg0, so the page never talks to WireGuard. Live
state is copied in by SyncVpnPeers on the provisioning queue (the only worker
whose container owns the interface); the page polls the database and merely
nudges that job, throttled so many open tabs cannot flood the queue.
- enabled (operator intent) and present (observed on the hub) are stored
separately, so a sync can never quietly undo a block and drift stays visible
as "wird angewendet".
- The sync adopts peers the host pipeline registered, naming them after their
host — otherwise "who is on the VPN" would have blind spots.
- Keypairs are generated in PHP via libsodium rather than shelling out to
wg genkey (no interface in this container, and it keeps generation testable).
The private key is shown once and never stored.
- allocateIp() now also considers vpn_peers, which would otherwise be handed a
tunnel address a host already holds.
- vpn.manage is a new capability held by Owner/Admin only, and it hides the nav
entry too — a VPN access reaches the management network.
Verified end to end against the real hub, not just mocks: created an access in
the browser, connected with the generated config, watched the console flip to
"Verbunden" with real counters, then blocked it and confirmed the peer was
gone from wg0 and the client could no longer handshake.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- MONITORING_ATTEMPTS is capped by the run's max_attempts, so a large value can
no longer burn the retry budget and fail the very provisioning the degradation
exists to protect (covered by a new test at the budget boundary)
- /health is now dependency-free (200 in ~2ms with Kuma absent, verified: the
container healthcheck stays healthy); Kuma state is exclusively in /ready
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- attempt is 0-based, so compare one-based: ATTEMPTS=2 really means two tries
- bridge accepts add_monitor returning {monitorID}|{id}|int (v1.2.1 returns a
dict — re-verified: first create on a fresh Kuma is HTTP 200 with the id)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.
Verified end-to-end against a real Uptime Kuma 1.x:
create -> id, create again -> same id (idempotent), list, status, delete
and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.
Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
monitor legitimately reports 'pending' — confirmed in the e2e run); it records
the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Clone recovery: once the lock clears but the task ref is lost, destroy the
possibly-incomplete VM and re-clone instead of advancing onto it.
- applyFirewall clears existing rules before adding, so retries don't accumulate
duplicate Proxmox firewall entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Nextcloud: parse occ status JSON (installed + not in maintenance).
- Admin: query occ user:info for the account.
- Monitoring: MonitoringClient::isHealthy checks the provider, not the local row.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo
Claude