A booked storage pack was a row in `subscription_addons` and nothing else:
priced, frozen, charged every month, and delivered nowhere. Nextcloud's
quota came from `instances.quota_gb`, which is the package alone; the disk
was sized for the package alone; and on the one occasion a disk did grow —
a plan upgrade — the guest never saw it, because nothing stretched the
partition or the filesystem over the new space.
One authority for the allowance. StorageAllowance adds the package and the
booked packs, and everything that needs the figure asks it: the quota step,
the downgrade check, the portal, the console, the repair command. It is
DERIVED rather than stored — a column would have to be rewritten by booking,
cancelling, granting, a grant expiring and a plan change, and the day one of
those paths forgot it would be silently wrong in whichever direction costs
somebody money. `quota_applied_gb` keeps its own meaning: what the guest was
last actually told, which is how a machine with the allowance enforced is
told from one where the figure has only ever been a row in our database.
Buying it delivers it. BookAddon asks ApplyStorageAllowance for a run on
booking AND on cancellation; the new `storage` pipeline grows the virtual
disk to the allowance plus the package's own overhead (read off the package
— 20/40/50/100 GB on the four catalogue packages — never a ratio invented
here), then GrowGuestFilesystem makes the guest see it, then the quota is
applied. No cold boot is involved: the data disk is scsi0 and Proxmox's
resize on a running guest is a qemu block_resize, so the capacity reaches
the guest while it runs. The new step rescans, growparts and grows the
filesystem with the tool its type actually needs — ext2/3/4, xfs, btrfs —
and fails loudly on one it does not know rather than handing it to resize2fs
and hoping. Idempotent end to end: NOCHANGE from growpart is not a failure,
and every tool here exits 0 when there is nothing left to grow.
The same step now runs in the plan-change pipeline, where an upgrade used to
stop one step short of the customer.
A downgrade blocked by data gets a way out. The block stays — it is correct
— but the check now measures against the target package PLUS the packs the
customer already owns, and reports the numbers behind the refusal: what is
stored, what would be allowed, how much has to go, and how many packs would
cover it instead. The portal offers both routes: book exactly those packs
(confirmed in a modal, R23), or delete data and have the fill level measured
on demand rather than waiting for the nightly sampler. That reading is taken
by DiskUsageProbe, which CollectInstanceTraffic now uses too, so there is one
notion of "how full is it" and not two.
A plan change keeps booked packs. They were paid for separately and have
nothing to do with which package the customer is on; the new package's disk
and quota are sized with them included.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things, all from trying to actually use the incident page.
── Where the later entries go ───────────────────────────────────────────────
The form only ever creates the first one, and nothing said so — so it looked
as though the stage had to be chosen up front and there was only a dropdown
for the impact. The field now says what it becomes ("wird untersucht") and
where the rest are added: on the incident itself, once it exists.
── Deleting ────────────────────────────────────────────────────────────────
Asked for in order to test the thing at all, and right. A whole incident can
now be removed, with its entries, behind a confirmation in the product's own
dialog (R23).
Deliberately not the same as editing an entry, which there is still no way to
do: an update is a statement made at a time, and rewriting one afterwards is
what the record exists to prevent. Removing the whole incident is a different
act — it is how a test entry, or one published against the wrong service, is
taken back, and it leaves nothing half-standing behind. A test holds that
distinction so the delete button does not quietly become an exception to it.
── The hosts, which nothing was watching ────────────────────────────────────
`hosts.last_seen_at` drove the health dot in the console and was written
exactly once, at onboarding — RegisterCapacity stamped it and that was the
only write in the codebase. So every host read "offline" thirty minutes after
it was added, permanently, and the dot meant nothing. It is also why host
reachability could not appear on the status page: it was never being measured.
PingHosts asks every host's API the cheapest question it answers, once a
minute, and stamps the timestamp on success. Failure writes nothing on
purpose: absence IS the signal, and a second column counting failures would be
a second version of the same fact.
The status page has a fifth component for it. Counts only — never a host name,
never an address; the page is world-readable and the estate is not public
information. A host between five and thirty minutes silent reads as degraded,
not as healthy: "stale" is one of three answers healthState() gives and it is
not the good one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things.
── The update screen, still opening twice ───────────────────────────────────
Reported again on 1.3.9, and the cause was not the one fixed in 1.3.8. The
agent consumes the request file BEFORE it resolves the release — deliberately,
because update.sh may kill the shell and a request left in place would loop —
and writes `state: running` only once it has decided to go ahead. In between,
the request is gone and the status does not say running yet, so the endpoint
honestly answers "nothing is running". The watcher read that as "the run has
finished" and reloaded the page: overlay on the click, gone a poll later, 503
after it.
The overlay now closes only once the server has BOTH confirmed a run and then
stopped reporting it. Before the confirmation, silence means the agent has not
got there yet. Bounded at twenty polls so a request the agent refuses does not
leave the console covered forever.
── Custom domains, proven and re-proven ─────────────────────────────────────
`custom_domain` was a free-text field and everything downstream believed it:
the proxy served it, the certificate was issued for it, Nextcloud trusted it.
Anyone who pointed any hostname at the platform got somebody else's files
under their own name.
The proof is a TXT record at _clupilot-challenge.<domain> holding a token only
this instance has. Nothing is served until it has been read. Every reader now
goes through Instance::address(), which is the one place the decision is
made — `custom_domain ?: subdomain` was the hole, written out four times.
It is re-read every night at 03:40, because a token checked once can be taken
straight back out and a domain that later lapses keeps resolving here. Three
consecutive misses before a live domain is withdrawn: one failed lookup is a
nameserver having a bad minute, and withdrawing takes a working Nextcloud off
its own address. The domain and its token stay on the row so the customer can
put the record back rather than start over.
Changing the domain mints a NEW token. Reusing it would let somebody who once
verified example.com claim any other domain later without touching its DNS —
the old record is still sitting there and only the value is compared.
The domain is changeable at any time, and removable. Fixing it once set was
considered and rejected: adding or moving an address is a proxy entry, a
certificate and one line in trusted_domains.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SSH private key CluPilot deploys to every host is now stored through
SecretVault (ssh.private_key), preferred over CLUPILOT_SSH_PRIVATE_KEY(_PATH)
at all three real consumers — SshTraefikWriter and HostStep::keyLogin.
kuma.password/kuma.totp stay in .env: they authenticate a separate Python
container at boot, and nothing in this app reads them, so the vault would
have nothing to wire them to.
A new /admin/infrastructure page, backed by App\Support\ProvisioningSettings,
makes the non-secret deployment settings that used to force a shell —
DNS zone, WireGuard hub endpoint and public key, Traefik's dynamic-config
path, the SSH public key, and the monitoring bridge URL — visible and
editable from the console, each wired to every real consumer. WG subnet/hub
IP and Kuma's own connection details stay in .env: the compose file and a
separate container read those before this application ever does.
Availability is now counted from our own monitoring answers. Uptime Kuma knows
the figure, but the REST bridge in front of it only answers "healthy, yes or
no", and waiting for the bridge to grow an endpoint would have left the panel
without the number indefinitely. The sync already asks on a schedule; counting
the answers gives a percentage that is ours and explainable.
An unanswered check is not a failed one — the counter only moves when a real
answer arrived, so our monitoring going down does not become the customer's
outage on the one figure they are asked to trust. An instance nobody has ever
checked shows no availability rather than 100 %.
The downgrade path existed nowhere. Going down is not the mirror of going up:
it can ask an instance to hold more than the target plan allows. Smaller plans
are now listed even when they cannot be taken, with the obstacle in the
customer's own numbers — "you have 31 users, this package allows 10" — because
a greyed-out button that does not name what is in the way is the thing people
ring about. Storage is checked against the last real reading rather than the
contractual allowance, or anyone who once bought a large plan could never leave
it. The limit is re-checked in the action: a rule enforced only in markup is
not enforced.
And the smaller corrections from this round:
- The cloud tab carried a hardcoded "B" as the instance's initial, and the PHP
and MariaDB versions, which tell a tenant nothing they can act on.
- "EU — Serverstandort im Angebot festgehalten" under a label already reading
"Serverstandort" was two sentences saying nothing. It is the country.
- Support promised a reply "binnen 4 Std." with nothing behind it. What is true
everywhere else is an answer on the same working day.
- The actions column appeared even when no row in the table had an action.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The panel could not show the storage ring or the transfer trend because nothing
measured them. instance_traffic keeps one row per billing period — right for an
allowance, useless as a series — and storage consumption was never sampled at
all, so the card could only ever state what was sold rather than what is used.
instance_metrics holds one row per instance per day, written by the traffic
collector on the visit it already makes. A second scheduler entry against the
same Proxmox API would have doubled the load and the failure modes for no gain.
Disk usage comes from `df` inside the guest rather than from Proxmox's own disk
figure, which reports the allocated image: that would show every customer a full
500 GB from their first day.
The rule the whole thing is built on: a reading that could not be taken is not a
reading of zero. The disk columns are nullable and left untouched when the guest
agent does not answer, so yesterday's figure stands instead of the ring dropping
to empty — which would tell a customer their data had vanished. Missing days
stay missing in the series rather than being filled with zeroes, which would
draw an outage that never happened. Both are covered by tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things the owner asked for, and one the review found underneath them.
The console leaves /admin. It has its own route file now, registered before
routes/web.php because once it has a hostname to itself it sits at the ROOT of
that host — where `/` and `/settings` also exist for the portal, and the first
matching route wins. Where it mounts is one decision in one place: `/` on the
console hostname where it has one to itself, `/admin` on any host otherwise.
Names stay admin.* in both modes, so nothing that builds a URL has to know.
Exclusivity is its own switch, not a consequence of having a hostname. The
first attempt made "this host is the console" follow from ADMIN_HOSTS, and a
development machine lists its own IP there so the console works without DNS —
which took the public site and the portal off that machine entirely. The
switch also registers the console on every listed hostname, canonical last, so
the alternates that exist as recovery paths keep working.
The status page moves out of /legal, where it sat beside the imprint and the
terms. Nothing about the current health of the platform is a legal document. It
is a real page now: portal, instances, provisioning and backups, each derived
from records, aggregate only, and a component with no signal reports "unknown"
rather than "operational".
That last rule is what exposed the real bug. monitoring_targets.status was
written once — 'up', at provisioning — and nothing ever updated it. Both the
console's notices and the new public page read it, so an outage would have been
published as healthy indefinitely. There is a sync job now, on a five-minute
schedule, and a checked_at column so a verdict can go stale instead of standing
forever.
The monitoring contract had to grow a third state for that to be honest.
isHealthy() answers true when no monitoring is configured and false when the
monitor is unreachable; recording that boolean would have published either a
fleet-wide all-clear nobody measured or a fleet-wide outage that was really one
broken monitor. health() returns null for both, the recorder leaves the old
verdict to go stale, and isHealthy() keeps its forgiving semantics for the one
caller that wants them — the provisioning acceptance check, which must not fail
a delivery because monitoring is not set up.
Backups are counted from the instances that need protecting rather than from
the backup rows that exist, so an instance with no schedule at all cannot be
missing from the arithmetic that declares the estate protected.
Also: the update panel polls itself, offers the button only when there is
something to install, and opens the log while a run is in progress.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two from Codex:
- The job accepted any stored instance with a host. A closed one's VMID can have
been reused on the same machine, and the reset would then land on another
customer's VM. It now requires a live instance, and the action is not offered
for anything else.
- An unreachable Proxmox or guest agent throws rather than returning an exit
code, so nothing was ever written to the handoff and the modal polled forever.
The throw is caught, and a failed() handler covers anything that still escapes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Impersonation borrows the customer's portal session; this is access to their
installation, which is a different thing and now a different button.
Nextcloud has no passwordless admin jump, so this does the only honest thing it
offers: it resets OUR managed admin account inside that installation and hands
the credentials over once. The customer's own accounts are untouched, and the
next request sets a new password again.
- New capability instances.adminlogin, Owner and Admin only — stronger than
impersonation, because it hands over control rather than a session.
- The operator's own password is required every time, rate-limited: taking over
a customer's installation is not something an unattended browser should manage
on its own.
- The reset runs on the provisioning worker (it owns the tunnel and the Proxmox
credentials); the console polls a handoff token, so the credentials never
enter a Livewire snapshot.
- A silent instance is reported as such instead of appearing to succeed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Once a key is free, someone can legitimately create a new access with it. The
forced revocation looked the key up and would have deleted that person's access
and pulled their key off the hub. It now acts only on an adoption artifact — a
system peer with no owner and no creator — and leaves anything else alone.
Re-issuing also threw when VPN_CONFIG_KEY was missing, which is exactly the
situation the console tells people to fix by re-issuing. It now hands the new
config over once instead of storing it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex found that a sync landing between the key swap and its removal would see
the old key as unknown. Writing the test showed something worse than the
duplicate he predicted: the adoption cannot even happen — the address still
belongs to the live access — so the insert violated the unique index and took
the ENTIRE reconciliation down with it. One stale key would have stopped every
peer's state from updating.
The sync now recognises that case and queues the removal instead of adopting,
and a key with no row of its own is revoked with force: it can only ever be
removed, never kept, so the removal cannot be talked out of it by a row that
should not exist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A registration finishing while the purge waited on the run locks wrote
dns_record_id after the purge had already loaded the host, so it skipped the
deletion and then deleted the row holding the only id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Swallowing it and carrying on deleted the row that held the only reference to
that record, publishing the machine's management address for good. The purge now
fails instead — and is retried, since it re-reads the host and its other steps
are idempotent — so a broken DNS provider becomes a visible failed job rather
than a silent leak.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex was right that my never-reuse claim did not hold:
- The lock was released with only a candidate in hand, so two concurrent
onboardings in one datacenter could pick the same name and overwrite each
other's record. The name is now persisted on the host while still locked.
- The number was derived from the hosts that happen to exist, so removing the
highest one handed that number straight back out — a cached name would then
resolve to a different machine. The counter is stored per datacenter, floored
by what is actually in use so a wiped settings store cannot reissue live names.
- PurgeHost deleted the row that carried the record id, leaving the machine's
management address published with nothing left to clean it up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Defaulting missing netin/netout to zero reset the baseline, so the next real
sample added the entire cumulative counter again — a customer throttled for
traffic that was counted twice. Such a sample is skipped and logged now.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- update.sh restarted the long-lived services before clearing the caches, so
they booted from the old cached config and kept it for the life of the
process — the release's configuration never reached the workers.
- install.sh put the Gitea token in the clone URL, which is visible in the
process list to every local user while the clone runs. It goes through a
short-lived askpass helper now.
- The collector's uniqueness lock expired at exactly the collection interval,
so a delayed queue could run two collectors on the same baseline and count
the same delta twice — throttling a customer for traffic they never used.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tying the release to the creation of the new month's row meant a single failed
call was never retried: the row exists from then on, and the current period
looks unthrottled, so nothing could detect the stale NIC limit. A customer would
have stayed slow for a whole month they had paid for. Checked on every sample
now, with a test that makes the first release fail.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two from Codex:
- A throttled instance kept its NIC limit into the new month: the new period
row starts unthrottled, so neither branch of enforce() fired and nothing ever
took the limit off. A customer would have stayed slow through a month they
had already paid for.
- PublicSiteGate exempted livewire/* wholesale. That endpoint is shared by the
console and the portal, so a signed-in customer could keep driving portal
components — billing included — while the portal was supposed to be offline.
Operators pass the normal check anyway, which is all the console needed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Customers can now see what they have used and what is left, and the service
slows down rather than stopping when the allowance is gone — a slow Nextcloud
gets a top-up, a dead one gets a cancellation.
- instance_traffic keeps one row per instance per month. Proxmox counters are
cumulative since the VM last started, so usage is the difference between two
samples, and a counter that went backwards means a restart, not a refund.
- Outbound is what counts: inbound is free at our providers and egress is what
Hetzner's 20 TB per server applies to.
- CollectInstanceTraffic samples every 15 minutes, warns once per threshold
(80/95 %) rather than on every run, and at 100 % limits the VM's NIC via
Proxmox — released again as soon as the customer tops up or the month rolls
over.
- The dashboard gets a full-width band (it throttles the service, so it is not
a tile among tiles) with the top-up offer right next to the warning.
- Bytes are formatted in SI units now: allowances are computed in SI, so
dividing by 1024 made a 1000 GB plan read as "931 GB".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex spotted the migration labelling sync-adopted peers as staff with no
owner, which breaks the invariant the same migration introduces. The live sync
had the identical hole — it never set kind at all, so every adopted peer landed
on the 'staff' default. Both now use system for a peer nobody is behind, and
promote it to host once the link is known.
.env.example carried ADMIN_HOSTS twice; phpdotenv keeps the first, so a fresh
checkout would have got the list without localhost and 404'd its own console.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The FK only nulled host_id, so the adopted peer stayed enabled-but-absent: shown
as "wird angewendet" forever, and re-enabling it would have put the deleted
host's key and address back on the hub. PurgeHost now tombstones that row, which
hands it to the same revocation machinery as everything else.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- A host's wg_pubkey may not be in vpn_peers yet, so the duplicate check passed
and an operator could create an access with it. ApplyVpnPeer would then run
wg set with a freshly allocated address, rewriting that host's allowed-ip and
cutting CluPilot's management tunnel to a live machine. Creation now rejects
keys already held by a host, checked under the allocation lock.
- A sync landing between addPeer() and the step storing wg_pubkey adopted the
peer as "unknown", and a later sync filled in host_id but left the name, so
the page showed that host as unknown forever. The name is now filled in when
the link becomes known — only for sync-adopted rows, so an access an operator
named keeps its name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard and RemoveWireguardPeer mutate the hub too, and were not
taking the lock, so a sync could still read the interface around them and
write stale state. Both now hold wireguard:hub, which is what makes the
read/mutation guarantee actually hold.
- The duplicate-key check ran before the allocation lock, so two concurrent
creations with the same key both passed it and the loser hit the unique index
as an unhandled 500. The check moved inside the lock, with the index kept as
a caught backstop for any writer that does not take it — reproduced in a test
by inserting a colliding row from within the allocation call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A sync could read the hub snapshot just before ApplyVpnPeer removed a peer and
purged its tombstone, then adopt what it had seen as a fresh enabled access —
restoring a revoked one. Both jobs now hold Cache::lock('wireguard:hub') around
read-decide-mutate, so a reconciliation can never straddle a removal. Retry
removals are dispatched after the lock is released, because the sync queue
driver runs them inline and they take the same lock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
If an add job was retried after the access had been revoked and its tombstone
purged, it found no row and trusted its own enabled=true payload — re-adding a
key that no longer had a row anyone could revoke it with. A missing row can only
mean the access is gone, so the captured payload may now remove but never
enable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two lifecycle holes Codex found in the new VPN management:
1. Delete removed the row before the hub knew. If the removal job was delayed
or failed, SyncVpnPeers saw a peer it did not recognise and adopted it back
as a live access — silently restoring what an operator had just revoked, and
with no row left to revoke it again. Deletion is now a soft-delete
tombstone: the row survives until the hub confirms the peer is gone, the
sync retries the removal instead of adopting, and the tombstone is purged
only once the peer is really absent. It also keeps the key and the tunnel
address reserved meanwhile, so neither is handed out twice.
2. ApplyVpnPeer applied the state captured at dispatch. A retried block job
could therefore undo a later unblock, and nothing would repair it — the sync
only observes state, it never re-applies intent. The job now resolves the
current desired state from the row and treats its payload as a fallback for
the case where the row is already gone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The console can now create, block and remove VPN accesses, and shows who is
actually connected — traffic counters, source endpoint and last handshake.
Design notes:
- The web container has no wg0, so the page never talks to WireGuard. Live
state is copied in by SyncVpnPeers on the provisioning queue (the only worker
whose container owns the interface); the page polls the database and merely
nudges that job, throttled so many open tabs cannot flood the queue.
- enabled (operator intent) and present (observed on the hub) are stored
separately, so a sync can never quietly undo a block and drift stays visible
as "wird angewendet".
- The sync adopts peers the host pipeline registered, naming them after their
host — otherwise "who is on the VPN" would have blind spots.
- Keypairs are generated in PHP via libsodium rather than shelling out to
wg genkey (no interface in this container, and it keeps generation testable).
The private key is shown once and never stored.
- allocateIp() now also considers vpn_peers, which would otherwise be handed a
tunnel address a host already holds.
- vpn.manage is a new capability held by Owner/Admin only, and it hides the nav
entry too — a VPN access reaches the management network.
Verified end to end against the real hub, not just mocks: created an access in
the browser, connected with the generated config, watched the console flip to
"Verbunden" with real counters, then blocked it and confirmed the peer was
gone from wg0 and the client could no longer handshake.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard checks setup command results and only records the wg_peer
resource after the handshake verifies, so a transient setup failure re-runs
the full setup instead of getting stuck on the idempotent path.
- Host removal now deactivates immediately and queues PurgeHost, which deletes
runs under the runner lock (waiting out a long step) — no 15s LockTimeout error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- PhpseclibRemoteShell sets a command timeout (default 2000s) so apt
full-upgrade/install don't hit phpseclib's ~10s default and fail.
- Host removal dispatches RemoveWireguardPeer to the provisioning queue, which
runs in the worker that owns wg0 (the web container can't manage WireGuard).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Dedicated 'provisioning' queue connection (retry_after 2400) + worker
(--timeout=2100); AdvanceRunJob tries=1, timeout=2100 on that queue.
- Run lock TTL raised to 2100s so a long step can't be run concurrently by a
duplicate tick-dispatched job.
- StartHostOnboarding creates host+run in a transaction, dispatches after commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo
Claude