A booked storage pack was a row in `subscription_addons` and nothing else:
priced, frozen, charged every month, and delivered nowhere. Nextcloud's
quota came from `instances.quota_gb`, which is the package alone; the disk
was sized for the package alone; and on the one occasion a disk did grow —
a plan upgrade — the guest never saw it, because nothing stretched the
partition or the filesystem over the new space.
One authority for the allowance. StorageAllowance adds the package and the
booked packs, and everything that needs the figure asks it: the quota step,
the downgrade check, the portal, the console, the repair command. It is
DERIVED rather than stored — a column would have to be rewritten by booking,
cancelling, granting, a grant expiring and a plan change, and the day one of
those paths forgot it would be silently wrong in whichever direction costs
somebody money. `quota_applied_gb` keeps its own meaning: what the guest was
last actually told, which is how a machine with the allowance enforced is
told from one where the figure has only ever been a row in our database.
Buying it delivers it. BookAddon asks ApplyStorageAllowance for a run on
booking AND on cancellation; the new `storage` pipeline grows the virtual
disk to the allowance plus the package's own overhead (read off the package
— 20/40/50/100 GB on the four catalogue packages — never a ratio invented
here), then GrowGuestFilesystem makes the guest see it, then the quota is
applied. No cold boot is involved: the data disk is scsi0 and Proxmox's
resize on a running guest is a qemu block_resize, so the capacity reaches
the guest while it runs. The new step rescans, growparts and grows the
filesystem with the tool its type actually needs — ext2/3/4, xfs, btrfs —
and fails loudly on one it does not know rather than handing it to resize2fs
and hoping. Idempotent end to end: NOCHANGE from growpart is not a failure,
and every tool here exits 0 when there is nothing left to grow.
The same step now runs in the plan-change pipeline, where an upgrade used to
stop one step short of the customer.
A downgrade blocked by data gets a way out. The block stays — it is correct
— but the check now measures against the target package PLUS the packs the
customer already owns, and reports the numbers behind the refusal: what is
stored, what would be allowed, how much has to go, and how many packs would
cover it instead. The portal offers both routes: book exactly those packs
(confirmed in a modal, R23), or delete data and have the fill level measured
on demand rather than waiting for the nightly sampler. That reading is taken
by DiskUsageProbe, which CollectInstanceTraffic now uses too, so there is one
notion of "how full is it" and not two.
A plan change keeps booked packs. They were paid for separately and have
nothing to do with which package the customer is on; the new package's disk
and quota are sized with them included.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A paid order that no host has room for used to FAIL at the reservation
step. The customer has paid by then, so that turned a sale into an
incident and left somebody holding money against nothing. The estate is
finite and booked thick, so "no host has room right now" is an ordinary
Tuesday — it means a machine has to be bought, which takes days.
So the order waits instead. poll(), not retry(): the wait is measured in
days and must not eat the run's attempt budget. It only fails after
PARK_DAYS (14) — long enough to buy, rack and onboard a server over a
weekend, short enough that a forgotten order surfaces instead of waiting
for ever.
Then say so, in the same words on both sides of the payment:
- The price sheet marks each package "wird sofort ausgeliefert" or
"Bereitstellung in 2-3 Werktagen", read from free capacity rather
than written into the page.
- The customer's own overview says their cloud is being prepared while
it is parked, instead of a stepper standing still for two days under
"wird eingerichtet" — which reads as broken.
- The console front page shows the same per-package answer, so an
operator can see what visitors are being promised.
And a capacity page for the decision that follows: who is waiting, what
the roomiest host still has, what the queue needs ("3x Start, 1x
Business" - the LARGEST parked package sets the minimum machine, because
an instance lives on one host), and what such a machine costs today from
the provider's live list. It orders nothing: buying a server is a
contract, and a bug in a capacity calculation must not sign one.
The operator can also say where each parked order goes. A pin is
honoured or it waits — never quietly redirected — because placing it
elsewhere answers a different question than the one they answered, and
they would find out from the finished instance.
Two things this shook out:
- `current_step` is an INDEX into the pipeline, not the step's key, and
it is cast to integer. `where('current_step', 'reserve_resources')`
reads as correct and matches step 0 of EVERY pipeline instead. Both
lookups go through HostCapacity::parkedRuns() now, which resolves the
index and filters by pipeline.
- The auction feed sends `hdd_hr` as a list, not a string. Casting it
printed "Array" into the console and raised a warning that took the
page down with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three places where the product said one thing and did another.
**Nothing could restart a virtual machine.** ProxmoxClient had startVm and no
way to stop anything, so `restart_required_since` — set by every plan change
that grows a running guest — could only ever be cleared by a resize step that
happened to find the machine stopped, which nothing in the product could
arrange. A paid upgrade's cores and RAM could stay unreached for the life of the
contract, and the cloud page's "Neu starten" was a toast.
There is now a `restart` pipeline: shutdown, start, wait for the agent, confirm.
A shutdown and a start rather than a reboot, because only a cold boot makes qemu
read the VM definition again — a reset would take the machine round and bring it
back exactly as small as it was. The shutdown is a REQUEST, bounded at ten
minutes, and there is no escalation: ProxmoxClient deliberately does not expose
Proxmox's forceStop, because the guest is a Nextcloud and cutting power to a
database mid-write to apply a CPU change trades an inconvenience for a restore
from backup. A guest that ignores ACPI for ten minutes fails the run instead, and
a person decides what happens to it.
`restart_required_since` is cleared by the last step, from what the RUNNING guest
reports (`cpus`/`maxmem`) — not by the button, and not by the run merely
finishing. A machine that comes back smaller than it was sold fails the run.
Started by App\Actions\RestartInstance, which re-checks authorisation itself
rather than trusting a hidden button: an operator holding the new
`instances.restart` may restart anything, a portal user their own machine and
nothing else. Refused while another run is in flight against the same order.
Confirmed in a modal on both sides (R23), addressed by uuid (R11).
**Existing instances had no storage quota.** ApplyStorageQuota reaches new and
changed machines; everything built before it kept the whole disk whatever it had
paid for, and nothing recorded which was which. `instances.quota_applied_gb` is
now what the guest was actually told, written by the step only after the occ call
succeeds — so "sold" and "enforced" are separate facts and can be compared.
`clupilot:apply-quotas` sweeps the estate: one `quota` run per instance through
the same step the pipelines use, skipping what it should and saying why, a no-op
once an allowance is enforced, and inert under --dry-run. Deliberately NOT
scheduled — a nightly sweep would quietly cover for the pipeline step the day it
stopped working, which is how the original hole survived this long.
**A Traefik router was never torn down.** TraefikWriter::remove() had no caller
anywhere. What was missing was the moment: ConfirmCancelPackage wrote a date into
`service_ends_at` and nothing ever went back to it, so every route this platform
ever wrote was written for good — pointing at a guest address the host is free to
reassign. `clupilot:end-due-services` (hourly) keeps that appointment through
App\Actions\EndInstanceService. "Ended" means cancellation_scheduled AND
`service_ends_at` passed, never the day a cancellation is scheduled: that
customer has paid to the end of the term and is working in it. The DNS record
goes with the route — it is in our zone, pointing at a host that serves other
people, which is the shape of every subdomain takeover there has ever been. The
customer's own domain is in their zone and is not ours to touch; the virtual
machine is left alone, because the cancellation flow promises a data export first
and destroying disks is not this change's decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The platform books thick: a package reserves its whole disk_gb — the sold
quota plus the machine's own overhead — on ONE host at placement. That is the
right choice for a small operator, because nothing can quietly overfill. It
has one consequence nobody was being told about: the catalogue goes on
offering a package no host has room for, and the first anyone hears of it is a
PAID order failing at the reservation step.
On the machine this was found on the numbers are stark. 2 x 512 GB NVMe in a
mirror is about 400 GB usable; Start reserves 120, Team 540, Business 1050,
Enterprise 2100. Three of the four packages on sale cannot be placed at all,
and the fourth fits three times.
So the console says so now, per package, with the shortfall — "Team needs 540
GB, the roomiest host has 260" — because "buy a server" is not the decision.
"Buy a server big enough for what I am selling" is.
Two things it deliberately does not do. It does not warn on an installation
with no host at all: that is a machine nobody has onboarded yet, it has its
own notice, and without the guard a fresh console greeted its operator with
one warning per package before they had done anything. And it does not take
the overview down when the catalogue cannot be read — a capacity warning does
not get to break the page somebody opens when something is already wrong.
The largest FREE HOST, never the sum: an instance goes on one machine, so two
hosts with 300 GB each cannot take a 540 GB package between them.
And when placement does fail, the log now names the shortfall instead of only
'no_capacity'.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PlanChange could say what a move would cost and whether it was allowed, and
that was all it could do. Billing::purchase() wrote an upgrade order and
nothing ever consumed it: same snapshot, same machine, same quota. A customer
could pay for a bigger package and receive nothing.
ApplyPlanChange is now the single place a change lands — it moves the contract
onto the target's current version, writes one register row, settles the custom
domain, and starts a run that resizes the machine. Applying the same order
twice is a no-op, enforced by a unique event key rather than by a check two
callers could both pass.
Two things a plan change must not do, and now does not. A disk is never shrunk
— Proxmox cannot, so the QUOTA shrinks instead, which is what was sold anyway.
And a live machine is never rebooted as a side effect: cores and RAM are
written, and where they need a restart the instance says so where an operator
and the customer can both see it.
The storage allowance also joins the build pipeline. ApplyStorageQuota was
written for plan changes and ran only there, so a brand-new customer still got
no quota at all — quota_gb reached the instance row and stopped, and every
package delivered the whole disk. The test guarding that pipeline only counted
its steps, which is how a list missing the one step that makes a package's
storage real stayed green for its whole life. It now names the step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A verified custom domain was reported as the customer's address by
Instance::address(), by the portal and by the credentials mail while
nothing on the platform routed it: the Traefik router's rule was
hard-coded to {subdomain}.{zone}. The address a customer was handed
answered nothing, and a withdrawn domain stayed in Nextcloud's
trusted_domains forever, because the only thing that ever wrote either
was the initial provisioning run.
- TraefikWriter::write() takes a LIST of hostnames under one stable
router name — the platform address always, the verified domain as
well when there is one. One file per instance, so a withdrawal is a
rewrite rather than a second thing somebody has to remember.
- ConfigureDnsAndTls records the custom domain's certificate instead of
waiting on it: it depends on an A record in the customer's own zone,
which may never appear, and must never fail a run. The platform
address keeps its 840s deadline. instances.domain_cert_ok tells
"proven" apart from "answering", and the portal now says which.
- ConfigureNextcloud deletes trusted_domains 2 when there is no verified
domain, so a withdrawn one stops being trusted.
- New `address` pipeline (those two steps) plus ReapplyInstanceAddress,
which starts one against the order and refuses to start a second while
any run is in flight. The route is rewritten when the hostname list
differs from what the router carries — not on route_written, which
would short-circuit exactly the case a re-apply exists for.
- Triggered where the address changes: verification flipping either way
in clupilot:verify-domains, the customer's own domain page, and
CustomDomainAccess::deactivate() on a package downgrade.
- A maintenance run no longer condemns its subject: an address run that
failed used to mark the order failed and release the live instance
with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things.
── The update screen, still opening twice ───────────────────────────────────
Reported again on 1.3.9, and the cause was not the one fixed in 1.3.8. The
agent consumes the request file BEFORE it resolves the release — deliberately,
because update.sh may kill the shell and a request left in place would loop —
and writes `state: running` only once it has decided to go ahead. In between,
the request is gone and the status does not say running yet, so the endpoint
honestly answers "nothing is running". The watcher read that as "the run has
finished" and reloaded the page: overlay on the click, gone a poll later, 503
after it.
The overlay now closes only once the server has BOTH confirmed a run and then
stopped reporting it. Before the confirmation, silence means the agent has not
got there yet. Bounded at twenty polls so a request the agent refuses does not
leave the console covered forever.
── Custom domains, proven and re-proven ─────────────────────────────────────
`custom_domain` was a free-text field and everything downstream believed it:
the proxy served it, the certificate was issued for it, Nextcloud trusted it.
Anyone who pointed any hostname at the platform got somebody else's files
under their own name.
The proof is a TXT record at _clupilot-challenge.<domain> holding a token only
this instance has. Nothing is served until it has been read. Every reader now
goes through Instance::address(), which is the one place the decision is
made — `custom_domain ?: subdomain` was the hole, written out four times.
It is re-read every night at 03:40, because a token checked once can be taken
straight back out and a domain that later lapses keeps resolving here. Three
consecutive misses before a live domain is withdrawn: one failed lookup is a
nameserver having a bad minute, and withdrawing takes a working Nextcloud off
its own address. The domain and its token stay on the row so the customer can
put the record back rather than start over.
Changing the domain mints a NEW token. Reusing it would let somebody who once
verified example.com claim any other domain later without touching its DNS —
the old record is still sitting there and only the value is compared.
The domain is changeable at any time, and removable. Fixing it once set was
considered and rejected: adding or moving an address is a proxy entry, a
certificate and one line in trusted_domains.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SSH private key CluPilot deploys to every host is now stored through
SecretVault (ssh.private_key), preferred over CLUPILOT_SSH_PRIVATE_KEY(_PATH)
at all three real consumers — SshTraefikWriter and HostStep::keyLogin.
kuma.password/kuma.totp stay in .env: they authenticate a separate Python
container at boot, and nothing in this app reads them, so the vault would
have nothing to wire them to.
A new /admin/infrastructure page, backed by App\Support\ProvisioningSettings,
makes the non-secret deployment settings that used to force a shell —
DNS zone, WireGuard hub endpoint and public key, Traefik's dynamic-config
path, the SSH public key, and the monitoring bridge URL — visible and
editable from the console, each wired to every real consumer. WG subnet/hub
IP and Kuma's own connection details stay in .env: the compose file and a
separate container read those before this application ever does.
The password no longer travels through third-party mail servers or sits
forever in an inbox: it moves onto the instance (encrypted, same as
nc_admin_ref) and shows on the dashboard until the customer clicks
"Notiert", which deletes it for good.
Three things the owner asked for, and one the review found underneath them.
The console leaves /admin. It has its own route file now, registered before
routes/web.php because once it has a hostname to itself it sits at the ROOT of
that host — where `/` and `/settings` also exist for the portal, and the first
matching route wins. Where it mounts is one decision in one place: `/` on the
console hostname where it has one to itself, `/admin` on any host otherwise.
Names stay admin.* in both modes, so nothing that builds a URL has to know.
Exclusivity is its own switch, not a consequence of having a hostname. The
first attempt made "this host is the console" follow from ADMIN_HOSTS, and a
development machine lists its own IP there so the console works without DNS —
which took the public site and the portal off that machine entirely. The
switch also registers the console on every listed hostname, canonical last, so
the alternates that exist as recovery paths keep working.
The status page moves out of /legal, where it sat beside the imprint and the
terms. Nothing about the current health of the platform is a legal document. It
is a real page now: portal, instances, provisioning and backups, each derived
from records, aggregate only, and a component with no signal reports "unknown"
rather than "operational".
That last rule is what exposed the real bug. monitoring_targets.status was
written once — 'up', at provisioning — and nothing ever updated it. Both the
console's notices and the new public page read it, so an outage would have been
published as healthy indefinitely. There is a sync job now, on a five-minute
schedule, and a checked_at column so a verdict can go stale instead of standing
forever.
The monitoring contract had to grow a third state for that to be honest.
isHealthy() answers true when no monitoring is configured and false when the
monitor is unreachable; recording that boolean would have published either a
fleet-wide all-clear nobody measured or a fleet-wide outage that was really one
broken monitor. health() returns null for both, the recorder leaves the old
verdict to go stale, and isHealthy() keeps its forgiving semantics for the one
caller that wants them — the provisioning acceptance check, which must not fail
a delivery because monitoring is not set up.
Backups are counted from the instances that need protecting rather than from
the backup rows that exist, so an instance with no schedule at all cannot be
missing from the arithmetic that declares the estate protected.
Also: the update panel polls itself, offers the button only when there is
something to install, and opens the log while a run is in progress.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pipeline re-resolved config('provisioning.plans') by order.plan, so the
subscription snapshot protected a customer's price but not their machine:
shrinking a plan resized an existing customer's VM on its next run. Nothing
created a subscription either, so closing this meant opening the contract at
purchase and pointing provisioning at it.
- OpenSubscription freezes the catalogue onto a subscription when a checkout
is paid; StartCustomerProvisioning calls it inside the order transaction.
- CustomerStep::plan() reads the frozen snapshot. ValidateOrder and
ReserveResources fail closed with no_subscription rather than falling back
to the catalogue, which is the bug itself.
- template_vmid joins the snapshot so a re-clone cannot pick up a blueprint
published after the sale. Deliberately outside FROZEN: it is how we build
the machine, not a term the customer is owed, and a dead template must be
replaceable without cancelling a contract.
- TrafficMeter reads the allowance off the contract too — cutting a plan's
traffic was otherwise enough to start throttling someone who bought more.
- The migration backfills contracts for orders that already bought something,
reconstructed from what was actually delivered where an instance exists,
and adopts an existing order-less contract instead of opening a second.
Orders paid in a currency the catalogue cannot price get none, matching the
checkout path.
price_cents stays the catalogue's NET price, which is what PlanChange
prorates against — not Order::amount_cents, which holds Stripe's GROSS total.
Reconciling the two belongs to the proof register and Stripe (phases 4/5).
Also pins STRIPE_WEBHOOK_SECRET blank in phpunit.xml: the operator's real
secret was reaching the suite from .env and rejecting every unsigned test
payload, which is why 7 webhook tests failed before any of this.
Verified in the browser: with team traffic cut from 3000 to 500 GB in the
catalogue, the customer's portal still shows 3 TB.
373 tests green. Codex review clean after three rounds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The failure may be a lost response to a request that did create the record.
Advancing straight away stored no id, so PurgeHost could never remove it and the
host's management address stayed published. The upsert is idempotent, so a retry
recovers the id; only after the attempts run out does the onboarding continue
without a name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two legacy codes can normalise to the same label (eu_west and eu-west), and
separate counters then handed both of them eu-west-01 — a unique-constraint
failure inside the reservation, which blocks onboarding rather than being a DNS
problem the step can shrug off. Lock, counter and the in-use check all key off
the label now.
(The helper was also called label(), which HostStep already declares as the
step's display name — renamed.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
alpha_dash accepted eu_west, -edge and edge-, none of which are valid DNS
labels — every host in such a datacenter would have failed registration and,
because DNS is non-fatal, silently ended up without a name. The console now
requires a proper label, and the step normalises anything created before that
rule so existing rows still get a usable name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex was right that my never-reuse claim did not hold:
- The lock was released with only a candidate in hand, so two concurrent
onboardings in one datacenter could pick the same name and overwrite each
other's record. The name is now persisted on the host while still locked.
- The number was derived from the hosts that happen to exist, so removing the
highest one handed that number straight back out — a cached name would then
resolve to a different machine. The counter is stored per datacenter, floored
by what is actually in use so a wiped settings store cannot reissue live names.
- PurgeHost deleted the row that carried the record id, leaving the machine's
management address published with nothing left to clean it up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fsn-01.node.clupilot.com, numbered per datacenter and never reused — removing a
host must not renumber its neighbours onto its name, so the number is stored
rather than derived.
The record points at the host's WireGuard address, not its public IP: the name
exists so an operator can reach a host by name over the VPN, and publishing a
Proxmox host's public address would hand every scanner a target, which is the
one thing this network design avoids.
DNS is convenience, not a prerequisite: a failure logs an event and lets the
onboarding finish rather than stranding a host that is otherwise ready.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard and RemoveWireguardPeer mutate the hub too, and were not
taking the lock, so a sync could still read the interface around them and
write stale state. Both now hold wireguard:hub, which is what makes the
read/mutation guarantee actually hold.
- The duplicate-key check ran before the allocation lock, so two concurrent
creations with the same key both passed it and the loser hit the unique index
as an unhandled 500. The check moved inside the lock, with the index kept as
a caught backstop for any writer that does not take it — reproduced in a test
by inserting a colliding row from within the allocation call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- MONITORING_ATTEMPTS is capped by the run's max_attempts, so a large value can
no longer burn the retry budget and fail the very provisioning the degradation
exists to protect (covered by a new test at the budget boundary)
- /health is now dependency-free (200 in ~2ms with Kuma absent, verified: the
container healthcheck stays healthy); Kuma state is exclusively in /ready
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- attempt is 0-based, so compare one-based: ATTEMPTS=2 really means two tries
- bridge accepts add_monitor returning {monitorID}|{id}|int (v1.2.1 returns a
dict — re-verified: first create on a fresh Kuma is HTTP 200 with the id)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.
Verified end-to-end against a real Uptime Kuma 1.x:
create -> id, create again -> same id (idempotent), list, status, delete
and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.
Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
monitor legitimately reports 'pending' — confirmed in the e2e run); it records
the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Clone recovery: once the lock clears but the task ref is lost, destroy the
possibly-incomplete VM and re-clone instead of advancing onto it.
- applyFirewall clears existing rules before adding, so retries don't accumulate
duplicate Proxmox firewall entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Nextcloud: parse occ status JSON (installed + not in maintenance).
- Admin: query occ user:info for the account.
- Monitoring: MonitoringClient::isHealthy checks the provider, not the local row.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TraefikWriter now targets the host that serves the traffic (DNS points at it):
SshTraefikWriter SSHes in and writes the file-provider YAML there. Interface
takes the traffic host + guest backend.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CompleteProvisioning scrubs admin_password whenever credentials_sent exists,
so a crash between recording and scrubbing can't leave the password in context.
- HttpMonitoringClient looks up an existing monitor by URL before creating one,
so a retry after a crashed POST doesn't create a duplicate external monitor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Clone recovery polls until the VM lock clears instead of advancing onto a
possibly-incomplete clone.
- RegisterBackup/RegisterMonitoring create the local row BEFORE the run-resource
breadcrumb, so a crash between them can't leave the row permanently missing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureCloudInit sets cores/memory from the plan (not the template default).
- RegisterBackup uses a deterministic Proxmox job id (ignore-exists) + firstOrCreate.
- RegisterMonitoring is idempotent by URL + firstOrCreate — no duplicate schedules
or monitors after a crash/retry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- applyFirewall sets policy_in=DROP so only 80/443 are exposed.
- Capacity/placement account for disk_gb (the real VM allocation), not the
smaller Nextcloud user quota — no systematic overcommit.
- CloudReady sent synchronously again; the step retries on mail failure with the
password preserved (no lost credential in a failed queue job).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CloudReady receives the ENCRYPTED password (decrypts only when rendering), so
the queued payload never holds plaintext.
- RegisterBackup uses a valid Proxmox calendar expression (02:00).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CreateCustomerAdmin puts OC_PASS on the docker invocation (survives the &&).
- RegisterBackup creates a real vzdump job via ProxmoxClient::createBackupJob.
- RegisterMonitoring registers via a new MonitoringClient service (interface +
fake + http). Both persist the external id [E] before the breadcrumb, so
acceptance reflects a real registration, not a fabricated row.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Clone: recover via vmExists() when the task ref was lost, instead of
re-cloning the reserved vmid (which Proxmox rejects).
- Deploy: persist the DB password ENCRYPTED before compose up and reuse it on
retry, so a crash can't regenerate a mismatching credential.
- Complete: gate credential delivery on a credentials_sent breadcrumb so a
retry after a crash doesn't re-send the email.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Failed run releases the instance (status=failed) so its quota stops counting
against host capacity.
- CompleteProvisioning activates instance+order only AFTER onboarding tasks +
credential delivery succeed.
- ConfigureNetwork polls until the guest has an address (no wrong Traefik route).
- customers.email unique + race-safe customer resolution in the intake action.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RunRunner::onPoll no longer resets started_at, so a poll step's own deadline
(WaitForGuestAgent 270s, ConfigureDnsAndTls cert 840s) actually fires instead
of resetting every poll. maxDuration raised above each own-deadline so the
step's fail wins over the generic timeout. Shared core fix (A + B).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Require a real customer email (no unknown@ fallback merging customers).
- Stripe signature: enforce 5-min replay tolerance + accept any rotating v1.
- DeployApplicationStack no longer persists the DB password in the run context.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Stripe webhook fails closed when the signing secret is missing (outside local/testing).
- ReserveResources places + creates the instance under a per-datacenter lock so
concurrent orders can't overcommit a host.
- CloneVirtualMachine reserves the vmid + breadcrumb BEFORE cloning, so a crash
can't mint a new vmid and orphan the first clone.
- CreateCustomerAdmin checks user existence and resets the password instead of
re-running user:add on retry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Stripe: only checkout.session.completed with payment_status=paid (dedupes the
paired payment_intent event and blocks unpaid async sessions).
- Capture the guest IP (ConfigureNetwork) and point Traefik at the VM, not the
Proxmox host, so ACME/HTTP-01 can reach Nextcloud.
- Resize the disk to an absolute target (not '+…') so a retry can't double it.
(Codex #4 timeout was a false positive — started_at resets per step.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keeps the wg_peer resource gated on a verified handshake (idempotency) while
ensuring PurgeHost can always remove the hub peer, even on terminal failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard checks setup command results and only records the wg_peer
resource after the handshake verifies, so a transient setup failure re-runs
the full setup instead of getting stuck on the idempotent path.
- Host removal now deactivates immediately and queues PurgeHost, which deletes
runs under the runner lock (waiting out a long step) — no 15s LockTimeout error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Removing a host now removes its WireGuard hub peer so a freed wg_ip can't
route to the removed server via a stale peer.
- RebootIntoPveKernel clears reboot_issued/deadline when it times out, so the
manual retry action actually re-issues a reboot instead of failing instantly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hub_ip is now defined in config (default: subnet .1), so a custom
CLUPILOT_WG_SUBNET no longer pings the wrong handshake target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- RunRunner returns early for a waiting run whose next_attempt_at is in the
future, so stale/duplicate jobs can't bypass backoff.
- RegisterCapacity takes the largest VM-capable datastore instead of summing
overlapping pools (local + local-lvm), preventing placement overcommit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- StepResult::poll — polling steps (reboot) wait without consuming the retry
budget; the step owns its deadline. Reboot maxDuration > deadline.
- Failed runs move a Host subject to 'error' via ProvisioningSubject hook
(no host stuck 'onboarding').
- ConfigureWireguard allocates + reserves the wg_ip under a global lock;
unique index on hosts.wg_ip as a backstop against duplicate addresses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CreateAutomationToken now bootstraps the pveum role/user/token over the
authenticated SSH session (a fresh host has no API token yet); ProxmoxClient
is read-only in A.
- ConfigureWireguard re-verifies the handshake on the idempotent replay path,
never advancing over a dead tunnel.
- PhpseclibRemoteShell treats a missing SSH exit status as failure (255).
- connectWithKey verifies the pinned host key fingerprint on later logins.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo
Claude