13 Commits (880b5f19982d51c9ecf42fdaf3b2e51fe5163643)
| Author | SHA1 | Message | Date |
|---|---|---|---|
nexxo
|
889b401faf |
Fix nine defects in the provisioning pipelines
The capacity park could not survive its own first poll. ReserveResources
polls every 120 s and declared a maxDuration of 60, and RunRunner measures
a step's budget from a started_at it deliberately does not reset on a poll
— so every re-entry was ruled timed out before the body ran, a timeout
consumes an attempt, and five of them failed a paid order in about six
minutes. The fourteen-day promise, the console's capacity queue and the
"go and buy a server" workflow were unreachable code. maxDuration is now
the whole park plus one poll interval, and a test drives a park through
the runner rather than calling execute() directly, which is why this was
invisible.
Every guest occ call ran as root. `docker compose exec` defaults to root
and the Nextcloud image's console.php exits 1 unless the caller owns
config/config.php, so all five call sites were failing on every instance.
There is now one builder, App\Support\NextcloudOcc, and a test refuses a
second: nothing in app/ may spell the invocation out by hand. deploy/
update.sh had learned this for our own container and nobody carried it
across.
RunAcceptanceChecks was terminal on the first no. certReachable() answers
false for a connect timeout as readily as for a missing certificate, so
one bad second ended a finished, certified Nextcloud as a failed order
with the instance released. The probes retry now and the run still fails
for good with the probe's own reason once the budget is spent; the two
facts a retry cannot change stay terminal.
The in-flight guard asked whether anything was running, not whether it
would do the work. App\Provisioning\WorkInFlight asks the second question,
reading the step lists the runner executes: a domain proven during a
restart is routed, and a storage pack booked during an address run is
delivered instead of charged monthly and forgotten.
The address pipeline ran its steps in the wrong order for the direction it
exists for. Nextcloud before the router in `address` and `plan-change`, as
in `customer`, because a run that got the certificate and then failed left
the customer's own domain serving an untrusted-domain error under a valid
certificate while the portal called it live. Safe in both directions; the
reasoning is written above the pipeline.
A stale guest_ip could never be corrected. ConfigureDnsAndTls compared the
hostnames but never the backend, and nothing re-read the address after the
build. It now records and compares `routed_backend`, and ConfigureNetwork
joins the `restart` pipeline — a cold boot is what moves a DHCP lease — and
asks for the address to be re-applied when the guest has actually moved.
Also: HetznerDnsClient::upsertRecord read only the first page of a zone
that pages at 100, so past a hundred records it created a second A record
for the same name and the cloud was up about half the time; the console's
retry left a live customer's order in `provisioning` for ever for any
maintenance pipeline; and a revived run's first pass burned an attempt on a
timeout that had already happened, because the clause written for it used
`??=`.
Seven test sites wrote config('provisioning.plans.*'), a key that has not
existed since the catalogue moved into the database. Every write was a
no-op, so each test proved something other than its name: the missing
template, the two snapshot regressions and the grandfathered-price scenario
are now actually constructed, ValidateOrder's reasons are asserted rather
than only its type, and the end-to-end run checks that the quota step
delivered something.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
|
nexxo
|
5b63fdb86c |
Deliver the storage a customer actually buys
A booked storage pack was a row in `subscription_addons` and nothing else: priced, frozen, charged every month, and delivered nowhere. Nextcloud's quota came from `instances.quota_gb`, which is the package alone; the disk was sized for the package alone; and on the one occasion a disk did grow — a plan upgrade — the guest never saw it, because nothing stretched the partition or the filesystem over the new space. One authority for the allowance. StorageAllowance adds the package and the booked packs, and everything that needs the figure asks it: the quota step, the downgrade check, the portal, the console, the repair command. It is DERIVED rather than stored — a column would have to be rewritten by booking, cancelling, granting, a grant expiring and a plan change, and the day one of those paths forgot it would be silently wrong in whichever direction costs somebody money. `quota_applied_gb` keeps its own meaning: what the guest was last actually told, which is how a machine with the allowance enforced is told from one where the figure has only ever been a row in our database. Buying it delivers it. BookAddon asks ApplyStorageAllowance for a run on booking AND on cancellation; the new `storage` pipeline grows the virtual disk to the allowance plus the package's own overhead (read off the package — 20/40/50/100 GB on the four catalogue packages — never a ratio invented here), then GrowGuestFilesystem makes the guest see it, then the quota is applied. No cold boot is involved: the data disk is scsi0 and Proxmox's resize on a running guest is a qemu block_resize, so the capacity reaches the guest while it runs. The new step rescans, growparts and grows the filesystem with the tool its type actually needs — ext2/3/4, xfs, btrfs — and fails loudly on one it does not know rather than handing it to resize2fs and hoping. Idempotent end to end: NOCHANGE from growpart is not a failure, and every tool here exits 0 when there is nothing left to grow. The same step now runs in the plan-change pipeline, where an upgrade used to stop one step short of the customer. A downgrade blocked by data gets a way out. The block stays — it is correct — but the check now measures against the target package PLUS the packs the customer already owns, and reports the numbers behind the refusal: what is stored, what would be allowed, how much has to go, and how many packs would cover it instead. The portal offers both routes: book exactly those packs (confirmed in a modal, R23), or delete data and have the fill level measured on demand rather than waiting for the nightly sampler. That reading is taken by DiskUsageProbe, which CollectInstanceTraffic now uses too, so there is one notion of "how full is it" and not two. A plan change keeps booked packs. They were paid for separately and have nothing to do with which package the customer is on; the new package's disk and quota are sized with them included. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
d76de5ada3 |
Restart a machine, enforce the quota that was sold, end a route that ended
Three places where the product said one thing and did another. **Nothing could restart a virtual machine.** ProxmoxClient had startVm and no way to stop anything, so `restart_required_since` — set by every plan change that grows a running guest — could only ever be cleared by a resize step that happened to find the machine stopped, which nothing in the product could arrange. A paid upgrade's cores and RAM could stay unreached for the life of the contract, and the cloud page's "Neu starten" was a toast. There is now a `restart` pipeline: shutdown, start, wait for the agent, confirm. A shutdown and a start rather than a reboot, because only a cold boot makes qemu read the VM definition again — a reset would take the machine round and bring it back exactly as small as it was. The shutdown is a REQUEST, bounded at ten minutes, and there is no escalation: ProxmoxClient deliberately does not expose Proxmox's forceStop, because the guest is a Nextcloud and cutting power to a database mid-write to apply a CPU change trades an inconvenience for a restore from backup. A guest that ignores ACPI for ten minutes fails the run instead, and a person decides what happens to it. `restart_required_since` is cleared by the last step, from what the RUNNING guest reports (`cpus`/`maxmem`) — not by the button, and not by the run merely finishing. A machine that comes back smaller than it was sold fails the run. Started by App\Actions\RestartInstance, which re-checks authorisation itself rather than trusting a hidden button: an operator holding the new `instances.restart` may restart anything, a portal user their own machine and nothing else. Refused while another run is in flight against the same order. Confirmed in a modal on both sides (R23), addressed by uuid (R11). **Existing instances had no storage quota.** ApplyStorageQuota reaches new and changed machines; everything built before it kept the whole disk whatever it had paid for, and nothing recorded which was which. `instances.quota_applied_gb` is now what the guest was actually told, written by the step only after the occ call succeeds — so "sold" and "enforced" are separate facts and can be compared. `clupilot:apply-quotas` sweeps the estate: one `quota` run per instance through the same step the pipelines use, skipping what it should and saying why, a no-op once an allowance is enforced, and inert under --dry-run. Deliberately NOT scheduled — a nightly sweep would quietly cover for the pipeline step the day it stopped working, which is how the original hole survived this long. **A Traefik router was never torn down.** TraefikWriter::remove() had no caller anywhere. What was missing was the moment: ConfirmCancelPackage wrote a date into `service_ends_at` and nothing ever went back to it, so every route this platform ever wrote was written for good — pointing at a guest address the host is free to reassign. `clupilot:end-due-services` (hourly) keeps that appointment through App\Actions\EndInstanceService. "Ended" means cancellation_scheduled AND `service_ends_at` passed, never the day a cancellation is scheduled: that customer has paid to the end of the term and is working in it. The DNS record goes with the route — it is in our zone, pointing at a host that serves other people, which is the shape of every subdomain takeover there has ever been. The customer's own domain is in their zone and is not ours to touch; the virtual machine is left alone, because the cancellation flow promises a data export first and destroying disks is not this change's decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
34874adec3 |
Apply a bought plan change instead of only pricing it
tests / pest (push) Failing after 8m6s
Details
tests / assets (push) Successful in 21s
Details
tests / release (push) Has been skipped
Details
PlanChange could say what a move would cost and whether it was allowed, and that was all it could do. Billing::purchase() wrote an upgrade order and nothing ever consumed it: same snapshot, same machine, same quota. A customer could pay for a bigger package and receive nothing. ApplyPlanChange is now the single place a change lands — it moves the contract onto the target's current version, writes one register row, settles the custom domain, and starts a run that resizes the machine. Applying the same order twice is a no-op, enforced by a unique event key rather than by a check two callers could both pass. Two things a plan change must not do, and now does not. A disk is never shrunk — Proxmox cannot, so the QUOTA shrinks instead, which is what was sold anyway. And a live machine is never rebooted as a side effect: cores and RAM are written, and where they need a restart the instance says so where an operator and the customer can both see it. The storage allowance also joins the build pipeline. ApplyStorageQuota was written for plan changes and ran only there, so a brand-new customer still got no quota at all — quota_gb reached the instance row and stopped, and every package delivered the whole disk. The test guarding that pipeline only counted its steps, which is how a list missing the one step that makes a package's storage real stayed green for its whole life. It now names the step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
712803edd6 |
Serve the custom domain, not just announce it
A verified custom domain was reported as the customer's address by
Instance::address(), by the portal and by the credentials mail while
nothing on the platform routed it: the Traefik router's rule was
hard-coded to {subdomain}.{zone}. The address a customer was handed
answered nothing, and a withdrawn domain stayed in Nextcloud's
trusted_domains forever, because the only thing that ever wrote either
was the initial provisioning run.
- TraefikWriter::write() takes a LIST of hostnames under one stable
router name — the platform address always, the verified domain as
well when there is one. One file per instance, so a withdrawal is a
rewrite rather than a second thing somebody has to remember.
- ConfigureDnsAndTls records the custom domain's certificate instead of
waiting on it: it depends on an A record in the customer's own zone,
which may never appear, and must never fail a run. The platform
address keeps its 840s deadline. instances.domain_cert_ok tells
"proven" apart from "answering", and the portal now says which.
- ConfigureNextcloud deletes trusted_domains 2 when there is no verified
domain, so a withdrawn one stops being trusted.
- New `address` pipeline (those two steps) plus ReapplyInstanceAddress,
which starts one against the order and refuses to start a second while
any run is in flight. The route is rewritten when the hostname list
differs from what the router carries — not on route_written, which
would short-circuit exactly the case a re-apply exists for.
- Triggered where the address changes: verification flipping either way
in clupilot:verify-domains, the customer's own domain page, and
CustomDomainAccess::deactivate() on a package downgrade.
- A maintenance run no longer condemns its subject: an address run that
failed used to mark the order failed and release the live instance
with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
|
nexxo
|
ea643b5e73 |
Prove a custom domain before serving it, and keep proving it
tests / pest (push) Failing after 8m17s
Details
tests / assets (push) Successful in 23s
Details
tests / release (push) Has been skipped
Details
Two things. ── The update screen, still opening twice ─────────────────────────────────── Reported again on 1.3.9, and the cause was not the one fixed in 1.3.8. The agent consumes the request file BEFORE it resolves the release — deliberately, because update.sh may kill the shell and a request left in place would loop — and writes `state: running` only once it has decided to go ahead. In between, the request is gone and the status does not say running yet, so the endpoint honestly answers "nothing is running". The watcher read that as "the run has finished" and reloaded the page: overlay on the click, gone a poll later, 503 after it. The overlay now closes only once the server has BOTH confirmed a run and then stopped reporting it. Before the confirmation, silence means the agent has not got there yet. Bounded at twenty polls so a request the agent refuses does not leave the console covered forever. ── Custom domains, proven and re-proven ───────────────────────────────────── `custom_domain` was a free-text field and everything downstream believed it: the proxy served it, the certificate was issued for it, Nextcloud trusted it. Anyone who pointed any hostname at the platform got somebody else's files under their own name. The proof is a TXT record at _clupilot-challenge.<domain> holding a token only this instance has. Nothing is served until it has been read. Every reader now goes through Instance::address(), which is the one place the decision is made — `custom_domain ?: subdomain` was the hole, written out four times. It is re-read every night at 03:40, because a token checked once can be taken straight back out and a domain that later lapses keeps resolving here. Three consecutive misses before a live domain is withdrawn: one failed lookup is a nameserver having a bad minute, and withdrawing takes a working Nextcloud off its own address. The domain and its token stay on the row so the customer can put the record back rather than start over. Changing the domain mints a NEW token. Reusing it would let somebody who once verified example.com claim any other domain later without touching its DNS — the old record is still sitting there and only the value is compared. The domain is changeable at any time, and removable. Fixing it once set was considered and rejected: adding or moving an address is a proxy entry, a certificate and one line in trusted_domains. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
75b4a47768 |
Stop mailing the initial admin password, hold it in the panel until noted
The password no longer travels through third-party mail servers or sits forever in an inbox: it moves onto the instance (encrypted, same as nc_admin_ref) and shows on the dashboard until the customer clicks "Notiert", which deletes it for good. |
|
Claude
|
30a80b6c15 |
Report the estate from the database, and print the traffic that is sold
Three console pages were fiction. The front page claimed 42 customers, 39 instances, four hosts named pve-fsn-1..3 and €7,842 a month, over a twelve-month growth curve; the instance list held seven invented machines; the revenue page reported churn and a trend for a business with no recorded history. All of it was hard-coded. It read like a running company and measured nothing. They now read the database. Two figures are gone rather than approximated — the revenue trend, which needs a monthly history nobody records, and churn, which needs a base the data cannot supply. ARR stays, labelled as the projection it is. The green "all systems normal" badge is computed from the notice list instead of asserted, and the notices themselves come from failed runs, hosts reporting errors or gone quiet, and monitoring that is down. Host load is the one number that had to agree with something else: placement counts the VM disk allocation, ignores a failed instance that never got a VM, and subtracts the host's reserve. A dashboard doing its own arithmetic would show a host as comfortable while orders were already being refused on it, so it uses the host's own accounting — with the filter moved into a scope both share, and the sum preloaded so listing hosts stays one query. The instance list drops the Nextcloud version column: that version is not recorded anywhere, and a column filled with a plausible number is worse than no column. Statuses the lifecycle writes but nobody had translated no longer render as "admin.status.failed". The price sheet also gains the included traffic, which the catalogue has always carried and the page simply never printed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|
|
nexxo
|
52b41bb0d5 |
fix(billing): a paid order opens a contract, and provisioning obeys it
The pipeline re-resolved config('provisioning.plans') by order.plan, so the
subscription snapshot protected a customer's price but not their machine:
shrinking a plan resized an existing customer's VM on its next run. Nothing
created a subscription either, so closing this meant opening the contract at
purchase and pointing provisioning at it.
- OpenSubscription freezes the catalogue onto a subscription when a checkout
is paid; StartCustomerProvisioning calls it inside the order transaction.
- CustomerStep::plan() reads the frozen snapshot. ValidateOrder and
ReserveResources fail closed with no_subscription rather than falling back
to the catalogue, which is the bug itself.
- template_vmid joins the snapshot so a re-clone cannot pick up a blueprint
published after the sale. Deliberately outside FROZEN: it is how we build
the machine, not a term the customer is owed, and a dead template must be
replaceable without cancelling a contract.
- TrafficMeter reads the allowance off the contract too — cutting a plan's
traffic was otherwise enough to start throttling someone who bought more.
- The migration backfills contracts for orders that already bought something,
reconstructed from what was actually delivered where an instance exists,
and adopts an existing order-less contract instead of opening a second.
Orders paid in a currency the catalogue cannot price get none, matching the
checkout path.
price_cents stays the catalogue's NET price, which is what PlanChange
prorates against — not Order::amount_cents, which holds Stripe's GROSS total.
Reconciling the two belongs to the proof register and Stripe (phases 4/5).
Also pins STRIPE_WEBHOOK_SECRET blank in phpunit.xml: the operator's real
secret was reaching the suite from .env and rejecting every unsigned test
payload, which is why 7 webhook tests failed before any of this.
Verified in the browser: with team traffic cut from 3000 to 500 GB in the
catalogue, the customer's portal still shows 3 TB.
373 tests green. Codex review clean after three rounds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
|
nexxo
|
c1e81808a7 |
feat(traffic): meter the monthly allowance, show it, throttle instead of blocking
Customers can now see what they have used and what is left, and the service slows down rather than stopping when the allowance is gone — a slow Nextcloud gets a top-up, a dead one gets a cancellation. - instance_traffic keeps one row per instance per month. Proxmox counters are cumulative since the VM last started, so usage is the difference between two samples, and a counter that went backwards means a restart, not a refund. - Outbound is what counts: inbound is free at our providers and egress is what Hetzner's 20 TB per server applies to. - CollectInstanceTraffic samples every 15 minutes, warns once per threshold (80/95 %) rather than on every run, and at 100 % limits the VM's NIC via Proxmox — released again as soon as the customer tops up or the month rolls over. - The dashboard gets a full-width band (it throttles the service, so it is not a tile among tiles) with the top-up offer right next to the warning. - Bytes are formatted in SI units now: allowances are computed in SI, so dividing by 1024 made a 1000 GB plan read as "931 GB". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
|
|
nexxo
|
df10448e5d |
feat(portal): settings page — company profile, branding (logo+colors), cancel package, close account
- customers gain profile + branding + closed_at; instances gain cancellation fields; branding resolver (NULL -> CluPilot defaults) snapshotted into the provisioning run context - cancel package: term-end, irreversible, typed-confirm modal (R5) - close account: guarded (no active package), typed-confirm modal (R5) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
|
|
nexxo
|
42fa92cf20 |
fix(engine-b): address Codex (paid-only stripe, guest routing, absolute disk)
- Stripe: only checkout.session.completed with payment_status=paid (dedupes the paired payment_intent event and blocks unpaid async sessions). - Capture the guest IP (ConfigureNetwork) and point Traefik at the VM, not the Proxmox host, so ACME/HTTP-01 can reach Nextcloud. - Resize the disk to an absolute target (not '+…') so a retry can't double it. (Codex #4 timeout was a false positive — started_at resets per step.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
|
|
nexxo
|
406251dfde |
feat(engine-b): customer domain models + migrations
customers, orders (ProvisioningSubject, stripe_event_id unique), instances (subdomain unique, encrypted nc_admin_ref), dns_records, backups, monitoring_targets, onboarding_tasks. Host capacity: committedGb/availableGb + datacenter placement. 7 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |