Commit Graph

29 Commits (976637afb207e8e9a7ac6afc9cfdb300b2fa43a3)

Author SHA1 Message Date
nexxo 889b401faf Fix nine defects in the provisioning pipelines
The capacity park could not survive its own first poll. ReserveResources
polls every 120 s and declared a maxDuration of 60, and RunRunner measures
a step's budget from a started_at it deliberately does not reset on a poll
— so every re-entry was ruled timed out before the body ran, a timeout
consumes an attempt, and five of them failed a paid order in about six
minutes. The fourteen-day promise, the console's capacity queue and the
"go and buy a server" workflow were unreachable code. maxDuration is now
the whole park plus one poll interval, and a test drives a park through
the runner rather than calling execute() directly, which is why this was
invisible.

Every guest occ call ran as root. `docker compose exec` defaults to root
and the Nextcloud image's console.php exits 1 unless the caller owns
config/config.php, so all five call sites were failing on every instance.
There is now one builder, App\Support\NextcloudOcc, and a test refuses a
second: nothing in app/ may spell the invocation out by hand. deploy/
update.sh had learned this for our own container and nobody carried it
across.

RunAcceptanceChecks was terminal on the first no. certReachable() answers
false for a connect timeout as readily as for a missing certificate, so
one bad second ended a finished, certified Nextcloud as a failed order
with the instance released. The probes retry now and the run still fails
for good with the probe's own reason once the budget is spent; the two
facts a retry cannot change stay terminal.

The in-flight guard asked whether anything was running, not whether it
would do the work. App\Provisioning\WorkInFlight asks the second question,
reading the step lists the runner executes: a domain proven during a
restart is routed, and a storage pack booked during an address run is
delivered instead of charged monthly and forgotten.

The address pipeline ran its steps in the wrong order for the direction it
exists for. Nextcloud before the router in `address` and `plan-change`, as
in `customer`, because a run that got the certificate and then failed left
the customer's own domain serving an untrusted-domain error under a valid
certificate while the portal called it live. Safe in both directions; the
reasoning is written above the pipeline.

A stale guest_ip could never be corrected. ConfigureDnsAndTls compared the
hostnames but never the backend, and nothing re-read the address after the
build. It now records and compares `routed_backend`, and ConfigureNetwork
joins the `restart` pipeline — a cold boot is what moves a DHCP lease — and
asks for the address to be re-applied when the guest has actually moved.

Also: HetznerDnsClient::upsertRecord read only the first page of a zone
that pages at 100, so past a hundred records it created a second A record
for the same name and the cloud was up about half the time; the console's
retry left a live customer's order in `provisioning` for ever for any
maintenance pipeline; and a revived run's first pass burned an attempt on a
timeout that had already happened, because the clause written for it used
`??=`.

Seven test sites wrote config('provisioning.plans.*'), a key that has not
existed since the catalogue moved into the database. Every write was a
no-op, so each test proved something other than its name: the missing
template, the two snapshot regressions and the grandfathered-price scenario
are now actually constructed, ValidateOrder's reasons are asserted rather
than only its type, and the end-to-end run checks that the quota step
delivered something.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 01:34:55 +02:00
nexxo 336ca9d88c Make host onboarding survive the machine it runs on
tests / pest (push) Failing after 7m52s Details
tests / assets (push) Successful in 20s Details
tests / release (push) Has been skipped Details
Nine defects in the host pipeline, every one of which either stopped an
install or let a host reach `active` while it could not do its job.

The install-stoppers:

- InstallProxmoxVe hard-coded `bookworm`. A server ordered today boots
  Debian 13 trixie, so it added the PVE 8 repo and keyring to a trixie
  base. The codename now comes from /etc/os-release, the suite and keyring
  are looked up per release, and an unknown one fails naming what it found
  instead of guessing. Keys land in /usr/share/keyrings with Signed-By
  rather than in the deprecated trusted.gpg.d, where they would vouch for
  every other repository on the machine too.

- keyLogin() dialled wg_ip whenever it was filled, but wg_ip is persisted
  inside the allocation lock BEFORE the handshake is ever proven. Attempt
  1 reserved the address and failed at the handshake; every later attempt,
  including a console Retry, died connecting to a tunnel that did not work
  — before reaching the only code that could repair wg0.conf. Recovery
  meant nulling hosts.wg_ip by hand. SSH now uses the tunnel only once the
  `wg_peer` breadcrumb proves a handshake succeeded, which also rescues
  the same shape in RebootIntoPveKernel, ConfigureProxmox,
  CreateAutomationToken and SecureHostFirewall. Not a hole: 22 on the
  public IP is open until SecureHostFirewall runs, that step runs last,
  and by then the tunnel is necessarily proven. And nothing validated the
  hub key/endpoint, so renderConfig() would emit `PublicKey = ` — now
  refused before anything is installed, allocated or written.

- `systemctl enable --now wg-quick@wg0 || wg-quick up wg0` brought the
  interface up without the enablement, so the tunnel did not come back
  after the step-6 reboot; on the next attempt both halves failed with
  "wg0 already exists" and the step retried forever on a working tunnel.
  The three states are handled separately now, and a changed wg0.conf is
  actually applied instead of only written.

- RebootIntoPveKernel removed the Debian kernel with `|| true`, ignored
  update-grub's exit status, and rebooted. A /boot that filled up during
  the full-upgrade could leave a machine only the provider's console can
  reach. The pve kernel image, its initramfs and update-grub's exit status
  are all proven first; anything missing fails loudly and does not reboot.

- ConfigureProxmox ran `ip link show vmbr0` and threw the result away
  (its comment claimed it recorded the absence; it recorded nothing), then
  repeated a rm -f, and always advanced. Proxmox on top of Debian does not
  create vmbr0, so onboarding reported `active` with no bridge. It now
  fails with the default route in the message and deliberately does not
  build the bridge over the primary NIC from a remote shell.

The things that were quietly inert:

- Proxmox applies a guest's firewall rules only while the firewall is
  enabled at datacenter level, and it ships disabled — so applyFirewall()'s
  "80/443 only" rules were inert on every customer VM. Enabling it blindly
  is the opposite trap (input policy defaults to DROP, node firewall
  defaults to on, and its management ipset is seeded from the PUBLIC
  subnet, not the tunnel), so: policy ACCEPT and the node firewall off
  first, master switch last, read back to confirm. nftables stays the one
  owner of the host's input policy.

- role_privs lacked Sys.Modify, which POST /cluster/backup requires
  (pve-manager's PVE/API2/Backup.pm), so RegisterBackup 403'd and failed
  EVERY customer run. And `pveum role add … || true` only ever applied the
  privilege list on the run that created the role — it converges now, above
  the token short-circuit so a replay reaches it.

- The nftables ruleset dropped all ICMP and ICMPv6. On a real host that
  breaks IPv6 outright once the neighbour cache expires and black-holes
  large transfers through PMTUD. Both families accept the types they need,
  policy drop stays, and a DHCP client reply is allowed so a rebind cannot
  lose the IPv4 address.

- api_token_ref used the `encrypted` cast, i.e. APP_KEY, against
  SecretVault's own written rule. One rotation would have made every host's
  Proxmox token undecryptable at once. Moved onto SecretCipher with a
  migration that leaves any value it cannot read exactly as it is and says
  so, rather than destroying a credential that exists nowhere else.

- Every plan version points at template_vmid 9000 and nothing created or
  verified it, so the first paid order died in CloneVirtualMachine after
  the customer had paid. VerifyVmTemplate reports that during onboarding
  instead. It does not build a template: what goes into the golden image is
  a product decision.

Three tests that proved nothing, fixed: `ran(…emergency-open-firewall.sh)`
only ever matched the `chmod 700` above it, because putFile() is not
recorded — the script's contents are now asserted, including that nothing
in it reopens the firewall on a timer or in the background. The port policy
was tested one-sidedly, so a mutation adding `tcp dport { 8006 } accept`
passed the suite; the ruleset is now read as a list of what it opens to an
unrestricted source. And the end-to-end test says in writing that it proves
sequencing only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 00:54:16 +02:00
nexxo 5b63fdb86c Deliver the storage a customer actually buys
A booked storage pack was a row in `subscription_addons` and nothing else:
priced, frozen, charged every month, and delivered nowhere. Nextcloud's
quota came from `instances.quota_gb`, which is the package alone; the disk
was sized for the package alone; and on the one occasion a disk did grow —
a plan upgrade — the guest never saw it, because nothing stretched the
partition or the filesystem over the new space.

One authority for the allowance. StorageAllowance adds the package and the
booked packs, and everything that needs the figure asks it: the quota step,
the downgrade check, the portal, the console, the repair command. It is
DERIVED rather than stored — a column would have to be rewritten by booking,
cancelling, granting, a grant expiring and a plan change, and the day one of
those paths forgot it would be silently wrong in whichever direction costs
somebody money. `quota_applied_gb` keeps its own meaning: what the guest was
last actually told, which is how a machine with the allowance enforced is
told from one where the figure has only ever been a row in our database.

Buying it delivers it. BookAddon asks ApplyStorageAllowance for a run on
booking AND on cancellation; the new `storage` pipeline grows the virtual
disk to the allowance plus the package's own overhead (read off the package
— 20/40/50/100 GB on the four catalogue packages — never a ratio invented
here), then GrowGuestFilesystem makes the guest see it, then the quota is
applied. No cold boot is involved: the data disk is scsi0 and Proxmox's
resize on a running guest is a qemu block_resize, so the capacity reaches
the guest while it runs. The new step rescans, growparts and grows the
filesystem with the tool its type actually needs — ext2/3/4, xfs, btrfs —
and fails loudly on one it does not know rather than handing it to resize2fs
and hoping. Idempotent end to end: NOCHANGE from growpart is not a failure,
and every tool here exits 0 when there is nothing left to grow.

The same step now runs in the plan-change pipeline, where an upgrade used to
stop one step short of the customer.

A downgrade blocked by data gets a way out. The block stays — it is correct
— but the check now measures against the target package PLUS the packs the
customer already owns, and reports the numbers behind the refusal: what is
stored, what would be allowed, how much has to go, and how many packs would
cover it instead. The portal offers both routes: book exactly those packs
(confirmed in a modal, R23), or delete data and have the fill level measured
on demand rather than waiting for the nightly sampler. That reading is taken
by DiskUsageProbe, which CollectInstanceTraffic now uses too, so there is one
notion of "how full is it" and not two.

A plan change keeps booked packs. They were paid for separately and have
nothing to do with which package the customer is on; the new package's disk
and quota are sized with them included.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 19:13:10 +02:00
nexxo d76de5ada3 Restart a machine, enforce the quota that was sold, end a route that ended
Three places where the product said one thing and did another.

**Nothing could restart a virtual machine.** ProxmoxClient had startVm and no
way to stop anything, so `restart_required_since` — set by every plan change
that grows a running guest — could only ever be cleared by a resize step that
happened to find the machine stopped, which nothing in the product could
arrange. A paid upgrade's cores and RAM could stay unreached for the life of the
contract, and the cloud page's "Neu starten" was a toast.

There is now a `restart` pipeline: shutdown, start, wait for the agent, confirm.
A shutdown and a start rather than a reboot, because only a cold boot makes qemu
read the VM definition again — a reset would take the machine round and bring it
back exactly as small as it was. The shutdown is a REQUEST, bounded at ten
minutes, and there is no escalation: ProxmoxClient deliberately does not expose
Proxmox's forceStop, because the guest is a Nextcloud and cutting power to a
database mid-write to apply a CPU change trades an inconvenience for a restore
from backup. A guest that ignores ACPI for ten minutes fails the run instead, and
a person decides what happens to it.

`restart_required_since` is cleared by the last step, from what the RUNNING guest
reports (`cpus`/`maxmem`) — not by the button, and not by the run merely
finishing. A machine that comes back smaller than it was sold fails the run.

Started by App\Actions\RestartInstance, which re-checks authorisation itself
rather than trusting a hidden button: an operator holding the new
`instances.restart` may restart anything, a portal user their own machine and
nothing else. Refused while another run is in flight against the same order.
Confirmed in a modal on both sides (R23), addressed by uuid (R11).

**Existing instances had no storage quota.** ApplyStorageQuota reaches new and
changed machines; everything built before it kept the whole disk whatever it had
paid for, and nothing recorded which was which. `instances.quota_applied_gb` is
now what the guest was actually told, written by the step only after the occ call
succeeds — so "sold" and "enforced" are separate facts and can be compared.
`clupilot:apply-quotas` sweeps the estate: one `quota` run per instance through
the same step the pipelines use, skipping what it should and saying why, a no-op
once an allowance is enforced, and inert under --dry-run. Deliberately NOT
scheduled — a nightly sweep would quietly cover for the pipeline step the day it
stopped working, which is how the original hole survived this long.

**A Traefik router was never torn down.** TraefikWriter::remove() had no caller
anywhere. What was missing was the moment: ConfirmCancelPackage wrote a date into
`service_ends_at` and nothing ever went back to it, so every route this platform
ever wrote was written for good — pointing at a guest address the host is free to
reassign. `clupilot:end-due-services` (hourly) keeps that appointment through
App\Actions\EndInstanceService. "Ended" means cancellation_scheduled AND
`service_ends_at` passed, never the day a cancellation is scheduled: that
customer has paid to the end of the term and is working in it. The DNS record
goes with the route — it is in our zone, pointing at a host that serves other
people, which is the shape of every subdomain takeover there has ever been. The
customer's own domain is in their zone and is not ours to touch; the virtual
machine is left alone, because the cancellation flow promises a data export first
and destroying disks is not this change's decision.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 18:28:28 +02:00
nexxo 8f2252e81b Book a downgrade for a date that cannot move, and sell a module once
Two ways of charging a customer for something they did not get.

A downgrade was due when `subscriptions.current_period_end` said so, and
Stripe pushes that column forward on every renewal. So a downgrade booked in
March was deferred to the end of April by April's renewal, and to the end of
May by May's — each time by a whole term, and each time the customer was
billed again for the package they had asked to leave. The due date was being
re-derived from a moving target on every scheduler tick.

It is decided once now, at the moment the customer decides, and stamped onto
the contract: `pending_plan` and `pending_effective_at`, the two columns that
have been in the schema since the first migration and were written by nothing
at all. The shop stamps them, PlanChange reads them instead of the period end
when a move is already booked, and clupilot:apply-due-plan-changes finds
contracts by the stamped date rather than orders by a date it recomputes.

The order stays what it always was — the customer's own record of the request,
what the cart shows them and what they remove to change their mind — so it is
still consumed with the change, and removing it unbooks the change. What it no
longer does is decide when. A second downgrade replaces the first rather than
queueing behind it; an upgrade clears one, or it would come due months later
and undo the bigger package; a contract cancelled before the date takes the
booking with it; and a customer with no contract can no longer place a
downgrade at all, which was accepted before and could never be carried out.

A booked change is now visible where the contract is read: on the plan card in
the portal, because the cart entry disappears when it is paid for and the
booking does not, and on the customer row in the console, because that is
where an operator answers "what is this customer on".

The second defect is the same money in one step: BookAddon guarded duplicates
per ORDER — the unique index on (order_id, addon_key) — so two orders for the
same module on one contract both went through. A stale tab or a double click
bought priority support twice.

But storage is deliberately sold in packs, and AddonCatalogue sums the
quantities for exactly that reason, so "refuse the second" is only right for
some of them. Which is which is declared beside each price in
config/provisioning.php (`sold_as`), because it is a commercial decision about
each module and not something to infer from a key: storage is a quantity;
off-site backups are on or off, support is prioritised or it is not, Collabora
Pro is one licence, and a machine answers to one own domain. An undeclared
module counts as an entitlement — the cheaper of the two mistakes — and a test
refuses to let one ship undeclared.

Enforced in the action, where the portal, the console's grant screen and a
webhook all pass, with the sentence the customer is shown rather than a
developer's. A retried webhook for the SAME order still gets its one booking
back; that is one purchase arriving twice, not two purchases. The portal stops
offering what would be refused — booked, or already waiting in the cart — and
goes on offering the pack.

What is missing is on the Stripe side and is not invented here: nothing moves
the subscription's item onto the new Price, so the renewal after a plan change
still bills the old plan at Stripe. StripeClient only creates and archives
catalogue objects, and swapping a price also needs the subscription ITEM id,
which nothing stores.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 17:51:37 +02:00
nexxo 34874adec3 Apply a bought plan change instead of only pricing it
tests / pest (push) Failing after 8m6s Details
tests / assets (push) Successful in 21s Details
tests / release (push) Has been skipped Details
PlanChange could say what a move would cost and whether it was allowed, and
that was all it could do. Billing::purchase() wrote an upgrade order and
nothing ever consumed it: same snapshot, same machine, same quota. A customer
could pay for a bigger package and receive nothing.

ApplyPlanChange is now the single place a change lands — it moves the contract
onto the target's current version, writes one register row, settles the custom
domain, and starts a run that resizes the machine. Applying the same order
twice is a no-op, enforced by a unique event key rather than by a check two
callers could both pass.

Two things a plan change must not do, and now does not. A disk is never shrunk
— Proxmox cannot, so the QUOTA shrinks instead, which is what was sold anyway.
And a live machine is never rebooted as a side effect: cores and RAM are
written, and where they need a restart the instance says so where an operator
and the customer can both see it.

The storage allowance also joins the build pipeline. ApplyStorageQuota was
written for plan changes and ran only there, so a brand-new customer still got
no quota at all — quota_gb reached the instance row and stopped, and every
package delivered the whole disk. The test guarding that pipeline only counted
its steps, which is how a list missing the one step that makes a package's
storage real stayed green for its whole life. It now names the step.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 17:18:55 +02:00
nexxo 712803edd6 Serve the custom domain, not just announce it
A verified custom domain was reported as the customer's address by
Instance::address(), by the portal and by the credentials mail while
nothing on the platform routed it: the Traefik router's rule was
hard-coded to {subdomain}.{zone}. The address a customer was handed
answered nothing, and a withdrawn domain stayed in Nextcloud's
trusted_domains forever, because the only thing that ever wrote either
was the initial provisioning run.

- TraefikWriter::write() takes a LIST of hostnames under one stable
  router name — the platform address always, the verified domain as
  well when there is one. One file per instance, so a withdrawal is a
  rewrite rather than a second thing somebody has to remember.
- ConfigureDnsAndTls records the custom domain's certificate instead of
  waiting on it: it depends on an A record in the customer's own zone,
  which may never appear, and must never fail a run. The platform
  address keeps its 840s deadline. instances.domain_cert_ok tells
  "proven" apart from "answering", and the portal now says which.
- ConfigureNextcloud deletes trusted_domains 2 when there is no verified
  domain, so a withdrawn one stops being trusted.
- New `address` pipeline (those two steps) plus ReapplyInstanceAddress,
  which starts one against the order and refuses to start a second while
  any run is in flight. The route is rewritten when the hostname list
  differs from what the router carries — not on route_written, which
  would short-circuit exactly the case a re-apply exists for.
- Triggered where the address changes: verification flipping either way
  in clupilot:verify-domains, the customer's own domain page, and
  CustomDomainAccess::deactivate() on a package downgrade.
- A maintenance run no longer condemns its subject: an address run that
  failed used to mark the order failed and release the live instance
  with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 16:44:52 +02:00
nexxo f8874f32ea Put customer instances on their own domain, and say which domains are ours
tests / pest (push) Failing after 8m1s Details
tests / assets (push) Successful in 20s Details
tests / release (push) Has been skipped Details
── The zone ────────────────────────────────────────────────────────────────
A fresh install now addresses customer instances under clupilot.cloud rather
than clupilot.com. Not branding: a Nextcloud is third-party software that
strangers sign in to, and on the same registrable domain as the portal it
shares cookie scope, document.domain and CAA with it. The same reason
googleusercontent.com and vercel.app exist.

Existing installations keep whatever the console says — the zone has been a
setting all along. `clupilot:check-zone <zone>` reports what a change would
touch before anything is touched: the DNS records, the monitoring targets,
the certificates and Nextcloud's trusted_domains all carry the old name and
the setting reaches none of them. With no instances in service it says so and
stops. Changing it later is a migration with an outage, and the command says
that too.

── Which addresses are ours ────────────────────────────────────────────────
A new public page at /sicherheit: the domains that belong to us, the rule for
reading an address bar, what we never do, and — the part that matters — what
to do if you have already typed your password into a copy of our sign-in
form. Change it, end every other session, change it wherever else you use it,
tell us. In that order.

Reachable without an account, deliberately: somebody who has just given their
password away is signed in nowhere, and the page they need cannot be behind
the thing they lost.

Linked from the three places a person is when the question comes up: every
mail footer (the mail is the vector), the sign-in form (this is the page a
phishing kit copies, so the real one says which host you are on), and the
site footer.

The domain list is DERIVED from configuration and keyed by what each domain
is for. A list typed into a page would eventually tell a customer that a
phishing domain is one of ours, or that one of ours is not — the same
two-sources-of-truth failure that had the price sheet promising an address on
a domain the company does not own. The operator console is deliberately
absent from it: naming it on a public page tells an attacker where to aim.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 16:03:13 +02:00
nexxo ed4167eba3 Close SSH and the Proxmox UI to the world once the tunnel is proven up
tests / pest (push) Failing after 7m48s Details
tests / assets (push) Successful in 20s Details
tests / release (push) Has been skipped Details
2026-07-28 23:57:07 +02:00
nexxo 2b1989f53d Move host DNS names off the public zone into vpn-dns's internal hostsdir
tests / pest (push) Failing after 8m2s Details
tests / assets (push) Successful in 19s Details
tests / release (push) Has been skipped Details
2026-07-28 23:44:31 +02:00
Claude 457eeeeaef Serve the console inside the tunnel, without publishing the internal network
tests / pest (push) Successful in 10m12s Details
tests / assets (push) Successful in 21s Details
tests / release (push) Successful in 5s Details
The owner's phone shows a blank page on the VPN. That page is the reverse proxy
refusing the request, and it refuses because the phone is not on the tunnel: the
client config routes only the management subnet, so a request to the console's
public hostname leaves over the mobile network and arrives from a carrier
address. It is also why the peer reads "last contact: never" — WireGuard only
handshakes when it has traffic to send, and nothing is ever routed in.

The obvious fixes are both wrong. Adding the server's public address to
AllowedIPs routes the WireGuard endpoint into the tunnel it is trying to
establish. A public DNS record pointing at 10.66.0.1 publishes the internal
subnet to anyone enumerating the domain — the owner raised that himself, and he
was right.

So the console is served INSIDE the tunnel instead. A small resolver and a small
gateway share the hub container's network namespace, which is where wg0 lives,
and answer on 10.66.0.1 directly. A client therefore needs no route beyond the
subnet it already has, the host's docker bridge is never exposed, and the
request reaches the application with its real 10.66.0.x source — which is what
the console's own allowlist checks. The gateway reuses the certificate the
public Caddy already renews: a certificate is bound to the name, not to the
address serving it, so nothing new is issued and nothing internal reaches a
public zone.

Most of this commit is the arithmetic of not lying about it. Review found
fourteen ways the arrangement could report itself working while it was not:
enabling the compose profile after the services were started rather than before;
starting a gateway with empty certificate paths, which can only crash; treating
"container created" as "container running"; carrying a stale readiness marker
through a failed restart; reusing the previous hostname's certificate after a
rename; leaving the profile enabled when the hostname was cleared, in a script
that exits early precisely when nothing else would fix it; reading the renewal
timestamp as a user who cannot traverse Caddy's storage; and a `find | head`
that aborts the whole installer under pipefail the moment two issuers hold a
certificate for the same name.

One of them was mine and worth naming: to let the container read the
certificate, I had made the TLS private key world-readable. On a multi-user host
that hands the console's identity to every local account.

The client is told about the resolver only when both services are confirmed
running, because a config naming a resolver that does not exist takes the
device's entire name resolution with it for as long as the tunnel is up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-27 10:14:33 +02:00
nexxo 6387c747d0 feat(billing): the plan catalogue becomes three tables, and config stops selling
Plans lived in config/provisioning.php, where the owner cannot reach them and
where a plan has no history. They now live in plan_families / plan_versions /
plan_prices — a name that never moves, what the plan WAS at a point in time,
and a price per version and term with its own Stripe Price id.

- Availability is computed on every read (available_from <= now < until,
  half-open, UTC) plus a per-family sales kill switch. Nothing schedules a plan
  into or out of sale: a job that fails to run is a plan that silently
  misbehaves.
- Overlaps crash rather than resolve. currentVersion() uses sole(), and
  scheduling takes a lock on the family and rejects an overlapping window —
  two versions on sale at once would decide a customer's price by row order.
- A version is frozen from publication, not from first sale, and so is its
  price: a checkout is not instant, and an amount edited between the session
  opening and the webhook landing would contract someone at a price they were
  never quoted. Repricing publishes a new version, as Stripe requires anyway.
- Neither a published version, its price, nor a family with customers can be
  deleted, and a family key cannot be renamed. All of those would null the
  provenance off existing contracts.
- Subscriptions and orders record plan_version_id. A historical reference
  resolved by plan NAME hands back today's terms, which is the split-brain one
  level up. A checkout that carried its version is honoured even after the
  window closes — they paid for what they were shown.

Switched atomically and failing closed: config('provisioning.plans') and
plan_features are gone, and nothing falls back to them. A fallback would
resurrect a plan the owner had just switched off. The seed lives in the
migration, and PlanCatalogueTest pins what the config catalogue sold as the
shadow comparison. `php artisan plans:check` reports overlaps, gaps and
missing prices before a customer finds them.

Contract-backed displays, which were reading the live catalogue:
seat limits, the cloud card, the plan card, and admin MRR — the last three now
divide a yearly contract down via Subscription::monthlyPriceCents(), since all
three label the figure per month.

Two recovery gaps closed along the way: the order now commits before the
contract is opened (so a contract failure can never erase the record of a
payment), a Stripe retry repairs a missing contract and restarts a run stranded
with no_subscription, and TickProvisioning sweeps runs left pending by a crash
rather than only running/waiting ones.

402 tests green. Codex review clean after thirteen rounds. Verified in the
browser: portal, billing and console render unchanged off the new catalogue.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 12:05:56 +02:00
nexxo 52b41bb0d5 fix(billing): a paid order opens a contract, and provisioning obeys it
The pipeline re-resolved config('provisioning.plans') by order.plan, so the
subscription snapshot protected a customer's price but not their machine:
shrinking a plan resized an existing customer's VM on its next run. Nothing
created a subscription either, so closing this meant opening the contract at
purchase and pointing provisioning at it.

- OpenSubscription freezes the catalogue onto a subscription when a checkout
  is paid; StartCustomerProvisioning calls it inside the order transaction.
- CustomerStep::plan() reads the frozen snapshot. ValidateOrder and
  ReserveResources fail closed with no_subscription rather than falling back
  to the catalogue, which is the bug itself.
- template_vmid joins the snapshot so a re-clone cannot pick up a blueprint
  published after the sale. Deliberately outside FROZEN: it is how we build
  the machine, not a term the customer is owed, and a dead template must be
  replaceable without cancelling a contract.
- TrafficMeter reads the allowance off the contract too — cutting a plan's
  traffic was otherwise enough to start throttling someone who bought more.
- The migration backfills contracts for orders that already bought something,
  reconstructed from what was actually delivered where an instance exists,
  and adopts an existing order-less contract instead of opening a second.
  Orders paid in a currency the catalogue cannot price get none, matching the
  checkout path.

price_cents stays the catalogue's NET price, which is what PlanChange
prorates against — not Order::amount_cents, which holds Stripe's GROSS total.
Reconciling the two belongs to the proof register and Stripe (phases 4/5).

Also pins STRIPE_WEBHOOK_SECRET blank in phpunit.xml: the operator's real
secret was reaching the suite from .env and rejecting every unsigned test
payload, which is why 7 webhook tests failed before any of this.

Verified in the browser: with team traffic cut from 3000 to 500 GB in the
catalogue, the customer's portal still shows 3 TB.

373 tests green. Codex review clean after three rounds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 11:10:00 +02:00
nexxo 460fac01b1 fix(billing): upgrade or downgrade is decided by rank, not by price
tests / pest (push) Successful in 6m51s Details
tests / assets (push) Successful in 20s Details
tests / release (push) Has been skipped Details
With grandfathered prices the comparison inverts: a business plan bought when
it cost less than today's team plan would have treated a move to team as an
upgrade — charged immediately, while the customer loses resources. The plan's
rank is frozen with the rest of the snapshot and decides the direction; prices
only decide the amount.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:52:28 +02:00
nexxo 76510a59a3 fix(billing): VAT follows the customer, not a global setting
tests / pest (push) Successful in 7m28s Details
tests / assets (push) Successful in 21s Details
tests / release (push) Has been skipped Details
Codex was right that this could misstate real charges: an EU business with a
VAT ID registered in another country is billed under reverse charge, and we
were adding 20 % Austrian VAT to their total anyway. TaxTreatment resolves it
from the customer's VAT ID, and the whole page — cart, plan cards, add-on
cards — now states one treatment instead of contradicting itself.

Explicitly NOT handled: cross-border sales to private individuals, which are
taxed at the buyer's national rate under OSS. That needs a maintained rate
table and a tax adviser, not a guess, so those fall back to the domestic rate —
over-collecting rather than under-collecting.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:30:17 +02:00
nexxo a87a4f512d feat(billing): one plan change at a time, and every price says what it is
Two contradicting plan upgrades sat in the cart at 799 and 399 € — no checkout
could resolve which one the customer meant. Choosing another now replaces the
pending one and says so; add-ons still stack, because buying 200 GB as two packs
is a sensible thing to want.

Every price now states net or gross and how often. The cart shows net per line
with "pro Monat" or "einmalig", then subtotal, VAT and gross — and separates
the monthly recurring amount from a one-off traffic top-up sharing the same
cart, because those are two different commitments. The rate is configurable
(CLUPILOT_TAX_PERCENT, 20 % default) since it follows the seller's country.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:23:49 +02:00
nexxo 5db2f7fda7 feat(hosts): give each host a DNS name under the public zone
fsn-01.node.clupilot.com, numbered per datacenter and never reused — removing a
host must not renumber its neighbours onto its name, so the number is stored
rather than derived.

The record points at the host's WireGuard address, not its public IP: the name
exists so an operator can reach a host by name over the VPN, and publishing a
Proxmox host's public address would hand every scanner a target, which is the
one thing this network design avoids.

DNS is convenience, not a prerequisite: a failure logs an event and lets the
onboarding finish rather than stranding a host that is otherwise ready.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 23:49:18 +02:00
nexxo c1e81808a7 feat(traffic): meter the monthly allowance, show it, throttle instead of blocking
Customers can now see what they have used and what is left, and the service
slows down rather than stopping when the allowance is gone — a slow Nextcloud
gets a top-up, a dead one gets a cancellation.

- instance_traffic keeps one row per instance per month. Proxmox counters are
  cumulative since the VM last started, so usage is the difference between two
  samples, and a counter that went backwards means a restart, not a refund.
- Outbound is what counts: inbound is free at our providers and egress is what
  Hetzner's 20 TB per server applies to.
- CollectInstanceTraffic samples every 15 minutes, warns once per threshold
  (80/95 %) rather than on every run, and at 100 % limits the VM's NIC via
  Proxmox — released again as soon as the customer tops up or the month rolls
  over.
- The dashboard gets a full-width band (it throttles the service, so it is not
  a tile among tiles) with the top-up offer right next to the warning.
- Bytes are formatted in SI units now: allowances are computed in SI, so
  dividing by 1024 made a 1000 GB plan read as "931 GB".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 23:33:47 +02:00
nexxo 5301ce4309 feat(monitoring): Uptime Kuma support via a REST bridge + resilient monitoring policy
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.

Verified end-to-end against a real Uptime Kuma 1.x:
  create -> id, create again -> same id (idempotent), list, status, delete
  and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.

Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
  continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
  monitor legitimately reports 'pending' — confirmed in the e2e run); it records
  the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:04:58 +02:00
nexxo f06f00c428 chore(config): documented env placeholders + SSH keys from file paths
- .env.example gains a full, commented CluPilot operations block (Stripe, Hetzner
  DNS, Traefik, WireGuard hub, SSH identity, monitoring, prod mail) with a note
  that everything is optional for local dev
- config/provisioning.php: CLUPILOT_SSH_{PUBLIC,PRIVATE}_KEY_PATH read the key
  from a file (a multi-line PEM cannot live in .env), falling back to inline vars
- documents that Uptime Kuma's REST API is read-only (monitor CRUD is Socket.IO),
  so it needs a bridge rather than being drop-in for the generic REST client

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 19:37:25 +02:00
nexxo df10448e5d feat(portal): settings page — company profile, branding (logo+colors), cancel package, close account
- customers gain profile + branding + closed_at; instances gain cancellation
  fields; branding resolver (NULL -> CluPilot defaults) snapshotted into the
  provisioning run context
- cancel package: term-end, irreversible, typed-confirm modal (R5)
- close account: guarded (no active package), typed-confirm modal (R5)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 14:36:52 +02:00
nexxo e3ab3cb501 feat(portal): service framing + gradient line charts + billing button fix
- plans gain seats + performance class; customer views show storage/seats/
  performance/features, no raw vCPU/RAM (admin keeps specs)
- billing: scoped wire:target + wire:key so one purchase no longer spins all
- chart island: line-colour->transparent gradient for filled line datasets;
  MRR / backups / invoices bar charts converted to gradient line

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 14:18:08 +02:00
nexxo 515f26234e feat(portal): billing page — current plan, upgrades, extra storage, add-ons
New /billing page + nav; plans gained price_cents, storage_addon + addons
catalogue in config. Purchases create a pending Order intent (fulfillment
mocked). Dashboard storage upsell links here. DE+EN. 6 tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:36:52 +02:00
nexxo 491800e09c fix(engine-b): grant VM.Backup to the Proxmox automation role
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:57:11 +02:00
nexxo 770a6cf7cd feat(engine-b): CustomerStep base + shared resource trait + config
ManagesRunResources trait (host+customer), CustomerStep base (order/instance/
plan/guest helpers), config plans + 15-step customer pipeline + dns/traefik.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:39:56 +02:00
nexxo a6a41719bb fix(engine): long SSH command timeout; wg peer removal on privileged worker
- PhpseclibRemoteShell sets a command timeout (default 2000s) so apt
  full-upgrade/install don't hit phpseclib's ~10s default and fail.
- Host removal dispatches RemoveWireguardPeer to the provisioning queue, which
  runs in the worker that owns wg0 (the web container can't manage WireGuard).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:01:02 +02:00
nexxo 00b8897b9d fix(engine): compute default WireGuard hub IP from the CIDR network
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:49:43 +02:00
nexxo 18d1ed2424 fix(engine): derive WireGuard hub IP from the configured subnet
hub_ip is now defined in config (default: subnet .1), so a custom
CLUPILOT_WG_SUBNET no longer pings the wrong handshake target.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:38:17 +02:00
nexxo 4f724d57c9 feat(engine): orchestrator core (state machine + tick + lock)
StepResult, ProvisioningStep contract, PipelineRegistry, RunRunner (per-run
lock, advance/retry/fail, backoff, timeout, append-only events + Reverb
StepAdvanced), AdvanceRunJob (provisioning queue), minutely Tick, admin.runs
channel. 21 tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 09:54:52 +02:00