Commit Graph

48 Commits (c119e8d96e2cf38b2149c4424bb2ae4b06f3df53)

Author SHA1 Message Date
nexxo 0560ae743d feat(billing): Stripe owns the billing cycle, we own capability
A Product per plan family and a Price per priced row of a published version,
plus the four subscribed webhook events that were arriving and being ignored.

`stripe:sync-catalogue` mirrors the catalogue. Idempotent twice over: a stored
id is skipped, and each call carries an idempotency key derived from our own
row, so a crash between Stripe creating a Price and us recording its id gives
back the same object rather than a second one. That matters more here than
usual — a Stripe Price cannot be edited or deleted, so a duplicate is
permanent. Drafts are not synced at all: a version that has promised nothing
has no business in a price list. --dry-run shows what would be created.

The webhook now handles invoice.paid, invoice.payment_failed,
customer.subscription.updated and customer.subscription.deleted. It never
re-derives an amount: the invoice is the authority for what was charged, and
recomputing it from our catalogue would produce a second, disagreeing answer.
A failed payment is recorded and nothing else — Stripe runs the dunning
schedule, and cutting a customer off on the first failure would punish an
expired card as though it were a refusal to pay.

Only a cycle renewal moves the term on. Stripe also sends paid invoices for
prorations and manual charges, and the checkout's own invoice — which is
already in the register as the purchase, and would otherwise double every
customer's first payment.

Stripe does not guarantee delivery order, which is the source of most of the
care here:
- state changes are judged and written in one transaction with the row held,
  so two deliveries cannot both pass a check made on a stale copy;
- an older event never overwrites a newer one, with a rank breaking ties
  between events sharing a second — their timestamps have one-second
  resolution, and a failed attempt and its successful retry routinely do;
- a term is never shortened, and an ended contract is never revived;
- an event for a contract that does not exist yet is HELD and replayed the
  moment it appears, rather than acknowledged and forgotten. A cancellation
  dropped that way would leave us serving someone who had already left. Held
  rows that never match are pruned after a week.

Register entries are keyed by what they are about — the invoice, the attempt,
the subscription — with a unique index doing the work, because a check two
concurrent deliveries can both pass is not idempotency.

PlanChange is now documented as the preview shown before someone confirms, not
the invoice: Stripe's proration accounts for tax, existing credit and the exact
second the change lands, and ours cannot.

Not run against the live account — `stripe:sync-catalogue` creates objects that
cannot be deleted, so that is the owner's call. The dry run lists 12.

457 tests green. Codex review clean after seven rounds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 13:36:28 +02:00
nexxo 7582df3de6 feat(billing): a proof register, and modules frozen at their booked price
Two things the contract could not answer on its own: what happened, and what
the customer's whole bill is.

`subscription_records` is append-only, one row per commercial event. Flat
columns for everything searched or relied on as evidence — event, customer,
subscription, plan family and version, term, net/tax/gross, currency, tax rate,
reverse charge, Stripe ids — PLUS a versioned JSON copy of the whole snapshot.
Not JSON alone: you cannot query it, and "the amount is in there somewhere" is
poor evidence. Copied, not joined, so a row still answers after the customer,
the plan or the version have gone.

The flat columns describe the TRANSACTION. Gross is what was actually taken;
net and tax are that gross split by the rate that applied on the day. A
discount lowers the taxable amount rather than creating negative VAT, and a
free checkout is recorded as free instead of as paid in full. What was agreed
sits beside it in the snapshot, so a charge that differs from the catalogue is
preserved as a question rather than reconciled away.

The register refuses to be rewritten, in bulk as well as one row at a time —
model events do not fire for `query()->update()`, which is exactly the shape a
careless data fix takes.

`subscription_addons` carries the owner's rule: a customer's total is their
subscription plus their modules, and all of it is frozen at what was agreed.
Price protection covering only the plan would let a module quietly double for
someone who booked it two years ago. A module they have NOT booked is a sale
still to be made, at today's price — so the catalogue is consulted only for
what is not in this table. Several bookings of one module are summed rather
than reduced to one arbitrary row, or the page and the bill would disagree.

Concurrency: one order books one module, enforced by a unique index rather than
a lookup two retries can both pass; cancellation is an atomic claim; and each
booking commits with its own register entry, so a failure cannot leave a sale
without evidence and a retry that finds the booking cannot skip the event.

Verified in the browser: with the module raised from 29,00 € to 59,00 € in the
catalogue, the customer who booked it still sees 29,00 € and a total of
208,00 €, while a new customer is quoted 59,00 €.

436 tests green. Codex review clean after seven rounds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 13:07:34 +02:00
nexxo f6ddf45dc0 feat(admin): a console for creating, pricing and scheduling plans
The owner can now do from the console what only a config edit could do before:
create a plan line, draft a version, price it, publish it into a window, and
pull a plan out of the shop.

Two pages and one modal, because the catalogue has exactly two levels. The
plan list is a name, a rank and a kill switch. Everything that can be got
wrong — capabilities, prices, windows — lives on the versions, where the page
is built around the one rule that matters: a draft is freely editable, a
published version is not touchable at all. So drafting and publishing are
separate acts, and publishing says plainly that it is final.

The console refuses everything the catalogue refuses, on the form rather than
as a stack trace: an overlapping window, a window that ends before it starts,
a version with no price for a term we sell on, and — new here — a version with
no VM template, which would be bought and then fail provisioning every time.
Numbers are bounded on both sides, because a mistyped price otherwise
overflows the column and answers with a 500 instead of saying which field was
wrong. Performance class and features are picked from the keys we have labels
for; free text would be frozen at publication and shown to customers as a raw
translation key forever.

Concurrency, since two admins share one catalogue: draft numbers are allocated
under a lock on the family, publication is an atomic conditional claim, and
discarding a draft is one statement conditional on it still being a draft —
otherwise a draft published in the meantime could be deleted out from under
the customers now contracted to it.

`plans.manage` (Owner and Admin) guards the pages themselves, not only the
buttons, and the modal authorises itself — prices, drafts and unreleased plans
are not something to leave readable to anyone who types the URL.

Also fixes the shared checkbox, which ticked in the browser's own blue on
every page that used one: a native checkbox ignores text colour and needs
accent-color.

Verified in the browser: withdrawing a plan in the console removes it from the
customer's billing page immediately, and restoring it brings it back. The
public landing page carries no catalogue-driven plan list, so there is nothing
there to hide.

421 tests green. Codex review clean after six rounds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 12:41:51 +02:00
nexxo 6387c747d0 feat(billing): the plan catalogue becomes three tables, and config stops selling
Plans lived in config/provisioning.php, where the owner cannot reach them and
where a plan has no history. They now live in plan_families / plan_versions /
plan_prices — a name that never moves, what the plan WAS at a point in time,
and a price per version and term with its own Stripe Price id.

- Availability is computed on every read (available_from <= now < until,
  half-open, UTC) plus a per-family sales kill switch. Nothing schedules a plan
  into or out of sale: a job that fails to run is a plan that silently
  misbehaves.
- Overlaps crash rather than resolve. currentVersion() uses sole(), and
  scheduling takes a lock on the family and rejects an overlapping window —
  two versions on sale at once would decide a customer's price by row order.
- A version is frozen from publication, not from first sale, and so is its
  price: a checkout is not instant, and an amount edited between the session
  opening and the webhook landing would contract someone at a price they were
  never quoted. Repricing publishes a new version, as Stripe requires anyway.
- Neither a published version, its price, nor a family with customers can be
  deleted, and a family key cannot be renamed. All of those would null the
  provenance off existing contracts.
- Subscriptions and orders record plan_version_id. A historical reference
  resolved by plan NAME hands back today's terms, which is the split-brain one
  level up. A checkout that carried its version is honoured even after the
  window closes — they paid for what they were shown.

Switched atomically and failing closed: config('provisioning.plans') and
plan_features are gone, and nothing falls back to them. A fallback would
resurrect a plan the owner had just switched off. The seed lives in the
migration, and PlanCatalogueTest pins what the config catalogue sold as the
shadow comparison. `php artisan plans:check` reports overlaps, gaps and
missing prices before a customer finds them.

Contract-backed displays, which were reading the live catalogue:
seat limits, the cloud card, the plan card, and admin MRR — the last three now
divide a yearly contract down via Subscription::monthlyPriceCents(), since all
three label the figure per month.

Two recovery gaps closed along the way: the order now commits before the
contract is opened (so a contract failure can never erase the record of a
payment), a Stripe retry repairs a missing contract and restarts a run stranded
with no_subscription, and TickProvisioning sweeps runs left pending by a crash
rather than only running/waiting ones.

402 tests green. Codex review clean after thirteen rounds. Verified in the
browser: portal, billing and console render unchanged off the new catalogue.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 12:05:56 +02:00
nexxo 52b41bb0d5 fix(billing): a paid order opens a contract, and provisioning obeys it
The pipeline re-resolved config('provisioning.plans') by order.plan, so the
subscription snapshot protected a customer's price but not their machine:
shrinking a plan resized an existing customer's VM on its next run. Nothing
created a subscription either, so closing this meant opening the contract at
purchase and pointing provisioning at it.

- OpenSubscription freezes the catalogue onto a subscription when a checkout
  is paid; StartCustomerProvisioning calls it inside the order transaction.
- CustomerStep::plan() reads the frozen snapshot. ValidateOrder and
  ReserveResources fail closed with no_subscription rather than falling back
  to the catalogue, which is the bug itself.
- template_vmid joins the snapshot so a re-clone cannot pick up a blueprint
  published after the sale. Deliberately outside FROZEN: it is how we build
  the machine, not a term the customer is owed, and a dead template must be
  replaceable without cancelling a contract.
- TrafficMeter reads the allowance off the contract too — cutting a plan's
  traffic was otherwise enough to start throttling someone who bought more.
- The migration backfills contracts for orders that already bought something,
  reconstructed from what was actually delivered where an instance exists,
  and adopts an existing order-less contract instead of opening a second.
  Orders paid in a currency the catalogue cannot price get none, matching the
  checkout path.

price_cents stays the catalogue's NET price, which is what PlanChange
prorates against — not Order::amount_cents, which holds Stripe's GROSS total.
Reconciling the two belongs to the proof register and Stripe (phases 4/5).

Also pins STRIPE_WEBHOOK_SECRET blank in phpunit.xml: the operator's real
secret was reaching the suite from .env and rejecting every unsigned test
payload, which is why 7 webhook tests failed before any of this.

Verified in the browser: with team traffic cut from 3000 to 500 GB in the
catalogue, the customer's portal still shows 3 TB.

373 tests green. Codex review clean after three rounds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 11:10:00 +02:00
nexxo 6119044669 fix(billing): no plan changes after cancellation, no unknown terms
tests / pest (push) Successful in 6m46s Details
tests / assets (push) Successful in 19s Details
tests / release (push) Has been skipped Details
A cancelled subscription still reported an upgrade as allowed and priced it, so
a caller trusting that would have provisioned and billed a customer who had
already left. And any term string other than the two we support was silently
priced monthly while keeping the unknown value — a subscription whose price and
billing period disagree. Both are refused now.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:57:53 +02:00
nexxo 35324b986a fix(billing): no free upgrade in the last hours of a term
tests / pest (push) Successful in 6m48s Details
tests / assets (push) Successful in 19s Details
tests / release (push) Has been skipped Details
Flooring the remaining time made an upgrade with under a day left cost nothing,
and an upgrade requested after the period had ended cost nothing while being
applied immediately — the bigger plan for free. Remaining time now rounds up
while any service is left, and an expired period defers the change to the next
term instead of pricing it at zero.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:55:53 +02:00
nexxo 52b1acbb80 fix(billing): a scheduled downgrade must be applicable, and never cost money
tests / pest (push) Successful in 6m49s Details
tests / assets (push) Successful in 19s Details
tests / release (push) Has been skipped Details
Once the term is over, the downgrade has to be allowed — otherwise the job that
is supposed to carry it out never can, and the change waits forever for a date
that has already passed.

And a grandfathered plan can be cheaper than the smaller plan costs today, which
made the goodwill credit negative: an invoice for the privilege of downgrading.
Clamped to zero.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:54:16 +02:00
nexxo 460fac01b1 fix(billing): upgrade or downgrade is decided by rank, not by price
tests / pest (push) Successful in 6m51s Details
tests / assets (push) Successful in 20s Details
tests / release (push) Has been skipped Details
With grandfathered prices the comparison inverts: a business plan bought when
it cost less than today's team plan would have treated a move to team as an
upgrade — charged immediately, while the customer loses resources. The plan's
rank is frozen with the rest of the snapshot and decides the direction; prices
only decide the amount.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:52:28 +02:00
nexxo ac250071cb feat(billing): immutable subscription snapshots and pro-rata plan changes
The plan catalogue describes what we sell today; a customer who signed up last
year bought last year's terms. Every commercially relevant condition — price,
quotas, seats, the hardware behind the plan — is now frozen onto a subscription
at signup, and the model refuses to let any of it be rewritten afterwards. A
price rise applies to new subscriptions and cannot reach back into an existing
contract.

PlanChange holds the two rules, computed against the frozen price rather than
today's catalogue:

- Upgrading is immediate and pro rata: the new plan for the days left in the
  paid term, minus what the old plan was worth over those same days. On the
  last day of a month that is one day's difference, not a month's.
- Downgrading waits for the end of the term. A yearly customer bought a year
  and can move down when it is up; a month is a month. A mid-term downgrade is
  a goodwill decision, not a self-service button, and its credit covers only
  the unused part of the difference.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:49:48 +02:00
nexxo 5aef5e693c fix(billing): verification vouches for a value, not for a row
tests / pest (push) Successful in 6m52s Details
tests / assets (push) Successful in 20s Details
tests / release (push) Has been skipped Details
A timestamp alone said "some number was checked once": editing the field left
it intact, so a customer could swap a verified foreign VAT ID for any
plausible-looking one and keep zero-VAT pricing. The verified value is stored
and compared, which makes the rule self-enforcing — no writer has to remember
to clear a flag, and there are several writers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:34:40 +02:00
nexxo 42ceafb57f fix(billing): a self-declared VAT ID must not zero the tax
tests / pest (push) Successful in 7m13s Details
tests / assets (push) Successful in 22s Details
tests / release (push) Has been skipped Details
Any non-empty string starting with two characters other than AT switched the
customer to reverse charge — typing "XX123" was a 20 % discount. Reverse charge
now requires a VAT ID that is verified, belongs to an EU member state other than
ours, and looks like a VAT number at all. Unverified is the normal state and
means the domestic rate: over-collecting is correctable, under-collecting is a
tax liability.

Changing the number clears its verification. Verification itself (VIES) is not
built yet, so reverse charge stays off until someone confirms a number — which
is the safe direction to be wrong in.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:32:42 +02:00
nexxo 76510a59a3 fix(billing): VAT follows the customer, not a global setting
tests / pest (push) Successful in 7m28s Details
tests / assets (push) Successful in 21s Details
tests / release (push) Has been skipped Details
Codex was right that this could misstate real charges: an EU business with a
VAT ID registered in another country is billed under reverse charge, and we
were adding 20 % Austrian VAT to their total anyway. TaxTreatment resolves it
from the customer's VAT ID, and the whole page — cart, plan cards, add-on
cards — now states one treatment instead of contradicting itself.

Explicitly NOT handled: cross-border sales to private individuals, which are
taxed at the buyer's national rate under OSS. That needs a maintained rate
table and a tax adviser, not a guess, so those fall back to the domestic rate —
over-collecting rather than under-collecting.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 09:30:17 +02:00
nexxo aae0457e19 feat(admin): administrator access to a customer's Nextcloud
Impersonation borrows the customer's portal session; this is access to their
installation, which is a different thing and now a different button.

Nextcloud has no passwordless admin jump, so this does the only honest thing it
offers: it resets OUR managed admin account inside that installation and hands
the credentials over once. The customer's own accounts are untouched, and the
next request sets a new password again.

- New capability instances.adminlogin, Owner and Admin only — stronger than
  impersonation, because it hands over control rather than a session.
- The operator's own password is required every time, rate-limited: taking over
  a customer's installation is not something an unattended browser should manage
  on its own.
- The reset runs on the provisioning worker (it owns the tunnel and the Proxmox
  credentials); the console polls a handoff token, so the credentials never
  enter a Livewire snapshot.
- A silent instance is reported as such instead of appearing to succeed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 05:30:59 +02:00
nexxo 69043f9ffc fix(dns): deleting a record that is already gone is success
A lost response, or a retry after a later step failed, made every further
attempt a 404 — and the host impossible to purge.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 23:54:45 +02:00
nexxo 9f8659da32 fix(hosts): a failed DNS deletion must not orphan the record
Swallowing it and carrying on deleted the row that held the only reference to
that record, publishing the machine's management address for good. The purge now
fails instead — and is retried, since it re-reads the host and its other steps
are idempotent — so a broken DNS provider becomes a visible failed job rather
than a silent leak.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 23:53:14 +02:00
nexxo 5db2f7fda7 feat(hosts): give each host a DNS name under the public zone
fsn-01.node.clupilot.com, numbered per datacenter and never reused — removing a
host must not renumber its neighbours onto its name, so the number is stored
rather than derived.

The record points at the host's WireGuard address, not its public IP: the name
exists so an operator can reach a host by name over the VPN, and publishing a
Proxmox host's public address would hand every scanner a target, which is the
one thing this network design avoids.

DNS is convenience, not a prerequisite: a failure logs an event and lets the
onboarding finish rather than stranding a host that is otherwise ready.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 23:49:18 +02:00
nexxo c1e81808a7 feat(traffic): meter the monthly allowance, show it, throttle instead of blocking
Customers can now see what they have used and what is left, and the service
slows down rather than stopping when the allowance is gone — a slow Nextcloud
gets a top-up, a dead one gets a cancellation.

- instance_traffic keeps one row per instance per month. Proxmox counters are
  cumulative since the VM last started, so usage is the difference between two
  samples, and a counter that went backwards means a restart, not a refund.
- Outbound is what counts: inbound is free at our providers and egress is what
  Hetzner's 20 TB per server applies to.
- CollectInstanceTraffic samples every 15 minutes, warns once per threshold
  (80/95 %) rather than on every run, and at 100 % limits the VM's NIC via
  Proxmox — released again as soon as the customer tops up or the month rolls
  over.
- The dashboard gets a full-width band (it throttles the service, so it is not
  a tile among tiles) with the top-up offer right next to the warning.
- Bytes are formatted in SI units now: allowances are computed in SI, so
  dividing by 1024 made a 1000 GB plan read as "931 GB".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 23:33:47 +02:00
nexxo 7e17bc74b2 fix(vpn): give the QR code the specified four-module quiet zone
Two modules is below the minimum, and against a bordered card some readers fail
to locate the symbol at all.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 22:40:54 +02:00
nexxo 0d9b62eb50 feat(vpn): ownership, a Developer role, and password-gated config retrieval
Reworked after a design consultation with Codex, which pushed back on my first
proposal in three useful ways.

Ownership and rights:
- vpn.manage split into vpn.view.all and vpn.manage.all. Seeing an access is
  not managing it, and neither is holding its private key.
- Record-level rules live in VpnPeerPolicy, not in permissions: an access
  belongs to a person, and ownership is what grants sight of it. Every operator
  reaches the page, but sees only their own unless they may see all.
- Issuing an access is not self-service — it reaches the management network, so
  it needs vpn.manage.all even for oneself.
- New Developer role: sees everything, manages nothing. Writing code does not
  imply authority over other people's access.
- kind (staff|host|system) replaces a null user_id that had to mean two
  different things at once.

Config storage, opt-in per access:
- Downloading is owner-only — explicitly NOT for view.all or manage.all. An
  admin who needs access revokes this one and issues their own, which keeps the
  record of who holds what honest.
- The password is asked on EVERY retrieval, rate-limited. Laravel's
  password.confirm keeps a 15-minute session stamp, which would authorise
  unlimited later downloads from an unattended browser.
- Stored under VPN_CONFIG_KEY, not APP_KEY: a leaked application key must not
  also hand over the management network. Purged on revocation and when a staff
  member is revoked — console taken away, tunnel left open was the worst case.
- The plaintext never enters a Livewire snapshot (the page polls every five
  seconds); the component carries an opaque handle instead.

Also: copy button works without HTTPS again (navigator.clipboard only exists in
a secure context, so over http it silently did nothing), plus config download
as a file and a QR code for the mobile apps.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 22:31:54 +02:00
nexxo 08670ecbee fix(vpn): stop revoked accesses coming back and stale jobs undoing intent
Two lifecycle holes Codex found in the new VPN management:

1. Delete removed the row before the hub knew. If the removal job was delayed
   or failed, SyncVpnPeers saw a peer it did not recognise and adopted it back
   as a live access — silently restoring what an operator had just revoked, and
   with no row left to revoke it again. Deletion is now a soft-delete
   tombstone: the row survives until the hub confirms the peer is gone, the
   sync retries the removal instead of adopting, and the tombstone is purged
   only once the peer is really absent. It also keeps the key and the tunnel
   address reserved meanwhile, so neither is handed out twice.

2. ApplyVpnPeer applied the state captured at dispatch. A retried block job
   could therefore undo a later unblock, and nothing would repair it — the sync
   only observes state, it never re-applies intent. The job now resolves the
   current desired state from the row and treats its payload as a fallback for
   the case where the row is already gone.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 21:41:03 +02:00
nexxo 71ca0e1394 feat(admin): VPN access management with live peer state
The console can now create, block and remove VPN accesses, and shows who is
actually connected — traffic counters, source endpoint and last handshake.

Design notes:
- The web container has no wg0, so the page never talks to WireGuard. Live
  state is copied in by SyncVpnPeers on the provisioning queue (the only worker
  whose container owns the interface); the page polls the database and merely
  nudges that job, throttled so many open tabs cannot flood the queue.
- enabled (operator intent) and present (observed on the hub) are stored
  separately, so a sync can never quietly undo a block and drift stays visible
  as "wird angewendet".
- The sync adopts peers the host pipeline registered, naming them after their
  host — otherwise "who is on the VPN" would have blind spots.
- Keypairs are generated in PHP via libsodium rather than shelling out to
  wg genkey (no interface in this container, and it keeps generation testable).
  The private key is shown once and never stored.
- allocateIp() now also considers vpn_peers, which would otherwise be handed a
  tunnel address a host already holds.
- vpn.manage is a new capability held by Owner/Admin only, and it hides the nav
  entry too — a VPN access reaches the management network.

Verified end to end against the real hub, not just mocks: created an access in
the browser, connected with the generated config, watched the console flip to
"Verbunden" with real counters, then blocked it and confirmed the peer was
gone from wg0 and the client could no longer handshake.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 21:38:12 +02:00
nexxo 5301ce4309 feat(monitoring): Uptime Kuma support via a REST bridge + resilient monitoring policy
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.

Verified end-to-end against a real Uptime Kuma 1.x:
  create -> id, create again -> same id (idempotent), list, status, delete
  and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.

Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
  continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
  monitor legitimately reports 'pending' — confirmed in the e2e run); it records
  the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:04:58 +02:00
nexxo 5b85806605 fix(admin): catch racing ledger insert and re-fetch (idempotent under concurrency)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 16:31:43 +02:00
nexxo 72731112f4 fix(admin): atomically claim maintenance notification retries under a row lock
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 16:30:17 +02:00
nexxo 8bb93137a3 refactor(admin): extract MaintenanceNotifier; atomic provisioning retry; catch-up cancellation for race-delivered announcements
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 16:28:02 +02:00
nexxo 66e8a4bbf6 fix(engine-b): clean re-clone on lost task ref; idempotent firewall rules
- Clone recovery: once the lock clears but the task ref is lost, destroy the
  possibly-incomplete VM and re-clone instead of advancing onto it.
- applyFirewall clears existing rules before adding, so retries don't accumulate
  duplicate Proxmox firewall entries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:04:48 +02:00
nexxo 9162446b1f fix(engine-b): acceptance gate verifies real health, not just breadcrumbs
- Nextcloud: parse occ status JSON (installed + not in maintenance).
- Admin: query occ user:info for the account.
- Monitoring: MonitoringClient::isHealthy checks the provider, not the local row.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:00:49 +02:00
nexxo 157496f0c5 fix(engine-b): write Traefik route to the serving host over SSH, not locally
TraefikWriter now targets the host that serves the traffic (DNS points at it):
SshTraefikWriter SSHes in and writes the file-provider YAML there. Interface
takes the traffic host + guest backend.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:55:24 +02:00
nexxo 483a79a822 fix(engine-b): treat unreachable cert endpoint as not-ready (poll, not retry-burn)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:52:37 +02:00
nexxo 03c4b6508f fix(engine-b): Proxmox agent exec repeated command fields; Traefik write throws on failure
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:50:57 +02:00
nexxo 8825f9abfe fix(engine-b): Proxmox config uses PUT; guest-agent exec is form-encoded
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:48:58 +02:00
nexxo 778bb7f117 fix(engine-b): always scrub credentials once delivered; idempotent monitoring lookup
- CompleteProvisioning scrubs admin_password whenever credentials_sent exists,
  so a crash between recording and scrubbing can't leave the password in context.
- HttpMonitoringClient looks up an existing monitor by URL before creating one,
  so a retry after a crashed POST doesn't create a duplicate external monitor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:46:34 +02:00
nexxo 661b52dbe0 fix(engine-b): clone recovery waits for lock; register rows before breadcrumb
- Clone recovery polls until the VM lock clears instead of advancing onto a
  possibly-incomplete clone.
- RegisterBackup/RegisterMonitoring create the local row BEFORE the run-resource
  breadcrumb, so a crash between them can't leave the row permanently missing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:44:22 +02:00
nexxo 3c5a18b5d7 fix(engine-b): apply plan CPU/RAM; idempotent backup + monitoring registration
- ConfigureCloudInit sets cores/memory from the plan (not the template default).
- RegisterBackup uses a deterministic Proxmox job id (ignore-exists) + firstOrCreate.
- RegisterMonitoring is idempotent by URL + firstOrCreate — no duplicate schedules
  or monitors after a crash/retry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:42:10 +02:00
nexxo 5cf1964d7d fix(engine-b): default-deny firewall, disk-based capacity, synchronous credential mail
- applyFirewall sets policy_in=DROP so only 80/443 are exposed.
- Capacity/placement account for disk_gb (the real VM allocation), not the
  smaller Nextcloud user quota — no systematic overcommit.
- CloudReady sent synchronously again; the step retries on mail failure with the
  password preserved (no lost credential in a failed queue job).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:39:15 +02:00
nexxo b4cf94ee1a fix(engine-b): real backup/monitoring registration + OC_PASS scoping
- CreateCustomerAdmin puts OC_PASS on the docker invocation (survives the &&).
- RegisterBackup creates a real vzdump job via ProxmoxClient::createBackupJob.
- RegisterMonitoring registers via a new MonitoringClient service (interface +
  fake + http). Both persist the external id [E] before the breadcrumb, so
  acceptance reflects a real registration, not a fabricated row.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:33:45 +02:00
nexxo e0e4c4e18d fix(engine-b): run guest-agent commands through /bin/sh -c
The Proxmox guest agent execs a program directly; our compound commands
(cd/&&/pipes/redirects/env) need an explicit shell.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:25:24 +02:00
nexxo f43ade79b0 fix(engine-b): address Codex round 6 (crash-window idempotency)
- Clone: recover via vmExists() when the task ref was lost, instead of
  re-cloning the reserved vmid (which Proxmox rejects).
- Deploy: persist the DB password ENCRYPTED before compose up and reuse it on
  retry, so a crash can't regenerate a mismatching credential.
- Complete: gate credential delivery on a credentials_sent breadcrumb so a
  retry after a crash doesn't re-send the email.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:23:28 +02:00
nexxo 446da58061 feat(engine-b): 15-step customer pipeline + DNS/Traefik services
validate->reserve(placement)->clone->cloudinit->start->guestagent->network->
deploy->nextcloud->admin->dns/tls->backup->monitoring->acceptance->complete.
HetznerDnsClient + TraefikWriter (interface+fake+real), CloudReady notification,
hosts.node (set by RegisterCapacity). Secrets transient/encrypted, never
plaintext. 21 tests incl. mocked end-to-end + crash idempotency.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:50:26 +02:00
nexxo e5aeb3989e feat(engine-b): ProxmoxClient VM-lifecycle extension (interface + fake + http)
nextVmid/cloneVm/setCloudInit/resizeDisk/startVm/vmStatus/guestAgentPing/
guestExec/taskStatus/applyFirewall. Fake is deterministic with configurable
task/guest failure hooks; Http polls UPID + guest exec-status.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:41:48 +02:00
nexxo a6a41719bb fix(engine): long SSH command timeout; wg peer removal on privileged worker
- PhpseclibRemoteShell sets a command timeout (default 2000s) so apt
  full-upgrade/install don't hit phpseclib's ~10s default and fail.
- Host removal dispatches RemoveWireguardPeer to the provisioning queue, which
  runs in the worker that owns wg0 (the web container can't manage WireGuard).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:01:02 +02:00
nexxo 8a392bfd06 fix(engine): parse WireGuard CIDR for address allocation (any prefix length)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:48:16 +02:00
nexxo 48ed5e7d34 fix(engine): fail if the WireGuard peer can't be persisted (wg-quick save)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:41:11 +02:00
nexxo fbe87d99d5 fix(engine): honour backoff timing + don't double-count Proxmox storage
- RunRunner returns early for a waiting run whose next_attempt_at is in the
  future, so stale/duplicate jobs can't bypass backoff.
- RegisterCapacity takes the largest VM-capable datastore instead of summing
  overlapping pools (local + local-lvm), preventing placement overcommit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:32:14 +02:00
nexxo 65717bd3bd fix(engine): address Codex review (auth token bootstrap, tunnel recheck, ssh)
- CreateAutomationToken now bootstraps the pveum role/user/token over the
  authenticated SSH session (a fresh host has no API token yet); ProxmoxClient
  is read-only in A.
- ConfigureWireguard re-verifies the handshake on the idempotent replay path,
  never advancing over a dead tunnel.
- PhpseclibRemoteShell treats a missing SSH exit status as failure (255).
- connectWithKey verifies the pinned host key fingerprint on later logins.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:20:59 +02:00
nexxo ac3d17cd6a feat(engine): 11-step host onboarding pipeline (SSH -> WG -> Proxmox)
Idempotent steps: validate, ssh-trust (deploy key + scrub password),
prepare base, wireguard (hub peer), install proxmox-ve, reboot-into-pve
(retry-poll), configure, automation token, verify api, register capacity,
complete. StartHostOnboarding action. 22 tests incl. mocked end-to-end +
crash idempotency.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:04:39 +02:00
nexxo c06ca0ae5d feat(engine): SSH / WireGuard / Proxmox service layer
RemoteShell (phpseclib + fake), WireguardHub (local wg + fake), ProxmoxClient
(REST + fake). Interfaces bound in AppServiceProvider; tests swap fakes via
fakeServices() helper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 09:58:02 +02:00