Commit Graph

23 Commits (feat/mailboxes)

Author SHA1 Message Date
Claude de6821b53e Move the console off /admin, give the status page its own address, and measure monitoring
tests / pest (push) Successful in 7m43s Details
tests / assets (push) Successful in 19s Details
tests / release (push) Successful in 5s Details
Three things the owner asked for, and one the review found underneath them.

The console leaves /admin. It has its own route file now, registered before
routes/web.php because once it has a hostname to itself it sits at the ROOT of
that host — where `/` and `/settings` also exist for the portal, and the first
matching route wins. Where it mounts is one decision in one place: `/` on the
console hostname where it has one to itself, `/admin` on any host otherwise.
Names stay admin.* in both modes, so nothing that builds a URL has to know.

Exclusivity is its own switch, not a consequence of having a hostname. The
first attempt made "this host is the console" follow from ADMIN_HOSTS, and a
development machine lists its own IP there so the console works without DNS —
which took the public site and the portal off that machine entirely. The
switch also registers the console on every listed hostname, canonical last, so
the alternates that exist as recovery paths keep working.

The status page moves out of /legal, where it sat beside the imprint and the
terms. Nothing about the current health of the platform is a legal document. It
is a real page now: portal, instances, provisioning and backups, each derived
from records, aggregate only, and a component with no signal reports "unknown"
rather than "operational".

That last rule is what exposed the real bug. monitoring_targets.status was
written once — 'up', at provisioning — and nothing ever updated it. Both the
console's notices and the new public page read it, so an outage would have been
published as healthy indefinitely. There is a sync job now, on a five-minute
schedule, and a checked_at column so a verdict can go stale instead of standing
forever.

The monitoring contract had to grow a third state for that to be honest.
isHealthy() answers true when no monitoring is configured and false when the
monitor is unreachable; recording that boolean would have published either a
fleet-wide all-clear nobody measured or a fleet-wide outage that was really one
broken monitor. health() returns null for both, the recorder leaves the old
verdict to go stale, and isHealthy() keeps its forgiving semantics for the one
caller that wants them — the provisioning acceptance check, which must not fail
a delivery because monitoring is not set up.

Backups are counted from the instances that need protecting rather than from
the backup rows that exist, so an instance with no schedule at all cannot be
missing from the arithmetic that declares the estate protected.

Also: the update panel polls itself, offers the button only when there is
something to install, and opens the log while a run is in progress.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-27 06:05:40 +02:00
nexxo 52b41bb0d5 fix(billing): a paid order opens a contract, and provisioning obeys it
The pipeline re-resolved config('provisioning.plans') by order.plan, so the
subscription snapshot protected a customer's price but not their machine:
shrinking a plan resized an existing customer's VM on its next run. Nothing
created a subscription either, so closing this meant opening the contract at
purchase and pointing provisioning at it.

- OpenSubscription freezes the catalogue onto a subscription when a checkout
  is paid; StartCustomerProvisioning calls it inside the order transaction.
- CustomerStep::plan() reads the frozen snapshot. ValidateOrder and
  ReserveResources fail closed with no_subscription rather than falling back
  to the catalogue, which is the bug itself.
- template_vmid joins the snapshot so a re-clone cannot pick up a blueprint
  published after the sale. Deliberately outside FROZEN: it is how we build
  the machine, not a term the customer is owed, and a dead template must be
  replaceable without cancelling a contract.
- TrafficMeter reads the allowance off the contract too — cutting a plan's
  traffic was otherwise enough to start throttling someone who bought more.
- The migration backfills contracts for orders that already bought something,
  reconstructed from what was actually delivered where an instance exists,
  and adopts an existing order-less contract instead of opening a second.
  Orders paid in a currency the catalogue cannot price get none, matching the
  checkout path.

price_cents stays the catalogue's NET price, which is what PlanChange
prorates against — not Order::amount_cents, which holds Stripe's GROSS total.
Reconciling the two belongs to the proof register and Stripe (phases 4/5).

Also pins STRIPE_WEBHOOK_SECRET blank in phpunit.xml: the operator's real
secret was reaching the suite from .env and rejecting every unsigned test
payload, which is why 7 webhook tests failed before any of this.

Verified in the browser: with team traffic cut from 3000 to 500 GB in the
catalogue, the customer's portal still shows 3 TB.

373 tests green. Codex review clean after three rounds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-26 11:10:00 +02:00
nexxo 46d821a047 fix(monitoring): cap monitoring retries at the run budget; dependency-free liveness probe
- MONITORING_ATTEMPTS is capped by the run's max_attempts, so a large value can
  no longer burn the retry budget and fail the very provisioning the degradation
  exists to protect (covered by a new test at the budget boundary)
- /health is now dependency-free (200 in ~2ms with Kuma absent, verified: the
  container healthcheck stays healthy); Kuma state is exclusively in /ready

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:19:32 +02:00
nexxo 82916167cf fix(monitoring): MONITORING_ATTEMPTS is now exact (off-by-one); accept int or dict monitor id
- attempt is 0-based, so compare one-based: ATTEMPTS=2 really means two tries
- bridge accepts add_monitor returning {monitorID}|{id}|int (v1.2.1 returns a
  dict — re-verified: first create on a fresh Kuma is HTTP 200 with the id)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:08:07 +02:00
nexxo 5301ce4309 feat(monitoring): Uptime Kuma support via a REST bridge + resilient monitoring policy
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.

Verified end-to-end against a real Uptime Kuma 1.x:
  create -> id, create again -> same id (idempotent), list, status, delete
  and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.

Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
  continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
  monitor legitimately reports 'pending' — confirmed in the e2e run); it records
  the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:04:58 +02:00
nexxo d7c2edb1df fix(engine-b): idempotent local DNS record (firstOrCreate on record_id)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:08:13 +02:00
nexxo 66e8a4bbf6 fix(engine-b): clean re-clone on lost task ref; idempotent firewall rules
- Clone recovery: once the lock clears but the task ref is lost, destroy the
  possibly-incomplete VM and re-clone instead of advancing onto it.
- applyFirewall clears existing rules before adding, so retries don't accumulate
  duplicate Proxmox firewall entries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:04:48 +02:00
nexxo 9162446b1f fix(engine-b): acceptance gate verifies real health, not just breadcrumbs
- Nextcloud: parse occ status JSON (installed + not in maintenance).
- Admin: query occ user:info for the account.
- Monitoring: MonitoringClient::isHealthy checks the provider, not the local row.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:00:49 +02:00
nexxo 157496f0c5 fix(engine-b): write Traefik route to the serving host over SSH, not locally
TraefikWriter now targets the host that serves the traffic (DNS points at it):
SshTraefikWriter SSHes in and writes the file-provider YAML there. Interface
takes the traffic host + guest backend.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:55:24 +02:00
nexxo 778bb7f117 fix(engine-b): always scrub credentials once delivered; idempotent monitoring lookup
- CompleteProvisioning scrubs admin_password whenever credentials_sent exists,
  so a crash between recording and scrubbing can't leave the password in context.
- HttpMonitoringClient looks up an existing monitor by URL before creating one,
  so a retry after a crashed POST doesn't create a duplicate external monitor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:46:34 +02:00
nexxo 661b52dbe0 fix(engine-b): clone recovery waits for lock; register rows before breadcrumb
- Clone recovery polls until the VM lock clears instead of advancing onto a
  possibly-incomplete clone.
- RegisterBackup/RegisterMonitoring create the local row BEFORE the run-resource
  breadcrumb, so a crash between them can't leave the row permanently missing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:44:22 +02:00
nexxo 3c5a18b5d7 fix(engine-b): apply plan CPU/RAM; idempotent backup + monitoring registration
- ConfigureCloudInit sets cores/memory from the plan (not the template default).
- RegisterBackup uses a deterministic Proxmox job id (ignore-exists) + firstOrCreate.
- RegisterMonitoring is idempotent by URL + firstOrCreate — no duplicate schedules
  or monitors after a crash/retry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:42:10 +02:00
nexxo 5cf1964d7d fix(engine-b): default-deny firewall, disk-based capacity, synchronous credential mail
- applyFirewall sets policy_in=DROP so only 80/443 are exposed.
- Capacity/placement account for disk_gb (the real VM allocation), not the
  smaller Nextcloud user quota — no systematic overcommit.
- CloudReady sent synchronously again; the step retries on mail failure with the
  password preserved (no lost credential in a failed queue job).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:39:15 +02:00
nexxo c79874ccfe fix(engine-b): keep admin password ciphertext in the queued mail; valid backup schedule
- CloudReady receives the ENCRYPTED password (decrypts only when rendering), so
  the queued payload never holds plaintext.
- RegisterBackup uses a valid Proxmox calendar expression (02:00).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:35:49 +02:00
nexxo b4cf94ee1a fix(engine-b): real backup/monitoring registration + OC_PASS scoping
- CreateCustomerAdmin puts OC_PASS on the docker invocation (survives the &&).
- RegisterBackup creates a real vzdump job via ProxmoxClient::createBackupJob.
- RegisterMonitoring registers via a new MonitoringClient service (interface +
  fake + http). Both persist the external id [E] before the breadcrumb, so
  acceptance reflects a real registration, not a fabricated row.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:33:45 +02:00
nexxo f43ade79b0 fix(engine-b): address Codex round 6 (crash-window idempotency)
- Clone: recover via vmExists() when the task ref was lost, instead of
  re-cloning the reserved vmid (which Proxmox rejects).
- Deploy: persist the DB password ENCRYPTED before compose up and reuse it on
  retry, so a crash can't regenerate a mismatching credential.
- Complete: gate credential delivery on a credentials_sent breadcrumb so a
  retry after a crash doesn't re-send the email.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:23:28 +02:00
nexxo e2b8f25e78 fix(engine-b): address Codex round 5 (capacity release, activate-last, guest-ip poll, email unique)
- Failed run releases the instance (status=failed) so its quota stops counting
  against host capacity.
- CompleteProvisioning activates instance+order only AFTER onboarding tasks +
  credential delivery succeed.
- ConfigureNetwork polls until the guest has an address (no wrong Traefik route).
- customers.email unique + race-safe customer resolution in the intake action.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:19:31 +02:00
nexxo c56aa14f9d fix(engine): don't reset started_at on poll so step deadlines accumulate
RunRunner::onPoll no longer resets started_at, so a poll step's own deadline
(WaitForGuestAgent 270s, ConfigureDnsAndTls cert 840s) actually fires instead
of resetting every poll. maxDuration raised above each own-deadline so the
step's fail wins over the generic timeout. Shared core fix (A + B).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:15:26 +02:00
nexxo fffcecb152 fix(engine-b): address Codex round 3 (webhook email/replay/rotation, db secret)
- Require a real customer email (no unknown@ fallback merging customers).
- Stripe signature: enforce 5-min replay tolerance + accept any rotating v1.
- DeployApplicationStack no longer persists the DB password in the run context.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:12:24 +02:00
nexxo e6bd6c9354 fix(engine-b): address Codex round 2 (fail-closed webhook, placement lock, clone/admin idempotency)
- Stripe webhook fails closed when the signing secret is missing (outside local/testing).
- ReserveResources places + creates the instance under a per-datacenter lock so
  concurrent orders can't overcommit a host.
- CloneVirtualMachine reserves the vmid + breadcrumb BEFORE cloning, so a crash
  can't mint a new vmid and orphan the first clone.
- CreateCustomerAdmin checks user existence and resets the password instead of
  re-running user:add on retry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:09:21 +02:00
nexxo 42fa92cf20 fix(engine-b): address Codex (paid-only stripe, guest routing, absolute disk)
- Stripe: only checkout.session.completed with payment_status=paid (dedupes the
  paired payment_intent event and blocks unpaid async sessions).
- Capture the guest IP (ConfigureNetwork) and point Traefik at the VM, not the
  Proxmox host, so ACME/HTTP-01 can reach Nextcloud.
- Resize the disk to an absolute target (not '+…') so a retry can't double it.
(Codex #4 timeout was a false positive — started_at resets per step.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:05:31 +02:00
nexxo 446da58061 feat(engine-b): 15-step customer pipeline + DNS/Traefik services
validate->reserve(placement)->clone->cloudinit->start->guestagent->network->
deploy->nextcloud->admin->dns/tls->backup->monitoring->acceptance->complete.
HetznerDnsClient + TraefikWriter (interface+fake+real), CloudReady notification,
hosts.node (set by RegisterCapacity). Secrets transient/encrypted, never
plaintext. 21 tests incl. mocked end-to-end + crash idempotency.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:50:26 +02:00
nexxo 770a6cf7cd feat(engine-b): CustomerStep base + shared resource trait + config
ManagesRunResources trait (host+customer), CustomerStep base (order/instance/
plan/guest helpers), config plans + 15-step customer pipeline + dns/traefik.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:39:56 +02:00