Commit Graph

13 Commits (0219e3c987ca9e1eefac6d569c655c8dc43e25ca)

Author SHA1 Message Date
nexxo 46d821a047 fix(monitoring): cap monitoring retries at the run budget; dependency-free liveness probe
- MONITORING_ATTEMPTS is capped by the run's max_attempts, so a large value can
  no longer burn the retry budget and fail the very provisioning the degradation
  exists to protect (covered by a new test at the budget boundary)
- /health is now dependency-free (200 in ~2ms with Kuma absent, verified: the
  container healthcheck stays healthy); Kuma state is exclusively in /ready

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:19:32 +02:00
nexxo 82916167cf fix(monitoring): MONITORING_ATTEMPTS is now exact (off-by-one); accept int or dict monitor id
- attempt is 0-based, so compare one-based: ATTEMPTS=2 really means two tries
- bridge accepts add_monitor returning {monitorID}|{id}|int (v1.2.1 returns a
  dict — re-verified: first create on a fresh Kuma is HTTP 200 with the id)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:08:07 +02:00
nexxo 5301ce4309 feat(monitoring): Uptime Kuma support via a REST bridge + resilient monitoring policy
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.

Verified end-to-end against a real Uptime Kuma 1.x:
  create -> id, create again -> same id (idempotent), list, status, delete
  and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.

Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
  continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
  monitor legitimately reports 'pending' — confirmed in the e2e run); it records
  the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:04:58 +02:00
nexxo 66e8a4bbf6 fix(engine-b): clean re-clone on lost task ref; idempotent firewall rules
- Clone recovery: once the lock clears but the task ref is lost, destroy the
  possibly-incomplete VM and re-clone instead of advancing onto it.
- applyFirewall clears existing rules before adding, so retries don't accumulate
  duplicate Proxmox firewall entries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:04:48 +02:00
nexxo 9162446b1f fix(engine-b): acceptance gate verifies real health, not just breadcrumbs
- Nextcloud: parse occ status JSON (installed + not in maintenance).
- Admin: query occ user:info for the account.
- Monitoring: MonitoringClient::isHealthy checks the provider, not the local row.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:00:49 +02:00
nexxo 661b52dbe0 fix(engine-b): clone recovery waits for lock; register rows before breadcrumb
- Clone recovery polls until the VM lock clears instead of advancing onto a
  possibly-incomplete clone.
- RegisterBackup/RegisterMonitoring create the local row BEFORE the run-resource
  breadcrumb, so a crash between them can't leave the row permanently missing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:44:22 +02:00
nexxo 3c5a18b5d7 fix(engine-b): apply plan CPU/RAM; idempotent backup + monitoring registration
- ConfigureCloudInit sets cores/memory from the plan (not the template default).
- RegisterBackup uses a deterministic Proxmox job id (ignore-exists) + firstOrCreate.
- RegisterMonitoring is idempotent by URL + firstOrCreate — no duplicate schedules
  or monitors after a crash/retry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:42:10 +02:00
nexxo f43ade79b0 fix(engine-b): address Codex round 6 (crash-window idempotency)
- Clone: recover via vmExists() when the task ref was lost, instead of
  re-cloning the reserved vmid (which Proxmox rejects).
- Deploy: persist the DB password ENCRYPTED before compose up and reuse it on
  retry, so a crash can't regenerate a mismatching credential.
- Complete: gate credential delivery on a credentials_sent breadcrumb so a
  retry after a crash doesn't re-send the email.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:23:28 +02:00
nexxo e2b8f25e78 fix(engine-b): address Codex round 5 (capacity release, activate-last, guest-ip poll, email unique)
- Failed run releases the instance (status=failed) so its quota stops counting
  against host capacity.
- CompleteProvisioning activates instance+order only AFTER onboarding tasks +
  credential delivery succeed.
- ConfigureNetwork polls until the guest has an address (no wrong Traefik route).
- customers.email unique + race-safe customer resolution in the intake action.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:19:31 +02:00
nexxo fffcecb152 fix(engine-b): address Codex round 3 (webhook email/replay/rotation, db secret)
- Require a real customer email (no unknown@ fallback merging customers).
- Stripe signature: enforce 5-min replay tolerance + accept any rotating v1.
- DeployApplicationStack no longer persists the DB password in the run context.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:12:24 +02:00
nexxo e6bd6c9354 fix(engine-b): address Codex round 2 (fail-closed webhook, placement lock, clone/admin idempotency)
- Stripe webhook fails closed when the signing secret is missing (outside local/testing).
- ReserveResources places + creates the instance under a per-datacenter lock so
  concurrent orders can't overcommit a host.
- CloneVirtualMachine reserves the vmid + breadcrumb BEFORE cloning, so a crash
  can't mint a new vmid and orphan the first clone.
- CreateCustomerAdmin checks user existence and resets the password instead of
  re-running user:add on retry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:09:21 +02:00
nexxo 42fa92cf20 fix(engine-b): address Codex (paid-only stripe, guest routing, absolute disk)
- Stripe: only checkout.session.completed with payment_status=paid (dedupes the
  paired payment_intent event and blocks unpaid async sessions).
- Capture the guest IP (ConfigureNetwork) and point Traefik at the VM, not the
  Proxmox host, so ACME/HTTP-01 can reach Nextcloud.
- Resize the disk to an absolute target (not '+…') so a retry can't double it.
(Codex #4 timeout was a false positive — started_at resets per step.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:05:31 +02:00
nexxo 446da58061 feat(engine-b): 15-step customer pipeline + DNS/Traefik services
validate->reserve(placement)->clone->cloudinit->start->guestagent->network->
deploy->nextcloud->admin->dns/tls->backup->monitoring->acceptance->complete.
HetznerDnsClient + TraefikWriter (interface+fake+real), CloudReady notification,
hosts.node (set by RegisterCapacity). Secrets transient/encrypted, never
plaintext. 21 tests incl. mocked end-to-end + crash idempotency.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:50:26 +02:00