Commit Graph

20 Commits (82916167cfa5fd990a1c0aee76795988983656b0)

Author SHA1 Message Date
nexxo 82916167cf fix(monitoring): MONITORING_ATTEMPTS is now exact (off-by-one); accept int or dict monitor id
- attempt is 0-based, so compare one-based: ATTEMPTS=2 really means two tries
- bridge accepts add_monitor returning {monitorID}|{id}|int (v1.2.1 returns a
  dict — re-verified: first create on a fresh Kuma is HTTP 200 with the id)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:08:07 +02:00
nexxo 5301ce4309 feat(monitoring): Uptime Kuma support via a REST bridge + resilient monitoring policy
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.

Verified end-to-end against a real Uptime Kuma 1.x:
  create -> id, create again -> same id (idempotent), list, status, delete
  and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.

Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
  continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
  monitor legitimately reports 'pending' — confirmed in the e2e run); it records
  the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:04:58 +02:00
nexxo d7c2edb1df fix(engine-b): idempotent local DNS record (firstOrCreate on record_id)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:08:13 +02:00
nexxo 66e8a4bbf6 fix(engine-b): clean re-clone on lost task ref; idempotent firewall rules
- Clone recovery: once the lock clears but the task ref is lost, destroy the
  possibly-incomplete VM and re-clone instead of advancing onto it.
- applyFirewall clears existing rules before adding, so retries don't accumulate
  duplicate Proxmox firewall entries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:04:48 +02:00
nexxo 9162446b1f fix(engine-b): acceptance gate verifies real health, not just breadcrumbs
- Nextcloud: parse occ status JSON (installed + not in maintenance).
- Admin: query occ user:info for the account.
- Monitoring: MonitoringClient::isHealthy checks the provider, not the local row.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:00:49 +02:00
nexxo 157496f0c5 fix(engine-b): write Traefik route to the serving host over SSH, not locally
TraefikWriter now targets the host that serves the traffic (DNS points at it):
SshTraefikWriter SSHes in and writes the file-provider YAML there. Interface
takes the traffic host + guest backend.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:55:24 +02:00
nexxo 778bb7f117 fix(engine-b): always scrub credentials once delivered; idempotent monitoring lookup
- CompleteProvisioning scrubs admin_password whenever credentials_sent exists,
  so a crash between recording and scrubbing can't leave the password in context.
- HttpMonitoringClient looks up an existing monitor by URL before creating one,
  so a retry after a crashed POST doesn't create a duplicate external monitor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:46:34 +02:00
nexxo 661b52dbe0 fix(engine-b): clone recovery waits for lock; register rows before breadcrumb
- Clone recovery polls until the VM lock clears instead of advancing onto a
  possibly-incomplete clone.
- RegisterBackup/RegisterMonitoring create the local row BEFORE the run-resource
  breadcrumb, so a crash between them can't leave the row permanently missing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:44:22 +02:00
nexxo 3c5a18b5d7 fix(engine-b): apply plan CPU/RAM; idempotent backup + monitoring registration
- ConfigureCloudInit sets cores/memory from the plan (not the template default).
- RegisterBackup uses a deterministic Proxmox job id (ignore-exists) + firstOrCreate.
- RegisterMonitoring is idempotent by URL + firstOrCreate — no duplicate schedules
  or monitors after a crash/retry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:42:10 +02:00
nexxo 5cf1964d7d fix(engine-b): default-deny firewall, disk-based capacity, synchronous credential mail
- applyFirewall sets policy_in=DROP so only 80/443 are exposed.
- Capacity/placement account for disk_gb (the real VM allocation), not the
  smaller Nextcloud user quota — no systematic overcommit.
- CloudReady sent synchronously again; the step retries on mail failure with the
  password preserved (no lost credential in a failed queue job).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:39:15 +02:00
nexxo c79874ccfe fix(engine-b): keep admin password ciphertext in the queued mail; valid backup schedule
- CloudReady receives the ENCRYPTED password (decrypts only when rendering), so
  the queued payload never holds plaintext.
- RegisterBackup uses a valid Proxmox calendar expression (02:00).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:35:49 +02:00
nexxo b4cf94ee1a fix(engine-b): real backup/monitoring registration + OC_PASS scoping
- CreateCustomerAdmin puts OC_PASS on the docker invocation (survives the &&).
- RegisterBackup creates a real vzdump job via ProxmoxClient::createBackupJob.
- RegisterMonitoring registers via a new MonitoringClient service (interface +
  fake + http). Both persist the external id [E] before the breadcrumb, so
  acceptance reflects a real registration, not a fabricated row.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:33:45 +02:00
nexxo f43ade79b0 fix(engine-b): address Codex round 6 (crash-window idempotency)
- Clone: recover via vmExists() when the task ref was lost, instead of
  re-cloning the reserved vmid (which Proxmox rejects).
- Deploy: persist the DB password ENCRYPTED before compose up and reuse it on
  retry, so a crash can't regenerate a mismatching credential.
- Complete: gate credential delivery on a credentials_sent breadcrumb so a
  retry after a crash doesn't re-send the email.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:23:28 +02:00
nexxo e2b8f25e78 fix(engine-b): address Codex round 5 (capacity release, activate-last, guest-ip poll, email unique)
- Failed run releases the instance (status=failed) so its quota stops counting
  against host capacity.
- CompleteProvisioning activates instance+order only AFTER onboarding tasks +
  credential delivery succeed.
- ConfigureNetwork polls until the guest has an address (no wrong Traefik route).
- customers.email unique + race-safe customer resolution in the intake action.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:19:31 +02:00
nexxo c56aa14f9d fix(engine): don't reset started_at on poll so step deadlines accumulate
RunRunner::onPoll no longer resets started_at, so a poll step's own deadline
(WaitForGuestAgent 270s, ConfigureDnsAndTls cert 840s) actually fires instead
of resetting every poll. maxDuration raised above each own-deadline so the
step's fail wins over the generic timeout. Shared core fix (A + B).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:15:26 +02:00
nexxo fffcecb152 fix(engine-b): address Codex round 3 (webhook email/replay/rotation, db secret)
- Require a real customer email (no unknown@ fallback merging customers).
- Stripe signature: enforce 5-min replay tolerance + accept any rotating v1.
- DeployApplicationStack no longer persists the DB password in the run context.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:12:24 +02:00
nexxo e6bd6c9354 fix(engine-b): address Codex round 2 (fail-closed webhook, placement lock, clone/admin idempotency)
- Stripe webhook fails closed when the signing secret is missing (outside local/testing).
- ReserveResources places + creates the instance under a per-datacenter lock so
  concurrent orders can't overcommit a host.
- CloneVirtualMachine reserves the vmid + breadcrumb BEFORE cloning, so a crash
  can't mint a new vmid and orphan the first clone.
- CreateCustomerAdmin checks user existence and resets the password instead of
  re-running user:add on retry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:09:21 +02:00
nexxo 42fa92cf20 fix(engine-b): address Codex (paid-only stripe, guest routing, absolute disk)
- Stripe: only checkout.session.completed with payment_status=paid (dedupes the
  paired payment_intent event and blocks unpaid async sessions).
- Capture the guest IP (ConfigureNetwork) and point Traefik at the VM, not the
  Proxmox host, so ACME/HTTP-01 can reach Nextcloud.
- Resize the disk to an absolute target (not '+…') so a retry can't double it.
(Codex #4 timeout was a false positive — started_at resets per step.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:05:31 +02:00
nexxo 446da58061 feat(engine-b): 15-step customer pipeline + DNS/Traefik services
validate->reserve(placement)->clone->cloudinit->start->guestagent->network->
deploy->nextcloud->admin->dns/tls->backup->monitoring->acceptance->complete.
HetznerDnsClient + TraefikWriter (interface+fake+real), CloudReady notification,
hosts.node (set by RegisterCapacity). Secrets transient/encrypted, never
plaintext. 21 tests incl. mocked end-to-end + crash idempotency.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:50:26 +02:00
nexxo 770a6cf7cd feat(engine-b): CustomerStep base + shared resource trait + config
ManagesRunResources trait (host+customer), CustomerStep base (order/instance/
plan/guest helpers), config plans + 15-step customer pipeline + dns/traefik.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:39:56 +02:00