Commit Graph

8 Commits (819872bb79a3caef7c547ba104ef8599545ff6c8)

Author SHA1 Message Date
nexxo c56aa14f9d fix(engine): don't reset started_at on poll so step deadlines accumulate
RunRunner::onPoll no longer resets started_at, so a poll step's own deadline
(WaitForGuestAgent 270s, ConfigureDnsAndTls cert 840s) actually fires instead
of resetting every poll. maxDuration raised above each own-deadline so the
step's fail wins over the generic timeout. Shared core fix (A + B).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:15:26 +02:00
nexxo c7fb1ce56d fix(engine): dispatch run continuation after releasing the lock
Prevents a second worker from consuming the follow-up job and bailing on the
still-held run lock, which would stall the run until the next scheduler tick.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 11:09:24 +02:00
nexxo 93191f5de9 fix(engine): fail on unresolvable pipeline step; make worker the WG hub
- RunRunner catches step-resolution errors (removed/renamed pipeline) and
  fails the run terminally instead of looping forever every tick.
- queue-provisioning worker gains NET_ADMIN + tun + persistent wireguard
  volume + published UDP port so LocalWireguardHub can manage wg0.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:53:40 +02:00
nexxo ced7e6103d fix(engine): durable queueing for long provisioning steps
- Dedicated 'provisioning' queue connection (retry_after 2400) + worker
  (--timeout=2100); AdvanceRunJob tries=1, timeout=2100 on that queue.
- Run lock TTL raised to 2100s so a long step can't be run concurrently by a
  duplicate tick-dispatched job.
- StartHostOnboarding creates host+run in a transaction, dispatches after commit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:36:49 +02:00
nexxo fbe87d99d5 fix(engine): honour backoff timing + don't double-count Proxmox storage
- RunRunner returns early for a waiting run whose next_attempt_at is in the
  future, so stale/duplicate jobs can't bypass backoff.
- RegisterCapacity takes the largest VM-capable datastore instead of summing
  overlapping pools (local + local-lvm), preventing placement overcommit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:32:14 +02:00
nexxo 8a2a2ee695 fix(engine): reset step timer on retry so timed-out steps recover
A retried step (manual or after a timeout) now gets a fresh started_at, so
RunRunner no longer immediately re-detects the timeout and burns the retry
budget without executing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:29:11 +02:00
nexxo 62c4412623 fix(engine): address Codex round 2 (poll budget, host error state, wg race)
- StepResult::poll — polling steps (reboot) wait without consuming the retry
  budget; the step owns its deadline. Reboot maxDuration > deadline.
- Failed runs move a Host subject to 'error' via ProvisioningSubject hook
  (no host stuck 'onboarding').
- ConfigureWireguard allocates + reserves the wg_ip under a global lock;
  unique index on hosts.wg_ip as a backstop against duplicate addresses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 10:26:37 +02:00
nexxo 4f724d57c9 feat(engine): orchestrator core (state machine + tick + lock)
StepResult, ProvisioningStep contract, PipelineRegistry, RunRunner (per-run
lock, advance/retry/fail, backoff, timeout, append-only events + Reverb
StepAdvanced), AdvanceRunJob (provisioning queue), minutely Tick, admin.runs
channel. 21 tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 09:54:52 +02:00