Prevents a second worker from consuming the follow-up job and bailing on the
still-held run lock, which would stall the run until the next scheduler tick.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keeps the wg_peer resource gated on a verified handshake (idempotency) while
ensuring PurgeHost can always remove the hub peer, even on terminal failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard checks setup command results and only records the wg_peer
resource after the handshake verifies, so a transient setup failure re-runs
the full setup instead of getting stuck on the idempotent path.
- Host removal now deactivates immediately and queues PurgeHost, which deletes
runs under the runner lock (waiting out a long step) — no 15s LockTimeout error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- PhpseclibRemoteShell sets a command timeout (default 2000s) so apt
full-upgrade/install don't hit phpseclib's ~10s default and fail.
- Host removal dispatches RemoveWireguardPeer to the provisioning queue, which
runs in the worker that owns wg0 (the web container can't manage WireGuard).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- RunRunner catches step-resolution errors (removed/renamed pipeline) and
fails the run terminally instead of looping forever every tick.
- queue-provisioning worker gains NET_ADMIN + tun + persistent wireguard
volume + published UDP port so LocalWireguardHub can manage wg0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Removing a host now removes its WireGuard hub peer so a freed wg_ip can't
route to the removed server via a stale peer.
- RebootIntoPveKernel clears reboot_issued/deadline when it times out, so the
manual retry action actually re-issues a reboot instead of failing instantly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hub_ip is now defined in config (default: subnet .1), so a custom
CLUPILOT_WG_SUBNET no longer pings the wrong handshake target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Dedicated 'provisioning' queue connection (retry_after 2400) + worker
(--timeout=2100); AdvanceRunJob tries=1, timeout=2100 on that queue.
- Run lock TTL raised to 2100s so a long step can't be run concurrently by a
duplicate tick-dispatched job.
- StartHostOnboarding creates host+run in a transaction, dispatches after commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- RunRunner returns early for a waiting run whose next_attempt_at is in the
future, so stale/duplicate jobs can't bypass backoff.
- RegisterCapacity takes the largest VM-capable datastore instead of summing
overlapping pools (local + local-lvm), preventing placement overcommit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A retried step (manual or after a timeout) now gets a fresh started_at, so
RunRunner no longer immediately re-detects the timeout and burns the retry
budget without executing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- StepResult::poll — polling steps (reboot) wait without consuming the retry
budget; the step owns its deadline. Reboot maxDuration > deadline.
- Failed runs move a Host subject to 'error' via ProvisioningSubject hook
(no host stuck 'onboarding').
- ConfigureWireguard allocates + reserves the wg_ip under a global lock;
unique index on hosts.wg_ip as a backstop against duplicate addresses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CreateAutomationToken now bootstraps the pveum role/user/token over the
authenticated SSH session (a fresh host has no API token yet); ProxmoxClient
is read-only in A.
- ConfigureWireguard re-verifies the handshake on the idempotent replay path,
never advancing over a dead tunnel.
- PhpseclibRemoteShell treats a missing SSH exit status as failure (255).
- connectWithKey verifies the pinned host key fingerprint on later logins.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo