- Clone: recover via vmExists() when the task ref was lost, instead of
re-cloning the reserved vmid (which Proxmox rejects).
- Deploy: persist the DB password ENCRYPTED before compose up and reuse it on
retry, so a crash can't regenerate a mismatching credential.
- Complete: gate credential delivery on a credentials_sent breadcrumb so a
retry after a crash doesn't re-send the email.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Failed run releases the instance (status=failed) so its quota stops counting
against host capacity.
- CompleteProvisioning activates instance+order only AFTER onboarding tasks +
credential delivery succeed.
- ConfigureNetwork polls until the guest has an address (no wrong Traefik route).
- customers.email unique + race-safe customer resolution in the intake action.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RunRunner::onPoll no longer resets started_at, so a poll step's own deadline
(WaitForGuestAgent 270s, ConfigureDnsAndTls cert 840s) actually fires instead
of resetting every poll. maxDuration raised above each own-deadline so the
step's fail wins over the generic timeout. Shared core fix (A + B).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Require a real customer email (no unknown@ fallback merging customers).
- Stripe signature: enforce 5-min replay tolerance + accept any rotating v1.
- DeployApplicationStack no longer persists the DB password in the run context.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Stripe webhook fails closed when the signing secret is missing (outside local/testing).
- ReserveResources places + creates the instance under a per-datacenter lock so
concurrent orders can't overcommit a host.
- CloneVirtualMachine reserves the vmid + breadcrumb BEFORE cloning, so a crash
can't mint a new vmid and orphan the first clone.
- CreateCustomerAdmin checks user existence and resets the password instead of
re-running user:add on retry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Stripe: only checkout.session.completed with payment_status=paid (dedupes the
paired payment_intent event and blocks unpaid async sessions).
- Capture the guest IP (ConfigureNetwork) and point Traefik at the VM, not the
Proxmox host, so ACME/HTTP-01 can reach Nextcloud.
- Resize the disk to an absolute target (not '+…') so a retry can't double it.
(Codex #4 timeout was a false positive — started_at resets per step.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Admin /admin/provisioning bound to real runs+steps (poll+admin.runs). Embedded
CustomerProvisioning card shows the logged-in customer their own run live
(per-customer StepAdvanced channel, email-bridged authz). Shared BuildsRunSteps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Prevents a second worker from consuming the follow-up job and bailing on the
still-held run lock, which would stall the run until the next scheduler tick.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keeps the wg_peer resource gated on a verified handshake (idempotency) while
ensuring PurgeHost can always remove the hub peer, even on terminal failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard checks setup command results and only records the wg_peer
resource after the handshake verifies, so a transient setup failure re-runs
the full setup instead of getting stuck on the idempotent path.
- Host removal now deactivates immediately and queues PurgeHost, which deletes
runs under the runner lock (waiting out a long step) — no 15s LockTimeout error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- PhpseclibRemoteShell sets a command timeout (default 2000s) so apt
full-upgrade/install don't hit phpseclib's ~10s default and fail.
- Host removal dispatches RemoveWireguardPeer to the provisioning queue, which
runs in the worker that owns wg0 (the web container can't manage WireGuard).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- RunRunner catches step-resolution errors (removed/renamed pipeline) and
fails the run terminally instead of looping forever every tick.
- queue-provisioning worker gains NET_ADMIN + tun + persistent wireguard
volume + published UDP port so LocalWireguardHub can manage wg0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Removing a host now removes its WireGuard hub peer so a freed wg_ip can't
route to the removed server via a stale peer.
- RebootIntoPveKernel clears reboot_issued/deadline when it times out, so the
manual retry action actually re-issues a reboot instead of failing instantly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hub_ip is now defined in config (default: subnet .1), so a custom
CLUPILOT_WG_SUBNET no longer pings the wrong handshake target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Dedicated 'provisioning' queue connection (retry_after 2400) + worker
(--timeout=2100); AdvanceRunJob tries=1, timeout=2100 on that queue.
- Run lock TTL raised to 2100s so a long step can't be run concurrently by a
duplicate tick-dispatched job.
- StartHostOnboarding creates host+run in a transaction, dispatches after commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- RunRunner returns early for a waiting run whose next_attempt_at is in the
future, so stale/duplicate jobs can't bypass backoff.
- RegisterCapacity takes the largest VM-capable datastore instead of summing
overlapping pools (local + local-lvm), preventing placement overcommit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A retried step (manual or after a timeout) now gets a fresh started_at, so
RunRunner no longer immediately re-detects the timeout and burns the retry
budget without executing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- StepResult::poll — polling steps (reboot) wait without consuming the retry
budget; the step owns its deadline. Reboot maxDuration > deadline.
- Failed runs move a Host subject to 'error' via ProvisioningSubject hook
(no host stuck 'onboarding').
- ConfigureWireguard allocates + reserves the wg_ip under a global lock;
unique index on hosts.wg_ip as a backstop against duplicate addresses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CreateAutomationToken now bootstraps the pveum role/user/token over the
authenticated SSH session (a fresh host has no API token yet); ProxmoxClient
is read-only in A.
- ConfigureWireguard re-verifies the handshake on the idempotent replay path,
never advancing over a dead tunnel.
- PhpseclibRemoteShell treats a missing SSH exit status as failure (255).
- connectWithKey verifies the pinned host key fingerprint on later logins.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo