Prevents a second worker from consuming the follow-up job and bailing on the
still-held run lock, which would stall the run until the next scheduler tick.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keeps the wg_peer resource gated on a verified handshake (idempotency) while
ensuring PurgeHost can always remove the hub peer, even on terminal failure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard checks setup command results and only records the wg_peer
resource after the handshake verifies, so a transient setup failure re-runs
the full setup instead of getting stuck on the idempotent path.
- Host removal now deactivates immediately and queues PurgeHost, which deletes
runs under the runner lock (waiting out a long step) — no 15s LockTimeout error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- PhpseclibRemoteShell sets a command timeout (default 2000s) so apt
full-upgrade/install don't hit phpseclib's ~10s default and fail.
- Host removal dispatches RemoveWireguardPeer to the provisioning queue, which
runs in the worker that owns wg0 (the web container can't manage WireGuard).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Image installs wireguard-tools/iproute2; the provisioning worker brings up
wg0 before queue:work so LocalWireguardHub can manage peers.
- ConfirmRemoveHost deletes each run under its run:<uuid> lock so an in-flight
worker can't keep mutating the server or write to a deleted run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- RunRunner catches step-resolution errors (removed/renamed pipeline) and
fails the run terminally instead of looping forever every tick.
- queue-provisioning worker gains NET_ADMIN + tun + persistent wireguard
volume + published UDP port so LocalWireguardHub can manage wg0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Removing a host now removes its WireGuard hub peer so a freed wg_ip can't
route to the removed server via a stale peer.
- RebootIntoPveKernel clears reboot_issued/deadline when it times out, so the
manual retry action actually re-issues a reboot instead of failing instantly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hub_ip is now defined in config (default: subnet .1), so a custom
CLUPILOT_WG_SUBNET no longer pings the wrong handshake target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Dedicated 'provisioning' queue connection (retry_after 2400) + worker
(--timeout=2100); AdvanceRunJob tries=1, timeout=2100 on that queue.
- Run lock TTL raised to 2100s so a long step can't be run concurrently by a
duplicate tick-dispatched job.
- StartHostOnboarding creates host+run in a transaction, dispatches after commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- RunRunner returns early for a waiting run whose next_attempt_at is in the
future, so stale/duplicate jobs can't bypass backoff.
- RegisterCapacity takes the largest VM-capable datastore instead of summing
overlapping pools (local + local-lvm), preventing placement overcommit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A retried step (manual or after a timeout) now gets a fresh started_at, so
RunRunner no longer immediately re-detects the timeout and burns the retry
budget without executing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- StepResult::poll — polling steps (reboot) wait without consuming the retry
budget; the step owns its deadline. Reboot maxDuration > deadline.
- Failed runs move a Host subject to 'error' via ProvisioningSubject hook
(no host stuck 'onboarding').
- ConfigureWireguard allocates + reserves the wg_ip under a global lock;
unique index on hosts.wg_ip as a backstop against duplicate addresses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CreateAutomationToken now bootstraps the pveum role/user/token over the
authenticated SSH session (a fresh host has no API token yet); ProxmoxClient
is read-only in A.
- ConfigureWireguard re-verifies the handshake on the idempotent replay path,
never advancing over a dead tunnel.
- PhpseclibRemoteShell treats a missing SSH exit status as failure (255).
- connectWithKey verifies the pinned host key fingerprint on later logins.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Real hosts list, add-host form (StartHostOnboarding), host detail with live
progress stepper (Reverb + wire:poll fallback), retry failed run, remove via
wire-elements/modal (deregister only). DE+EN. 8 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Separate admin console at /admin, gated to is_admin users (EnsureAdmin middleware
+ migration/seed; a plain customer user is seeded to prove the gate). Distinct dark
aesthetic achieved purely by token scoping: .theme-admin overrides every CSS design
token to a dark graphite / signal-orange palette, so ALL shared components (button,
card, badge, table, stat, chart) render dark with zero new markup (R3). Chart island
now reads tokens from its own element, so charts are theme-aware.
Sections (each full-page class-based Livewire, English routes R13, localized DE/EN):
- Overview: fleet KPIs, fleet-growth line, host-load bars, MRR bars, active runs, alerts.
- Customers: table + plan doughnut. Instances: fleet table (vmid/host/storage).
- Hosts: capacity cards (storage/CPU bars). Provisioning: runs table + live stepper.
- Revenue: MRR/ARR/ARPU/churn KPIs, MRR line, plan doughnut, recent payments.
- Locale-aware month labels/currency (Carbon/Number).
18 new Pest tests (guest redirect / non-admin 403 / admin render per section) → 44
green. R12 browser: all six admin pages HTTP 200, ZERO console errors (Chart.js dark).
Codex (R15) — clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Chart.js as an Alpine island (x-ui.chart): configs are PHP arrays; colours use
token: strings resolved from CSS design tokens at runtime (R3). Respects
reduced-motion; IBM Plex chart defaults.
- Tokens: soft --radius-xl (20px), --success-bright for dots/rings/charts.
Staggered 'rise' reveal keyframe. Global toast in the app shell.
- Overview rebuilt to the customer template: storage doughnut ring, availability
sparkline, KPI tiles, upsell, cloud card, interactive onboarding checklist
(Alpine), backups, modules (Lucide icons — no emoji, R9), activity feed.
- Locale-aware fixture display: Carbon isoFormat dates + Number::format sizes so
the EN dashboard is not mixed-language (R16).
- 14 Pest tests green; R12 browser: /dashboard 0 console errors with Chart.js;
R7 no overflow at 375/768/1280. Reviewed with Codex (R15) — clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nexxo