Uptime monitoring for the services the fleet hosts: HTTP and TCP probes with a live up/down status
board. Completes the monitoring trio (alerts, certs, uptime).
- HealthService::probe(check): native — Laravel HTTP client (withoutRedirecting; reports only the
status code + latency, never the response body → no internal-content exfil) for http, a PHP TCP
stream for tcp. No SSH, no shell, so the target is never interpolated into a command. 2xx/3xx = up;
a 4xx/5xx means "answered but unhealthy"; a connect error = down.
- Health\Index page (route /health, `operate`-gated: route + mount + per-method). Add/remove checks
(http target validated as a real http(s) URL; tcp target as a hostname/IP + a required port). Lazy
per-check probe, each guarded. Up/total summary. Delete via signed ConfirmToken + R5 modal.
Add/delete audited.
- lang/{de,en}/health.php + audit.php health.check_* + shell.nav_health (de/en parity). No emoji.
11 new tests: service (http 2xx up / 5xx down / connection-error down, tcp refused-port down),
component (route gating, add http + audit, reject non-url http target, tcp requires host+port,
reject missing port, scan probes each check, delete via confirmed token). 742 tests green, Pint,
lang parity, self-reviewed. A PUBLIC (unauthenticated) status page is a documented future.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Monitor the certificate expiry of configured TLS endpoints (the panel's own domain, fleet
services, anything reachable) and flag the ones due soon — before they lapse.
- CertService::check(host, port) reads the peer certificate NATIVELY via a PHP SSL stream +
openssl_x509_parse — no SSH, no shell, so `host` is never interpolated into a command. Trust is
intentionally NOT verified (verify_peer off): the goal is to report expiry even for a self-signed
or already-expired cert. certInfo() (subject/issuer/expiry/daysLeft) is split out for unit testing
against a generated cert; status() maps daysLeft → ok / warning (<30) / critical (<7) / expired.
- Certs\Index page (route /certs, `operate`-gated: route + mount + per-method). Add/remove endpoints
(host validated as a hostname/IP — rejects scheme/path/shell chars; port 1-65535). Lazy per-endpoint
scan, each guarded. Delete via signed ConfirmToken + R5 modal. Add/delete audited. Internal
endpoints are allowed on purpose (monitoring internal service certs is legitimate; the check only
reads a cert, it POSTs nothing).
- lang/{de,en}/certs.php + audit.php cert.endpoint_* + shell.nav_certs (de/en parity). No emoji.
10 new tests: service (certInfo extracts subject/expiry from a generated cert, status thresholds,
check errors on an unreachable endpoint), component (route gating, add + audit, reject scheme/
metachar host + out-of-range port, scan checks each endpoint, delete via confirmed token). 730 tests
green, Pint, lang parity, self-reviewed. LE issuance is a documented future (needs ACME on the host).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The multi-server amplifier: run a command (ad-hoc or a saved runbook) across the whole fleet,
a group, or a selection, with a per-server result. Because it is arbitrary remote execution,
authorization + audit are the whole feature.
- The command is intentionally arbitrary; it runs via FleetService::runPlain (the credential's
login user, base64 transport, NO extra sudo) — the same privilege the web terminal already
grants, batched. No injection to prevent; the concern is who may run + that every run is logged.
- CommandRunner.run: per-server runPlain, one server failing never aborts the batch, output
byte-capped (256 KiB) + mb_scrub'd before it reaches a Livewire property.
- Commands\Index page, fully `operate`-gated: route can:operate + mount() + a per-method gate() on
run/runRunbook/execute/createRunbook/confirmDeleteRunbook/deleteRunbook. Every run goes through a
signed single-use ConfirmToken + R5 confirm modal ("run on N servers?"); the modal audits
command.run from the sealed token so execute() does not re-audit. Targets resolve client UUIDs →
ids server-side (withActiveCredential + inGroup + whereIn), and {command, serverIds} is SEALED in
the token — a client can't swap the command or targets after confirm. A forged token is a no-op.
- Runbooks: saved name+command, operate-gated CRUD, delete via ConfirmToken.
- lang/{de,en}/commands.php + audit.php command.run / runbook.* + shell.nav_commands (de/en). No emoji.
15 new tests: CommandRunner (runs each server, one-failing-doesn't-abort, nonzero=not-ok, byte-cap
+scrub), component (route gating, ad-hoc run opens confirm, empty-command / no-target errors, execute
runs the SEALED targets, forged token runs nothing, group scope targets only members, runbook create
+audit + run + delete-via-token). 700 tests green, Pint, lang parity, Codex-reviewed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turns the existing metrics into an ops tool: rules that fire when a server crosses a
threshold and resolve when it recovers, with e-mail + webhook notifications. The event
sink later features (cert-expiry, uptime-down, patch-available, posture-drop) plug into.
- alert_rules (metric cpu|mem|disk|load|offline, comparator gt|lt, threshold, scope
all|group|server) + alert_incidents (firing|resolved, FK cascade).
- AlertEvaluator — per-server state machine: opens ONE firing incident on a fresh breach
(notifies), resolves on recovery (notifies), dedups a sustained breach (one breach = one
alert). Numeric metrics skip a truly-offline server (stale reading); the `offline` rule
fires only on offline, not a reachable "warning".
- AlertNotifier — best-effort e-mail (queued Mailable to configured recipients, falls back
to admin e-mails) + generic webhook JSON POST (Slack/Discord/Mattermost/custom; only http(s)
URLs). Every channel is wrapped so a broken SMTP/unreachable hook is logged, never thrown.
- PollMetrics evaluates after each poll (online, fresh status) and on a failed poll (offline),
each guarded so an alerting error can never kill the poll loop.
- Alerts\Index page, manage-panel (admin): route can:-mw + mount() + per-method gate() on every
mutation. Rule delete via signed ConfirmToken + R5 modal (no double audit). Scope group/server
resolves a client UUID to an id server-side (never trusts a client id). Channels saved to
Settings. Sidebar "Alarme" nav + firing-incident badge (cached 60s).
- lang/{de,en}/alerts.php + audit.php alert.* + shell.nav_alerts (de/en parity). No emoji.
Codex review raised three MEDIUMs, all fixed here:
- Webhook SSRF: strict http(s) scheme + reject hosts resolving to loopback/private/link-local/
reserved ranges (blocks the 169.254.169.254 cloud-metadata classic), validated at save AND send,
redirects disabled — so an admin-configured hook can't be pointed at an internal service.
- Incident dedup is now a DB invariant: a unique `firing_key` ("{rule}:{server}" while firing,
null on resolve) means two concurrent poll ticks can't both open a duplicate incident (the loser
hits a unique violation, caught → no-op).
- `load` is float-safe: threshold + incident value are decimal, compared as floats, so load 1.5 vs
threshold 1 fires (it used to truncate to 1 > 1 = false).
31 new tests: evaluator (fire/resolve/dedup, offline vs warning, scope all/group/server, disabled,
fractional-load, firing_key unique, key-freed-on-resolve), notifier (email/fallback/webhook/
non-http-ignored/failure-swallowed/SSRF-reject/public-accept), component (RBAC route gating, rule
CRUD, uuid→id scope resolve, offline→threshold 0, toggle, delete via token, channels, unsafe-webhook
rejected, incidents render). 669 tests green, Pint, lang parity, Codex-reviewed (fixes applied).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Foundation for the feature program (groups are the scoping primitive alerts, ad-hoc
commands, bulk actions and patch/posture will target). Lean by design: named groups,
no hierarchy, no per-group RBAC.
- server_groups (uuid route key, unique name, color = a @theme token key coerced to a real
token on save) + server_server_group M2M pivot (cascade on delete; servers survive a group
delete). Server::groups() + a `scopeInGroup`.
- Servers\Groups full-page component, manage-fleet (admin) only — route can:-mw + mount()
abort_unless + a per-method gate() on every mutation (create/rename/setColor/toggleMember/
confirmDelete/deleteGroup), since /livewire/update does not re-run route mw. create/rename/
recolour/membership audit directly; DELETE goes through the signed single-use ConfirmToken +
the R5 confirm modal (which writes group.delete from the sealed descriptor, so the handler
does not double-audit). 'groupConfirmed' added to ConfirmToken::ACTIONS.
- /servers/groups is registered BEFORE /servers/{server} so "groups" isn't captured as a uuid.
- Servers\Index: a #[Url] group filter (uuid; shareable) — an unknown uuid falls back to all,
never errors. Filtering is open to every role; only management is admin-gated. A "Gruppen"
button (admin) on the fleet header; colour pills use a literal @theme token map (R3).
- lang/{de,en}/groups.php + audit.php group.* labels (de/en parity). No emoji.
19 new tests (model + component: uuid/colour-allow-list/relations/cascade, RBAC route gating,
create/rename/recolour/membership, delete-via-confirmed-token, forged-token no-op, filter).
638 tests green, Pint, lang parity, Codex-reviewed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A guided tour for new operators: a dimmed+blurred overlay with an enterprise description card
(mono eyebrow "01 / 06", display title, body, progress bar, back/skip/next). A step with a
target spotlights the matching sidebar nav item (a box-shadow "hole" lights it while everything
else dims); welcome + finish are centred cards. On narrow viewports (sidebar hidden) it falls
back to centred cards.
- auto-opens once per account (users.onboarding_tour_completed_at is null → autostart); skip or
finish calls Tour::markSeen() which stamps it, so it never auto-opens again.
- relaunchable any time from Settings (a client-side 'onboarding:start' window event — no DB change).
- @persist'd in the layout + a sessionStorage guard so a wire:navigate or fast reload right after
dismissal can't re-open it before markSeen persists.
- nav-item now merges its attribute bag (data-tour passes through without duplicating class/href).
Browser-verified: auto-open → spotlight walks Dashboard/Servers/Terminal/Settings → skip stamps +
closes → navigation doesn't reopen → Settings relaunch works; zero console errors. 4 feature tests;
full suite 473 pass; Codex review CLEAN.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The "Clusev host" terminal was a shell inside the sidecar container (node@clusev, no
sudo, not the host). It is now a REAL SSH login into the host machine, like any fleet
server — so it shows the real root@<host> prompt with full rights.
- HostCredential (encrypted singleton) holds the host SSH login; a HostShell modal lets
the operator set host/port/user + password|key. The clear-text secret is never echoed
back; on edit a blank secret keeps the stored one, and switching auth method requires a
fresh secret (no password silently reused as a key).
- open('host') mints a kind=host token only when a login is configured, else opens the
setup modal; the internal resolve endpoint returns a server-shaped SSH spec (decrypted,
internal-net only) for kind=host, or 404 when unconfigured.
- compose: extra_hosts host.docker.internal:host-gateway so the sidecar reaches the host
sshd; Caddy now 404s /_internal/* at the public edge (the sidecar uses app:80 directly).
- rail gains a debounced search box past 5 servers; removed the now-obvious PTY hint line.
Browser-verified (R12): host tile shows "not configured" + gear → modal → SSH login as
root runs commands, zero console errors; search filters; resolve returns the host spec.
15 terminal tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A dedicated /terminal page with a target rail: one terminal per fleet server (SSH)
plus one for the Clusev host itself. php-fpm cannot hold an interactive PTY, so a
small Node sidecar (ws + ssh2 + node-pty) bridges xterm.js to the PTY.
Flow: the Livewire page mints a single-use, 60s TerminalSession token and dispatches
it to an Alpine xterm island; the browser opens a same-origin WS (/terminal/ws,
proxied by nginx in dev / Caddy in prod to terminal:3000); the sidecar burns the
token at an internal resolve endpoint (shared-secret header, private network only)
for a connection spec, then opens an SSH PTY (decrypted credential, never sent to the
browser) or a local host PTY. Real shell = native Tab/Shift-Tab completion.
Security:
- token burned ATOMICALLY (single conditional UPDATE) — no double-open race
- resolve endpoint guarded by hash_equals shared secret, exempt from CSRF + PanelScheme
host enforcement (private net only), returns decrypted creds over the internal net
- sidecar enforces same-origin on the WS upgrade (hostname match, port-agnostic)
- single-use 60s tokens, swept daily so the table can't grow unbounded
Verified in-browser (R12): host + server terminals connect, accept keyboard input,
run commands, Tab-complete; cross-origin WS rejected; zero console errors. 9 feature
tests cover minting, the resolve spec, single-use burn, expiry, and locked creds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A CPU/RAM/Disk history graph over selectable ranges (1h/24h/7d/30d) on the server
detail page, backed by a persisted time series (the live 15s view stays cache-only).
- metric_samples table (server_id cascade, cpu/mem/disk %, load, sampled_at indexed).
- clusev:sample-metrics (scheduled every minute, mirrors clusev:wg-sample): persists
one row per server from the cached live reading, prunes beyond --retention (30d).
- MetricHistory::series() buckets + averages samples for a window; empty buckets stay
null so the chart shows gaps, not fabricated values.
- Servers/Show: range selector + a gap-aware SVG line chart (theme-token strokes,
R3-compliant, mirrors the dashboard chart). historyRange is clamped via setRange()
and the updatedHistoryRange() hook (it's a public, client-settable property).
Codex review: fixed a right-edge bucket off-by-one, clamped percentages on write, and
guarded the public range property against direct client values. Browser-verified: the
chart renders with data, range buttons, legend and gaps, zero console errors. 447
tests green, pint clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adversarial audit of the auth surface confirmed several exploitable gaps. Every fix
auto-expires and never permanently locks the control plane (bare-IP recovery host and
the clusev:reset-admin CLI stay open).
Login:
- per-IP (20/min) + per-account (30/15min) buckets on top of the existing email+IP 5/min,
so a distributed multi-IP brute-force of the single admin account is capped instead of
scaling linearly with IPs. ip/acct counters decay on their own (a valid login doesn't
reset a flood's budget).
- constant-time: the unknown-email branch now runs one bcrypt against a fixed DUMMY_HASH
(evaluated before the `! $user ||` short-circuit), closing the account-enumeration
timing oracle.
2FA:
- TOTP replay protection — verifyTotp uses verifyKeyNewer with a persisted last-used step
(two_factor_last_used_step migration), so each 30s code is single-use. Passes 0 (never
null) as the old step, since null makes verifyKeyNewer return bare `true` not the step.
- challenge + 2FA-proof reset gain an IP-independent per-account backstop, capping the
distributed TOTP/backup brute-force surface.
Defense-in-depth / DoS:
- global throttle on /livewire/update (180/min per user-id-or-IP) via setUpdateRoute,
keeping 'web' + the persistent EnsureSecurityOnboarded gate intact.
- reauth throttle (5/min per user) on password change/profile; SMTP test-send capped
(3/10min per user) so the panel can't relay spam.
- SshClient gets a short, separate connect timeout (~5s) so a dead/tarpit host can't hold
a worker across several sequential 15s reads (the observed 8.3s /livewire/update DoS).
- profile password policy raised to 12 chars + mixed case + numbers (was 10).
Tests: BruteForceHardeningTest covers TOTP replay, per-email + per-IP login caps,
identical unknown/known rejection, and reauth throttling. Full suite 187 green; Codex
clean; verified in a real browser (login/polls/Livewire updates unaffected by the throttle).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- credential_id can exceed the unique-index key length; store it in TEXT and
enforce uniqueness on an auto-derived sha256 hash column (lookups use the hash).
- Disabling 2FA now also deletes the user's security keys + backup codes, so
re-enrolling starts clean and old factors never silently revive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add web-auth/webauthn-lib ^5.3, the webauthn_credentials table (per-user unique
credential_id), the WebauthnCredential model and User hasMany relation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per operator feedback ("das soll automatisch sein, die Caddy-Config muss man nicht
sehen, nur 'SSL steht'"):
- TLS is fully automatic via Caddy (auto-issue + auto-renew + HTTP→HTTPS). The System
page now shows only the STATUS (domain + access URL) — the generated Caddy config and
the manual `caddy reload` command are gone, and the domain/email edit form is removed.
- The panel domain is an INSTALL-TIME value (APP_DOMAIN → config('clusev.domain')), so it
stays consistent with APP_URL, the Reverb (wss) endpoint and cookie security — none of
which can be re-derived at runtime (Caddy's site address + the browser's Reverb host are
fixed at install/build, and .env is never rewritten). DeploymentService is now read-only.
- Removed AppServiceProvider's runtime app.url override (it let a stale DB value shadow the
install-time URL). Migration drops the inert empty dashboard_domain/dashboard_email rows
(a real configured value is preserved as history; re-apply via install.sh — see CHANGELOG).
- Hardening checklist: all toggle buttons share one style (secondary/bordered) instead of
varying with state (R10).
Bump 0.2.0 -> 0.2.1 + CHANGELOG. Pint clean; Codex reviewed (the remaining note is a
deliberate install-time-domain tradeoff — a runtime/DB domain cannot be served by the
templated Caddy without a false "TLS active" status); R12 — system/server-show/dashboard
HTTP 200, 0 console errors, raw config + edit form gone, buttons uniform.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Security pivot — make server hardening, SSH access, firewall, dashboard domain/TLS
and the release channel controllable from the dashboard (no SSH needed), with guards
so a remote change can never lock the operator out.
Foundation
- ssh_credentials gains name + disabled_at + last_used_at; CredentialVault refuses a
disabled credential. New key/value Setting model. FleetService::runPrivileged() /
runPlain() — central sudo-aware exec (base64-wrapped sh -c), live-verified as root.
A · control-plane self-hardening
- SecurityHeaders middleware: env-aware CSP (allows the Vite dev origin in dev, strict
same-origin in prod), X-Frame-Options DENY, nosniff, Referrer/Permissions-Policy,
HSTS when secure. 2FA brute-force throttle (5/60s). install.sh sets
SESSION_SAME_SITE=strict + EXPIRE_ON_CLOSE + SECURE_COOKIE (only behind TLS).
B · SSH credential management
- name/label on the access; a credential card on the server page with Bearbeiten /
Sperren-Entsperren (kill-switch) / Löschen (R5), all audited.
C · server hardening from the dashboard (guards + confirmation)
- HardeningService (PermitRootLogin no, PasswordAuthentication no, fail2ban,
unattended-upgrades) + FirewallService (UFW). HardeningAction modal previews the
exact root commands before applying. GUARDS: refuse to disable password-login when
Clusev itself logs in by password or no key exists; UFW opens the real sshd port +
80/443 before enabling. Live-verified non-destructively (previews, the password
guard refusing, ufw status read).
D+E · System page (/system)
- Dashboard Domain + Let's-Encrypt email (Setting) -> DeploymentService renders the
matching Caddy site block (honest: stages the file + shows the reload command, never
fakes TLS). Release channel (stable|beta|dev) configurable; Versions reads it.
Built largely by 4 parallel agents into disjoint files; shared files integrated + the
security-critical bits hardened by hand (the password-auth lock-out guard + env-aware CSP).
Verified (R12): /system + server detail + all 8 routes 200 / 0 console errors (CSP does
not break Livewire/Alpine/Vite); credential card + 5 hardening "Anwenden" buttons render;
the hardening modal opens with the command preview; System persists domain/channel +
renders valid Caddy config; runPrivileged runs as root + the vault refuses disabled creds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses "chart/gauges don't update" and "design": the visible live layer no
longer depends on the Reverb WS path (which was unreliable through the dev NAT).
- Poller is now the single SSH metrics source: FleetService::applyMetrics writes
the latest reading + a rolling cpu/mem history into the cache (and the DB).
- Dashboard reads latest + history from cache (no SSH on web render); the big
chart and KPI sparklines render real history; wire:poll.10s refreshes them.
- Server-Details gauges read the poller-updated row via wire:poll.10s; the donut
rings get a size="lg" variant (h-24) so they read clearly + animate.
- FleetSeeder no longer seeds a fake fleet (those servers had no credentials and
showed no data). It seeds one real server from CLUSEV_DEMO_SSH_* env vars, or
an empty fleet otherwise. Existing fake servers removed from the demo DB.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve all 12 confirmed findings from the SSH-integration review.
Security:
- TOFU host-key pinning (new VerifiesHostKey trait + servers.ssh_host_key column):
SshClient/Sftp record the server host key on first connect and refuse the
connection on any later mismatch — phpseclib does not pin host keys itself.
- PollMetrics keeps one long-lived SSH connection per server and reuses it across
ticks (a single login, not one per interval) — avoids fail2ban/auth-log churn;
dead connections are dropped and re-established next tick.
- @js() escaping for every server-controlled value interpolated into wire:click
(file names, paths, service names, SSH-key comments/fingerprints) — prevents
JS-string breakage / injection from untrusted remote data.
Correctness:
- parseKeys regex makes the key comment optional (keys without a comment were
silently dropped).
- parseVolumes pops the three trailing single-token df fields and rejoins the
rest, so mount points containing spaces parse correctly.
- Sftp gains a disconnect() method to match SshClient (explicit cleanup).
Rules:
- Services "Start" -> "Starten" (R9, German).
- Files breadcrumb buttons get min-w-11 + padding (R7, >=44px touch target).
Verified live: host key stored + mismatch refused + correct key reconnects;
poller reuse polls ok; all pages still render real data.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
boban