Metric history: the per-minute sampler (clusev:sample-metrics) now reads
cpu/mem/disk from the `servers` DB row — the same source the live gauges read,
shared via MariaDB — instead of a cross-container Redis cache the poller may
not share in prod. It gates on last_seen_at freshness (skip offline/unpolled)
and keeps `load` as a best-effort cache read. History now fills whenever the
gauges have data, independent of the cache driver or container split — the
prod detail page no longer shows "Noch keine Verlaufsdaten" after hours.
Service-action failures are persisted: a failed systemctl start/stop/restart
writes a red service.<op>.failed AuditEvent with the FULL systemctl reason in
meta['output'] (mb_scrub-ed, 4000-cap), shown expandable in the audit log — a
truncated, ephemeral toast is no longer the only trace. The failure toast is a
short pointer to the audit log, level=error, and error toasts now linger 9s.
Alert delivery: a warning banner appears on the alerts page when NO channel
would reach anyone (no SMTP + no safe webhook + no Gotify). A new Gotify push
channel (server URL + encrypted app token) posts title/message/priority with
the token in X-Gotify-Key; a self-hosted LAN Gotify is allowed (scheme-checked,
not private-range-blocked like the SSRF-guarded webhooks) — a real push when a
service goes down, no e-mail needed.
Secret-retarget hardening (Codex-found class): changing a target while leaving
its secret blank could send the OLD secret to the NEW endpoint. Closed for the
Gotify token, the SMTP password (save + test-send + the runtime mailer), and
the host-terminal SSH credential (username/port change) — each drops/forbids
the stale secret before the new target is stored. Setting::uncached() reads the
security-sensitive keys DB-direct so a cache-repopulation race can't re-pair an
old secret with a new target. Three Codex rounds, clean; 778 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two things the badge/live-update fixes missed:
Height — the enable/disable TOGGLES (Alerts, Servers, Commands, Settings)
were custom buttons at min-h-9 (36px) while x-btn / x-modal-trigger /
status-pill are h-8 (32px), so a status toggle never lined up with the
Delete/Create buttons next to it. Normalise those toggles to h-8.
Verified: Enabled/Delete/Create all render at 32px now.
Live — the topbar "X/Y online" pill and the sidebar server-switcher were
static Blade, so a server going up/down (shown live on the dashboard via
Reverb) didn't reach them until a navigate/refresh. Make the topbar pill
a small polled Livewire island (Shell\FleetStatus) and add wire:poll to
the server-switcher + the Alerts page, so fleet status and firing/
resolved incidents update live without a refresh.
(The alert e-mail is confirmed sending over SMTP — queued mail no longer
hits the log mailer, and a synchronous send to the admin returned SEND_OK
from mail.thinkidoo.com:587. Non-delivery is downstream: spam filtering
or missing SPF/DKIM for hello@clusev.com via that relay — a DNS/mail-
server config, not a panel code issue.)
750 tests; R12-verified (uniform heights, topbar is a live island, zero
console errors).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Feedback on the new feature pages: buttons were oversized and Docker showed an empty list where
containers exist.
- Buttons: the new pages (groups, alerts×2, commands×2, certs, health) plus settings/login-protection
used size="lg" (h-11) on inline form-submit buttons, dwarfing the h-9 inputs and the surrounding
pills. Drop to the default `sm` (h-8) — matching every other form in the app (e.g. the e-mail save
button). The auth full-width CTAs keep size="lg" (their intended use); x-ring size="lg" is unrelated.
- Docker "empty" bug: DockerService::containers used `docker ps --format '{{json .}}'` (needs a modern
Docker AND depends on JSON key casing) and returned [] on ANY failure — so a daemon/permission/
rootless-socket error read as "no containers". Switched to a tab-separated `--format` (portable to
much older Docker, state derived from the Status text) and it now THROWS the real error on failure;
the page shows that reason instead of a misleading empty state (so the actual cause is visible).
- Patch rows: the pending/security/up-to-date pills now match the action button height (h-8) so the
row reads cleanly.
DockerServiceTest updated (tab-parse + state-from-status + throws-real-error). 742 tests green, Pint,
vite build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turns the existing metrics into an ops tool: rules that fire when a server crosses a
threshold and resolve when it recovers, with e-mail + webhook notifications. The event
sink later features (cert-expiry, uptime-down, patch-available, posture-drop) plug into.
- alert_rules (metric cpu|mem|disk|load|offline, comparator gt|lt, threshold, scope
all|group|server) + alert_incidents (firing|resolved, FK cascade).
- AlertEvaluator — per-server state machine: opens ONE firing incident on a fresh breach
(notifies), resolves on recovery (notifies), dedups a sustained breach (one breach = one
alert). Numeric metrics skip a truly-offline server (stale reading); the `offline` rule
fires only on offline, not a reachable "warning".
- AlertNotifier — best-effort e-mail (queued Mailable to configured recipients, falls back
to admin e-mails) + generic webhook JSON POST (Slack/Discord/Mattermost/custom; only http(s)
URLs). Every channel is wrapped so a broken SMTP/unreachable hook is logged, never thrown.
- PollMetrics evaluates after each poll (online, fresh status) and on a failed poll (offline),
each guarded so an alerting error can never kill the poll loop.
- Alerts\Index page, manage-panel (admin): route can:-mw + mount() + per-method gate() on every
mutation. Rule delete via signed ConfirmToken + R5 modal (no double audit). Scope group/server
resolves a client UUID to an id server-side (never trusts a client id). Channels saved to
Settings. Sidebar "Alarme" nav + firing-incident badge (cached 60s).
- lang/{de,en}/alerts.php + audit.php alert.* + shell.nav_alerts (de/en parity). No emoji.
Codex review raised three MEDIUMs, all fixed here:
- Webhook SSRF: strict http(s) scheme + reject hosts resolving to loopback/private/link-local/
reserved ranges (blocks the 169.254.169.254 cloud-metadata classic), validated at save AND send,
redirects disabled — so an admin-configured hook can't be pointed at an internal service.
- Incident dedup is now a DB invariant: a unique `firing_key` ("{rule}:{server}" while firing,
null on resolve) means two concurrent poll ticks can't both open a duplicate incident (the loser
hits a unique violation, caught → no-op).
- `load` is float-safe: threshold + incident value are decimal, compared as floats, so load 1.5 vs
threshold 1 fires (it used to truncate to 1 > 1 = false).
31 new tests: evaluator (fire/resolve/dedup, offline vs warning, scope all/group/server, disabled,
fractional-load, firing_key unique, key-freed-on-resolve), notifier (email/fallback/webhook/
non-http-ignored/failure-swallowed/SSRF-reject/public-accept), component (RBAC route gating, rule
CRUD, uuid→id scope resolve, offline→threshold 0, toggle, delete via token, channels, unsafe-webhook
rejected, incidents render). 669 tests green, Pint, lang parity, Codex-reviewed (fixes applied).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
boban