Metric history: the per-minute sampler (clusev:sample-metrics) now reads
cpu/mem/disk from the `servers` DB row — the same source the live gauges read,
shared via MariaDB — instead of a cross-container Redis cache the poller may
not share in prod. It gates on last_seen_at freshness (skip offline/unpolled)
and keeps `load` as a best-effort cache read. History now fills whenever the
gauges have data, independent of the cache driver or container split — the
prod detail page no longer shows "Noch keine Verlaufsdaten" after hours.
Service-action failures are persisted: a failed systemctl start/stop/restart
writes a red service.<op>.failed AuditEvent with the FULL systemctl reason in
meta['output'] (mb_scrub-ed, 4000-cap), shown expandable in the audit log — a
truncated, ephemeral toast is no longer the only trace. The failure toast is a
short pointer to the audit log, level=error, and error toasts now linger 9s.
Alert delivery: a warning banner appears on the alerts page when NO channel
would reach anyone (no SMTP + no safe webhook + no Gotify). A new Gotify push
channel (server URL + encrypted app token) posts title/message/priority with
the token in X-Gotify-Key; a self-hosted LAN Gotify is allowed (scheme-checked,
not private-range-blocked like the SSRF-guarded webhooks) — a real push when a
service goes down, no e-mail needed.
Secret-retarget hardening (Codex-found class): changing a target while leaving
its secret blank could send the OLD secret to the NEW endpoint. Closed for the
Gotify token, the SMTP password (save + test-send + the runtime mailer), and
the host-terminal SSH credential (username/port change) — each drops/forbids
the stale secret before the new target is stored. Setting::uncached() reads the
security-sensitive keys DB-direct so a cache-repopulation race can't re-pair an
old secret with a new target. Three Codex rounds, clean; 778 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ROOT CAUSE of "alert fires but no e-mail", third and final layer: the
metrics poller (clusev:poll-metrics, a long-lived supervisor process) is
what queues alert mail — and Laravel bakes the DEFAULT MAILER NAME into a
mailable at Mail::to()->queue() time. A poller that booted before SMTP
was configured bakes `log` into every alert e-mail forever; the worker
then faithfully "sends" them into laravel.log no matter how correct its
own config is. This is why manual test sends (fresh processes → smtp)
arrived while real alert mails (poller → log) never did.
Fix: extract the SMTP-settings application into App\Support\MailSettings
(::apply(), now with Mail::purge on change so long-lived processes
rebuild the transport) and call it from all three process shapes:
- web requests → AppServiceProvider::boot()
- queue worker → Queue::before
- poller/loops → AlertNotifier::notify() right before queueing,
so the CURRENT mailer name is baked in.
Also: enterprise HTML alert e-mail (frontend-design). Dark Tactical
Terminal card — signal rail + status pill (red ALARM / green BEHOBEN),
rule headline, mono terminal readout (server/metric/value/threshold),
bulletproof dashboard CTA, CID-embedded logo (no external assets; renders
in Gmail/Outlook with images blocked), full de/en, plain-text part kept
as fallback. Offline rules render state text (offline/back online)
instead of a bare 0/1.
Verified: app restarted (poller reloaded), synchronous + queued sends to
the admin both deliver over real SMTP (476ms send, no new log-mailer
entries). 750 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turns the existing metrics into an ops tool: rules that fire when a server crosses a
threshold and resolve when it recovers, with e-mail + webhook notifications. The event
sink later features (cert-expiry, uptime-down, patch-available, posture-drop) plug into.
- alert_rules (metric cpu|mem|disk|load|offline, comparator gt|lt, threshold, scope
all|group|server) + alert_incidents (firing|resolved, FK cascade).
- AlertEvaluator — per-server state machine: opens ONE firing incident on a fresh breach
(notifies), resolves on recovery (notifies), dedups a sustained breach (one breach = one
alert). Numeric metrics skip a truly-offline server (stale reading); the `offline` rule
fires only on offline, not a reachable "warning".
- AlertNotifier — best-effort e-mail (queued Mailable to configured recipients, falls back
to admin e-mails) + generic webhook JSON POST (Slack/Discord/Mattermost/custom; only http(s)
URLs). Every channel is wrapped so a broken SMTP/unreachable hook is logged, never thrown.
- PollMetrics evaluates after each poll (online, fresh status) and on a failed poll (offline),
each guarded so an alerting error can never kill the poll loop.
- Alerts\Index page, manage-panel (admin): route can:-mw + mount() + per-method gate() on every
mutation. Rule delete via signed ConfirmToken + R5 modal (no double audit). Scope group/server
resolves a client UUID to an id server-side (never trusts a client id). Channels saved to
Settings. Sidebar "Alarme" nav + firing-incident badge (cached 60s).
- lang/{de,en}/alerts.php + audit.php alert.* + shell.nav_alerts (de/en parity). No emoji.
Codex review raised three MEDIUMs, all fixed here:
- Webhook SSRF: strict http(s) scheme + reject hosts resolving to loopback/private/link-local/
reserved ranges (blocks the 169.254.169.254 cloud-metadata classic), validated at save AND send,
redirects disabled — so an admin-configured hook can't be pointed at an internal service.
- Incident dedup is now a DB invariant: a unique `firing_key` ("{rule}:{server}" while firing,
null on resolve) means two concurrent poll ticks can't both open a duplicate incident (the loser
hits a unique violation, caught → no-op).
- `load` is float-safe: threshold + incident value are decimal, compared as floats, so load 1.5 vs
threshold 1 fires (it used to truncate to 1 > 1 = false).
31 new tests: evaluator (fire/resolve/dedup, offline vs warning, scope all/group/server, disabled,
fractional-load, firing_key unique, key-freed-on-resolve), notifier (email/fallback/webhook/
non-http-ignored/failure-swallowed/SSRF-reject/public-accept), component (RBAC route gating, rule
CRUD, uuid→id scope resolve, offline→threshold 0, toggle, delete via token, channels, unsafe-webhook
rejected, incidents render). 669 tests green, Pint, lang parity, Codex-reviewed (fixes applied).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
boban