fix(monitoring): cap monitoring retries at the run budget; dependency-free liveness probe

- MONITORING_ATTEMPTS is capped by the run's max_attempts, so a large value can
  no longer burn the retry budget and fail the very provisioning the degradation
  exists to protect (covered by a new test at the budget boundary)
- /health is now dependency-free (200 in ~2ms with Kuma absent, verified: the
  container healthcheck stays healthy); Kuma state is exclusively in /ready

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat/portal-design
nexxo 2026-07-25 20:19:32 +02:00
parent d1db23398d
commit 46d821a047
4 changed files with 25 additions and 13 deletions

View File

@ -43,7 +43,11 @@ class RegisterMonitoring extends CustomerStep
// `attempt` is 0-based (this call IS attempt number attempt+1), so
// compare one-based: MONITORING_ATTEMPTS=2 really means two tries.
$allowed = max(1, (int) config('provisioning.monitoring.attempts', 2));
// Cap at the run's own retry budget — otherwise a large ATTEMPTS
// value would burn through max_attempts and fail the run, which is
// exactly what this degradation exists to prevent.
$configured = max(1, (int) config('provisioning.monitoring.attempts', 2));
$allowed = max(1, min($configured, (int) $run->max_attempts));
if ($run->attempt + 1 < $allowed) {
return StepResult::retry(30, 'monitoring unavailable: '.$e->getMessage());
}

View File

@ -172,16 +172,12 @@ def _kuma_state() -> tuple[str, str | None]:
@app.get("/health")
def health() -> dict:
"""
LIVENESS (unauthenticated): is this process serving? Always 200 while it is.
Deliberately NOT tied to Kuma: the bridge being restarted because a
*dependency* is down would be wrong. Kuma's state is reported in the body,
and /ready is the endpoint that fails when Kuma is unreachable.
LIVENESS (unauthenticated): is this process serving? Dependency-FREE and
instant it must never touch Kuma. Probing Kuma here could block for the
HTTP + socket timeouts and blow the container healthcheck's own timeout,
marking a perfectly healthy bridge unhealthy. Kuma's state lives in /ready.
"""
state, detail = _kuma_state()
body = {"bridge": "up", "kuma": state}
if detail:
body["detail"] = detail
return body
return {"bridge": "up"}
@app.get("/ready")

View File

@ -23,7 +23,7 @@ ohnehin spricht — am PHP-Code musste nichts geändert werden:
| `POST /monitors` | Monitor anlegen (`{friendly_name, url, type}`) |
| `GET /monitors/{id}` | Status lesen (`up` / `down` / `pending` / `maintenance`) |
| `DELETE /monitors/{id}` | Monitor entfernen (bei Kündigung) |
| `GET /health` | Liveness: 200 solange die Bridge läuft (Kuma-Status im Body) |
| `GET /health` | Liveness: 200 solange die Bridge läuft (prüft Kuma NICHT) |
| `GET /ready` | Readiness: **503**, wenn Kuma nicht erreichbar ist |
Alle Monitor-Routen erfordern `Authorization: Bearer $BRIDGE_TOKEN`.
@ -55,7 +55,7 @@ Alle Monitor-Routen erfordern `Authorization: Bearer $BRIDGE_TOKEN`.
```bash
docker compose exec app curl -s http://kuma-bridge:8080/health
# {"bridge":"up","kuma":"up"}
# {"bridge":"up"}
# Readiness (eignet sich als Check in Kuma selbst — 503 wenn Kuma fehlt):
docker compose exec app curl -s -o /dev/null -w '%{http_code}\n' http://kuma-bridge:8080/ready
@ -84,7 +84,7 @@ notiert eine stabile Kennung und die **eigenen** Gesundheitsprüfungen
Die Bridge nutzt Kumas **interne** Socket.IO-API (über `uptime-kuma-api`). Diese
kann sich zwischen Kuma-Releases ändern:
- Nach einem Kuma-Upgrade: `GET /health` und einen Testkunden prüfen.
- Nach einem Kuma-Upgrade: `GET /ready` und einen Testkunden prüfen.
- Kuma-Version notieren, gegen die getestet wurde.
- Bei Bruch ist der Ausfall dank obigem Verhalten **nicht** kundenwirksam —
es fehlen nur neue Monitore, sichtbar als `info`-Ereignis.

View File

@ -385,6 +385,18 @@ it('keeps provisioning when monitoring is unreachable (retry, then degrade)', fu
->and($run->events()->where('outcome', 'info')->where('step', 'register_monitoring')->exists())->toBeTrue();
});
it('never lets monitoring retries exhaust the run budget', function () {
// A careless MONITORING_ATTEMPTS must not burn through max_attempts and fail
// the run — the cap degrades on the last permitted attempt instead.
config()->set('provisioning.monitoring.attempts', 999);
$s = fakeServices();
$s['monitoring']->failWith = 'connection refused';
['run' => $run] = reservedRun();
$run->update(['attempt' => $run->max_attempts - 1]); // last attempt in the budget
expect(app(RegisterMonitoring::class)->execute($run->fresh())->type)->toBe('advance');
});
it('fails the run on a monitoring outage when monitoring is declared required', function () {
config()->set('provisioning.monitoring.required', true);
$s = fakeServices();