fix(monitoring): cap monitoring retries at the run budget; dependency-free liveness probe
- MONITORING_ATTEMPTS is capped by the run's max_attempts, so a large value can no longer burn the retry budget and fail the very provisioning the degradation exists to protect (covered by a new test at the budget boundary) - /health is now dependency-free (200 in ~2ms with Kuma absent, verified: the container healthcheck stays healthy); Kuma state is exclusively in /ready Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>feat/portal-design
parent
d1db23398d
commit
46d821a047
|
|
@ -43,7 +43,11 @@ class RegisterMonitoring extends CustomerStep
|
|||
|
||||
// `attempt` is 0-based (this call IS attempt number attempt+1), so
|
||||
// compare one-based: MONITORING_ATTEMPTS=2 really means two tries.
|
||||
$allowed = max(1, (int) config('provisioning.monitoring.attempts', 2));
|
||||
// Cap at the run's own retry budget — otherwise a large ATTEMPTS
|
||||
// value would burn through max_attempts and fail the run, which is
|
||||
// exactly what this degradation exists to prevent.
|
||||
$configured = max(1, (int) config('provisioning.monitoring.attempts', 2));
|
||||
$allowed = max(1, min($configured, (int) $run->max_attempts));
|
||||
if ($run->attempt + 1 < $allowed) {
|
||||
return StepResult::retry(30, 'monitoring unavailable: '.$e->getMessage());
|
||||
}
|
||||
|
|
|
|||
|
|
@ -172,16 +172,12 @@ def _kuma_state() -> tuple[str, str | None]:
|
|||
@app.get("/health")
|
||||
def health() -> dict:
|
||||
"""
|
||||
LIVENESS (unauthenticated): is this process serving? Always 200 while it is.
|
||||
Deliberately NOT tied to Kuma: the bridge being restarted because a
|
||||
*dependency* is down would be wrong. Kuma's state is reported in the body,
|
||||
and /ready is the endpoint that fails when Kuma is unreachable.
|
||||
LIVENESS (unauthenticated): is this process serving? Dependency-FREE and
|
||||
instant — it must never touch Kuma. Probing Kuma here could block for the
|
||||
HTTP + socket timeouts and blow the container healthcheck's own timeout,
|
||||
marking a perfectly healthy bridge unhealthy. Kuma's state lives in /ready.
|
||||
"""
|
||||
state, detail = _kuma_state()
|
||||
body = {"bridge": "up", "kuma": state}
|
||||
if detail:
|
||||
body["detail"] = detail
|
||||
return body
|
||||
return {"bridge": "up"}
|
||||
|
||||
|
||||
@app.get("/ready")
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ ohnehin spricht — am PHP-Code musste nichts geändert werden:
|
|||
| `POST /monitors` | Monitor anlegen (`{friendly_name, url, type}`) |
|
||||
| `GET /monitors/{id}` | Status lesen (`up` / `down` / `pending` / `maintenance`) |
|
||||
| `DELETE /monitors/{id}` | Monitor entfernen (bei Kündigung) |
|
||||
| `GET /health` | Liveness: 200 solange die Bridge läuft (Kuma-Status im Body) |
|
||||
| `GET /health` | Liveness: 200 solange die Bridge läuft (prüft Kuma NICHT) |
|
||||
| `GET /ready` | Readiness: **503**, wenn Kuma nicht erreichbar ist |
|
||||
|
||||
Alle Monitor-Routen erfordern `Authorization: Bearer $BRIDGE_TOKEN`.
|
||||
|
|
@ -55,7 +55,7 @@ Alle Monitor-Routen erfordern `Authorization: Bearer $BRIDGE_TOKEN`.
|
|||
|
||||
```bash
|
||||
docker compose exec app curl -s http://kuma-bridge:8080/health
|
||||
# {"bridge":"up","kuma":"up"}
|
||||
# {"bridge":"up"}
|
||||
|
||||
# Readiness (eignet sich als Check in Kuma selbst — 503 wenn Kuma fehlt):
|
||||
docker compose exec app curl -s -o /dev/null -w '%{http_code}\n' http://kuma-bridge:8080/ready
|
||||
|
|
@ -84,7 +84,7 @@ notiert eine stabile Kennung und die **eigenen** Gesundheitsprüfungen
|
|||
Die Bridge nutzt Kumas **interne** Socket.IO-API (über `uptime-kuma-api`). Diese
|
||||
kann sich zwischen Kuma-Releases ändern:
|
||||
|
||||
- Nach einem Kuma-Upgrade: `GET /health` und einen Testkunden prüfen.
|
||||
- Nach einem Kuma-Upgrade: `GET /ready` und einen Testkunden prüfen.
|
||||
- Kuma-Version notieren, gegen die getestet wurde.
|
||||
- Bei Bruch ist der Ausfall dank obigem Verhalten **nicht** kundenwirksam —
|
||||
es fehlen nur neue Monitore, sichtbar als `info`-Ereignis.
|
||||
|
|
|
|||
|
|
@ -385,6 +385,18 @@ it('keeps provisioning when monitoring is unreachable (retry, then degrade)', fu
|
|||
->and($run->events()->where('outcome', 'info')->where('step', 'register_monitoring')->exists())->toBeTrue();
|
||||
});
|
||||
|
||||
it('never lets monitoring retries exhaust the run budget', function () {
|
||||
// A careless MONITORING_ATTEMPTS must not burn through max_attempts and fail
|
||||
// the run — the cap degrades on the last permitted attempt instead.
|
||||
config()->set('provisioning.monitoring.attempts', 999);
|
||||
$s = fakeServices();
|
||||
$s['monitoring']->failWith = 'connection refused';
|
||||
['run' => $run] = reservedRun();
|
||||
|
||||
$run->update(['attempt' => $run->max_attempts - 1]); // last attempt in the budget
|
||||
expect(app(RegisterMonitoring::class)->execute($run->fresh())->type)->toBe('advance');
|
||||
});
|
||||
|
||||
it('fails the run on a monitoring outage when monitoring is declared required', function () {
|
||||
config()->set('provisioning.monitoring.required', true);
|
||||
$s = fakeServices();
|
||||
|
|
|
|||
Loading…
Reference in New Issue