CluPilotCloud/routes/console.php

<?php

use App\Console\TickProvisioning;
use App\Models\StripePendingEvent;
use App\Provisioning\Jobs\CollectInstanceTraffic;
use App\Provisioning\Jobs\PingHosts;
use App\Provisioning\Jobs\RecordProvisioningHeartbeat;
use App\Provisioning\Jobs\SyncMonitoringStatus;
use App\Provisioning\Jobs\SyncVpnPeers;
use App\Support\Settings;
use Illuminate\Foundation\Inspiring;
use Illuminate\Support\Facades\Artisan;
use Illuminate\Support\Facades\Schedule;

Artisan::command('inspire', function () {
    $this->comment(Inspiring::quote());
})->purpose('Display an inspiring quote');

// Advance every due provisioning run once a minute (runs in the scheduler service).
Schedule::call(fn () => app(TickProvisioning::class)())
    ->everyMinute()
    ->name('provisioning-tick')
    ->withoutOverlapping();

// Two signs of life for the readiness page (App\Support\Readiness\
// OperationChecks). Written through Settings, not the cache: a heartbeat that
// vanishes on a Redis restart would report an outage that never happened.
//
// Two of them, because the scheduler and the provisioning queue worker fail
// independently. The dangerous case is the second one: the scheduler runs and
// enqueues jobs (this one included) that nobody ever picks up. A heartbeat
// only the scheduler writes would call that state healthy.
Schedule::call(fn () => Settings::set('heartbeat.scheduler', now()->toIso8601String()))
    ->everyMinute()
    ->name('heartbeat-scheduler');

Schedule::job(new RecordProvisioningHeartbeat)
    ->everyMinute()
    ->name('heartbeat-provisioning');

// Refresh the VPN peer state (handshakes, traffic) so the console does not go
// stale while nobody has it open. The job itself runs on the provisioning
// queue — that worker owns wg0.
Schedule::job(new SyncVpnPeers)
    ->everyMinute()
    ->name('vpn-sync');

// Sample network counters and enforce the monthly allowance. Runs on the
// provisioning queue, which is where the Proxmox credentials are usable.
// Turns the monitoring column from a claim into a measurement. Without it the
// console and the public status page report every instance healthy forever.
Schedule::job(new SyncMonitoringStatus)
    ->everyFiveMinutes()
    ->withoutOverlapping();

Schedule::job(new CollectInstanceTraffic)
    ->everyFifteenMinutes()
    ->name('traffic-collect');

// Discard Stripe events that were held for a contract which never appeared.
// Most of them never will: Stripe also delivers events for subscriptions
// belonging to other environments pointed at this endpoint, and for objects
// made by hand in the dashboard. A week is far longer than the moment a
// checkout takes to become a contract, and short enough that the table stays
// a holding area rather than a second copy of Stripe's event log.
Schedule::call(fn () => StripePendingEvent::query()
    ->where('created_at', '<', now()->subWeek())
    ->delete())
    ->daily()
    ->name('stripe-pending-prune');

// Copies of invoices that never reached the archive — a NAS that was rebooting
// when the invoice was issued, a mount that had gone away. Copy-on-issue is
// right and is not enough: nobody opens an archive to check whether last
// Tuesday is in it, so something has to go and look.
//
// Hourly rather than by the minute: the failure it repairs lasts as long as an
// outage lasts, and a retry storm against a mount that is down helps nobody.
// Registrations nobody confirmed. Daily is often enough for a five-day window,
// and it holds the address hostage until it runs: the unique index means the
// person cannot register again with the address they meant.
Schedule::command('clupilot:prune-unverified')
    ->dailyAt('03:20')
    ->withoutOverlapping();

// Confirmed accounts that never had a package. A year, with a fortnight's notice
// by mail — so this has to run daily even though the window is long: the notice
// is what permits the deletion, and an account whose warning was never sent is
// never removed. See App\Console\Commands\PruneDormantAccounts.
Schedule::command('clupilot:prune-dormant')
    ->dailyAt('03:35')
    ->withoutOverlapping();

// The support mailbox. Polling, because IMAP is a mailbox you look in — a mail
// server has no way to tell us something arrived — so the interval IS the
// answer to "how long until I see it". Two minutes is short enough that an
// operator watching the inbox does not reload it by hand, and long enough that
// a mail server is not being asked once a second for nothing.
//
// withoutOverlapping: a slow or unreachable mailbox must not stack runs on top
// of each other until a worker is doing nothing but timing out.
Schedule::command('clupilot:fetch-mail')
    ->everyTwoMinutes()
    ->withoutOverlapping();

Schedule::command('clupilot:archive-invoices')
    ->hourly()
    ->withoutOverlapping();

// Plan changes that landed here and never reached Stripe. The change is applied
// to the contract and to the machine before Stripe is told, and it is not rolled
// back when Stripe is away — so what is left is a customer being billed for a
// package they are no longer on, and nobody would ever go and look.
//
// Hourly, like the archive sweep and for the same reason: the failure it repairs
// lasts as long as an outage lasts, and hammering an API that is down helps
// nobody.
Schedule::command('clupilot:sync-stripe-subscriptions')
    ->hourly()
    ->withoutOverlapping();

// Empty a handover directory of folders past their keep-by. Only where a
// destination asks for it, and only folders whose name is a date this
// application wrote — see the command for why it leaves everything else alone.
Schedule::command('clupilot:prune-exports')
    ->dailyAt('03:20')
    ->withoutOverlapping();

// One sample of every public status component, into today's row.
//
// The ninety-day bar on the status page is the one figure a reader checks
// against their own memory, so it has to come from something recorded at the
// time — there is no way to reconstruct it later. Beside the monitoring sync it
// reads from, at the same cadence.
Schedule::command('clupilot:sample-status')
    ->everyFiveMinutes()
    ->withoutOverlapping();

// Re-read the DNS proof for every custom domain.
//
// Not a one-off: a token checked once can be taken straight back out, and a
// domain that later lapses and is registered by somebody else keeps resolving
// to the same instance. Nightly, and at an hour where withdrawing a domain
// inconveniences nobody — a withdrawal takes a working Nextcloud off its own
// address, which is not a thing to do at eleven in the morning.
Schedule::command('clupilot:verify-domains')
    ->dailyAt('03:40')
    ->withoutOverlapping();

// Take an available release inside the configured window, if the owner has
// switched that on. Every five minutes rather than once at the start of the
// window: a scheduler that was down for those sixty seconds would otherwise
// skip the whole night.
Schedule::command('clupilot:auto-update')
    ->everyFiveMinutes()
    ->withoutOverlapping();

// Ask every host whether it is still there.
//
// hosts.last_seen_at drove the console's health dot and was written exactly
// once, at onboarding — so every host read "offline" half an hour later,
// permanently. Nothing was measuring host reachability at all, which is also
// why it could not appear on the status page.
Schedule::job(new PingHosts)
    ->everyMinute()
    ->name('host-ping');

// Carry out the downgrades whose term has run out.
//
// An upgrade lands the moment it is paid for; a downgrade waits, because
// somebody on a yearly contract bought a year. Nothing was waiting with it — the
// order sat in the cart forever and the customer stayed on the bigger package.
//
// Every quarter of an hour rather than nightly: a term ends at the second it
// ends, and a customer who has asked to pay less should not spend another
// afternoon on the old price. Nothing is taken away that they did not ask to
// give up, so there is no bad hour for it.
Schedule::command('clupilot:apply-due-plan-changes')
    ->everyFifteenMinutes()
    ->withoutOverlapping();

// End the modules whose cancellation date has arrived.
//
// The same rule the downgrade above follows: a module cancelled on the tenth is
// kept until the term it was paid for runs out. Stripe stopped billing it at the
// moment of cancellation — its item is only ever about the NEXT cycle — so what
// waits for this sweep is the module itself: the entitlement, the space on the
// disk, and the entry in the register saying it ended.
//
// Beside the downgrades and at the same cadence, because it is the same
// appointment being kept.
Schedule::command('clupilot:end-cancelled-addons')
    ->everyFifteenMinutes()
    ->withoutOverlapping();

// Ask the VIES register again about the VAT numbers reverse charge rests on.
//
// A number is verified once, at the moment somebody types it in, and never again
// — and a registration that is wound up or withdrawn keeps earning rate 0 on
// every invoice issued after it lapsed. The unpaid VAT on those is the SELLER's
// liability, not the customer's, so the cadence here is the width of the window
// in which that can happen without anyone noticing.
//
// Monthly, on the first, because that is the rhythm a VAT return is filed on: a
// registration that lapsed in March is caught before the March return is
// prepared. Not nightly — it is somebody else's public service, it answers about
// a tax position rather than about our own machines, and a number does not lapse
// twice in a month. The command takes the longest-unchecked first, so a customer
// base larger than one run's --limit rotates through instead of starving.
Schedule::command('clupilot:verify-vat-ids')
    ->monthlyOn(1, '04:10')
    ->withoutOverlapping();

// Keep the appointment a cancellation made.
//
// ConfirmCancelPackage writes a date and nothing ever went back to it, which is
// why TraefikWriter::remove() had no caller in the whole application: a
// cancelled instance kept its router, its certificate and its route to a guest
// address the host is free to reassign. Only instances whose paid term has
// actually run out are touched — a cancellation that is merely SCHEDULED is a
// paying customer, and this command is written around that distinction.
//
// Hourly: a term ends at a moment, and neither giving away a day of service nor
// hammering a DNS provider by the minute is a decision worth making by accident.
Schedule::command('clupilot:end-due-services')
    ->hourly()
    ->withoutOverlapping();

// Zertifikate: welche Namen bedient diese Installation, und wie lange gelten
// sie noch.
//
// Nach Zeitplan und nicht nur auf Knopfdruck. Die Frage, auf die es ankommt,
// lautet nicht „ist es gerade gültig" — sondern „läuft die Erneuerung". Ein
// Zertifikat, das in vierzig Tagen abläuft, ist in Ordnung; dasselbe in
// zwanzig Tagen heißt, dass seit einer Woche etwas nicht funktioniert. Das
// sieht nur, wer misst, wenn niemand hinsieht.
//
// Früh, aber nicht zur vollen Stunde: Let's Encrypt bittet ausdrücklich darum,
// nicht auf den Punkt zu prüfen.
Schedule::command('clupilot:check-certificates')
    ->dailyAt('04:17')
    ->withoutOverlapping()
    ->name('check-certificates');