CluPilotCloud/app/Provisioning/Steps/Customer/ShutDownVirtualMachine.php

<?php

namespace App\Provisioning\Steps\Customer;

use App\Models\ProvisioningRun;
use App\Provisioning\StepResult;
use App\Services\Proxmox\ProxmoxClient;

/**
 * Ask the guest to shut itself down, and wait for it to do so.
 *
 * The first half of a restart, and the half that carries all the risk. Two
 * decisions are written into it.
 *
 * **It is a shutdown and a start, never a reboot.** An ACPI reset leaves the
 * same qemu process running with the same command line, so the cores and memory
 * a plan change wrote into the VM's definition would still not be there
 * afterwards — the machine would go round and come back exactly as small as it
 * was, and `restart_required_since` would be cleared over nothing. Only a cold
 * boot reads the definition again. That is why this step exists at all instead
 * of a single `status/reboot` call.
 *
 * **It never escalates to a power cut.** Proxmox will happily destroy the qemu
 * process once the timeout expires (`forceStop`), and ProxmoxClient deliberately
 * does not offer that — the guest is a Nextcloud, which is MariaDB plus a file
 * tree, and killing one mid-write to make a CPU change take effect can turn a
 * cosmetic delay into a restore from last night's backup. So the grace period
 * below is a bound on WAITING, not a fuse: when it runs out the run fails, the
 * machine is left running exactly as it was, and an operator gets a failed run
 * to look at. A guest that has ignored ACPI for ten minutes is a guest that is
 * doing something, and a person should decide what happens to it.
 */
class ShutDownVirtualMachine extends CustomerStep
{
    /**
     * How long the guest is given to go away by itself.
     *
     * Ten minutes. A Nextcloud stopping cleanly is docker compose stopping a
     * handful of containers and MariaDB flushing — seconds to a minute or two on
     * anything healthy. The rest of the allowance is for the unhealthy case that
     * is still going to succeed: a long-running cron job inside the guest, a
     * large InnoDB buffer pool being written out. Beyond that it is not slow,
     * it is stuck.
     */
    private const GRACE_SECONDS = 600;

    public function __construct(private ProxmoxClient $pve) {}

    public function key(): string
    {
        return 'shut_down_vm';
    }

    public function maxDuration(): int
    {
        // Above GRACE_SECONDS so this step's own deadline fires first and the
        // run fails with a reason somebody can read, rather than being turned
        // into a generic timeout-retry by the runner.
        return 720;
    }

    public function execute(ProvisioningRun $run): StepResult
    {
        $instance = $this->instance($run);

        if ($instance === null) {
            return StepResult::fail('no_instance');
        }

        $node = (string) $run->context('node');
        $vmid = (int) $run->context('vmid');
        $pve = $this->pve->forHost($instance->host);

        // Idempotent, and the ordinary way this step ends: the guest is gone.
        if (($pve->vmStatus($node, $vmid)['status'] ?? '') !== 'running') {
            return StepResult::advance();
        }

        // Asked once, remembered on the run. Re-sending ACPI to a guest that is
        // already half-way through its own shutdown achieves nothing and reads,
        // in the guest's log, like a second person pressing the power button.
        if ($run->context('shutdown_upid') === null) {
            $run->mergeContext([
                'shutdown_upid' => $pve->shutdownVm($node, $vmid, self::GRACE_SECONDS),
            ]);

            return StepResult::poll(10, 'waiting for the guest to shut down');
        }

        // A poll does not reset started_at, so this deadline genuinely
        // accumulates across the whole wait rather than restarting every time.
        if ($run->started_at !== null && $run->started_at->copy()->addSeconds(self::GRACE_SECONDS)->isPast()) {
            $run->events()->create([
                'step' => $this->key(),
                'attempt' => $run->attempt,
                'outcome' => 'info',
                'message' => 'Der Gast hat sich in '.self::GRACE_SECONDS.' Sekunden nicht heruntergefahren. '
                    .'Die Maschine läuft unverändert weiter — sie wird bewusst nicht hart ausgeschaltet, '
                    .'weil eine Nextcloud eine laufende Datenbank ist.',
            ]);

            return StepResult::fail('shutdown_timeout');
        }

        return StepResult::poll(10, 'waiting for the guest to shut down');
    }
}