CluPilotCloud/docs/wireguard-hub.md

# WireGuard management hub

Every Proxmox host CluPilot manages is reachable **only** over this tunnel — the
hosts have no public SSH. The hub is the one endpoint that accepts peers.

## Where it runs

Inside the `queue-provisioning` container, not on the VM host. That container is
the only one that needs it: `AdvanceRunJob`, `PurgeHost` and `RemoveWireguardPeer`
all `onQueue('provisioning')`, and it is the only worker consuming that queue.
`LocalWireguardHub` shells out to `wg` / `wg-quick`, so the interface must exist
in the same namespace as the PHP process.

Compose already grants what that needs: `cap_add: NET_ADMIN`, `/dev/net/tun`,
`net.ipv4.ip_forward=1`, the published `51820/udp`, and a named `wireguard`
volume for `/etc/wireguard`. The container command runs `wg-quick up wg0` before
the worker starts, so the tunnel returns after every restart.

The host kernel module does not need `modprobe`: creating the first
`type wireguard` link auto-loads `wireguard.ko`.

## First-time setup

Only needed when the `wireguard` volume is empty (the config is not in git — it
holds the private key).

```bash
docker compose exec queue-provisioning sh -c 'umask 077; wg genkey > /etc/wireguard/hub.key; wg pubkey < /etc/wireguard/hub.key'
```

Write `/etc/wireguard/wg0.conf` with that private key:

```ini
[Interface]
Address = 10.66.0.1/24
ListenPort = 51820
PrivateKey = <hub.key>
```

Then set in `.env` — `CLUPILOT_WG_ENDPOINT` must be the address a **Proxmox host**
can reach, not a container address:

```
CLUPILOT_WG_ENDPOINT=10.10.90.185:51820
CLUPILOT_WG_HUB_PUBKEY=<the pubkey printed above>
```

`wg-quick save wg0` rewrites `wg0.conf` on every peer change, so keep manual
edits to a minimum — comments in that file are lost.

## Peer lifecycle

`LocalWireguardHub::allocateIp()` hands out the next free address in
`CLUPILOT_WG_SUBNET`, skipping the hub IP, the network and the broadcast address,
and skipping every `hosts.wg_ip` already taken. `addPeer()`/`removePeer()` run
`wg set` and then `wg-quick save wg0` — without the save, peers are lost on the
next restart.

## The console (/admin/vpn)

Create, block and remove accesses, and see who is actually connected. Requires
the `vpn.manage` capability (Owner/Admin) — it also hides the nav entry, because
a VPN access reaches the management network.

The web container has no wg0, so the page never talks to WireGuard. `SyncVpnPeers`
runs on the provisioning queue (every minute from the scheduler, plus a throttled
nudge from the page's five-second poll) and copies handshakes, traffic counters
and source endpoints into `vpn_peers`. The page reads the database.

Two fields carry the state, and keeping them apart is what makes the page honest:

- `enabled` — the operator's intent
- `present` — what the hub actually reports

They disagree only while a change is in flight, which the UI shows as
"wird angewendet". A sync never writes `enabled`.

Peers the host pipeline registers are adopted into the same list and named after
their host, so "who is on the VPN" has no blind spots.

Things that are deliberate, and worth knowing before changing them:

- **Deleting is a soft delete.** The row survives as a tombstone until the hub
  confirms the peer is gone. A hard delete would let the next sync see the
  still-present peer and adopt it back as a live access — silently restoring
  what was just revoked. The tombstone also keeps the key and tunnel address
  reserved meanwhile.
- **Every hub mutation takes `Cache::lock('wireguard:hub')`** — ApplyVpnPeer,
  RemoveWireguardPeer and ConfigureWireguard. Without it a sync could read the
  interface around a mutation and then write what it saw.
- **Address allocation takes `Cache::lock('wireguard:allocate')`.** Addresses
  come from one subnet but live in two tables (`hosts.wg_ip`,
  `vpn_peers.allowed_ip`), so no unique index can catch the other side's insert.
- **ApplyVpnPeer reads the current row**, not the state captured at dispatch: a
  retried job must not undo a newer decision. A missing row can only mean the
  access is gone, so a stale payload may remove but never enable.
- **A host's key cannot be re-used** for an operator access: `wg set` would
  rewrite that host's allowed-ip and cut its management tunnel.
- Generated keypairs come from libsodium in PHP, not `wg genkey` — this
  container has no interface, and it keeps generation testable. The private key
  is shown once and never stored.

## Verifying it actually works

Mocked tests cannot prove a tunnel. Against the running stack:

1. Start a throwaway peer that stands in for a Proxmox host:
   `docker run -d --name wgpeer --cap-add NET_ADMIN --device /dev/net/tun clupilot-app:dev sleep 900`
2. Generate its keypair, register it through the app (not by hand) —
   `app(WireguardHub::class)->addPeer('<peer pubkey>', '10.66.0.2')`.
3. Bring up the peer with `Endpoint = 10.10.90.185:51820`,
   `AllowedIPs = 10.66.0.0/24`, `PersistentKeepalive = 25`.
4. `wg show wg0` on both sides must report a recent handshake.
5. Prove the direction the pipeline needs — hub → peer TCP, which is what the SSH
   steps use: listen on `10.66.0.2:2222` in the peer and `fsockopen` to it from
   `queue-provisioning`. A handshake alone does not prove routed TCP.
6. Clean up with `removePeer()` and delete the container.

Verified end-to-end on 2026-07-25: handshake established, hub→peer TCP banner
received, peer persisted to `wg0.conf` by `wg-quick save` and removed again.

## Before production

- The endpoint becomes the production VM's address; regenerate the keypair there
  (the dev private key must never travel).
- `51820/udp` is the only port that has to be open inbound — the console and the
  Proxmox hosts stay off the public hostnames (see `config/admin_access.php`).
- Back up `/etc/wireguard/` (the `wireguard` volume): losing the hub private key
  means re-peering every host.