86 lines
3.5 KiB
Markdown
86 lines
3.5 KiB
Markdown
# WireGuard management hub
|
|
|
|
Every Proxmox host CluPilot manages is reachable **only** over this tunnel — the
|
|
hosts have no public SSH. The hub is the one endpoint that accepts peers.
|
|
|
|
## Where it runs
|
|
|
|
Inside the `queue-provisioning` container, not on the VM host. That container is
|
|
the only one that needs it: `AdvanceRunJob`, `PurgeHost` and `RemoveWireguardPeer`
|
|
all `onQueue('provisioning')`, and it is the only worker consuming that queue.
|
|
`LocalWireguardHub` shells out to `wg` / `wg-quick`, so the interface must exist
|
|
in the same namespace as the PHP process.
|
|
|
|
Compose already grants what that needs: `cap_add: NET_ADMIN`, `/dev/net/tun`,
|
|
`net.ipv4.ip_forward=1`, the published `51820/udp`, and a named `wireguard`
|
|
volume for `/etc/wireguard`. The container command runs `wg-quick up wg0` before
|
|
the worker starts, so the tunnel returns after every restart.
|
|
|
|
The host kernel module does not need `modprobe`: creating the first
|
|
`type wireguard` link auto-loads `wireguard.ko`.
|
|
|
|
## First-time setup
|
|
|
|
Only needed when the `wireguard` volume is empty (the config is not in git — it
|
|
holds the private key).
|
|
|
|
```bash
|
|
docker compose exec queue-provisioning sh -c 'umask 077; wg genkey > /etc/wireguard/hub.key; wg pubkey < /etc/wireguard/hub.key'
|
|
```
|
|
|
|
Write `/etc/wireguard/wg0.conf` with that private key:
|
|
|
|
```ini
|
|
[Interface]
|
|
Address = 10.66.0.1/24
|
|
ListenPort = 51820
|
|
PrivateKey = <hub.key>
|
|
```
|
|
|
|
Then set in `.env` — `CLUPILOT_WG_ENDPOINT` must be the address a **Proxmox host**
|
|
can reach, not a container address:
|
|
|
|
```
|
|
CLUPILOT_WG_ENDPOINT=10.10.90.185:51820
|
|
CLUPILOT_WG_HUB_PUBKEY=<the pubkey printed above>
|
|
```
|
|
|
|
`wg-quick save wg0` rewrites `wg0.conf` on every peer change, so keep manual
|
|
edits to a minimum — comments in that file are lost.
|
|
|
|
## Peer lifecycle
|
|
|
|
`LocalWireguardHub::allocateIp()` hands out the next free address in
|
|
`CLUPILOT_WG_SUBNET`, skipping the hub IP, the network and the broadcast address,
|
|
and skipping every `hosts.wg_ip` already taken. `addPeer()`/`removePeer()` run
|
|
`wg set` and then `wg-quick save wg0` — without the save, peers are lost on the
|
|
next restart.
|
|
|
|
## Verifying it actually works
|
|
|
|
Mocked tests cannot prove a tunnel. Against the running stack:
|
|
|
|
1. Start a throwaway peer that stands in for a Proxmox host:
|
|
`docker run -d --name wgpeer --cap-add NET_ADMIN --device /dev/net/tun clupilot-app:dev sleep 900`
|
|
2. Generate its keypair, register it through the app (not by hand) —
|
|
`app(WireguardHub::class)->addPeer('<peer pubkey>', '10.66.0.2')`.
|
|
3. Bring up the peer with `Endpoint = 10.10.90.185:51820`,
|
|
`AllowedIPs = 10.66.0.0/24`, `PersistentKeepalive = 25`.
|
|
4. `wg show wg0` on both sides must report a recent handshake.
|
|
5. Prove the direction the pipeline needs — hub → peer TCP, which is what the SSH
|
|
steps use: listen on `10.66.0.2:2222` in the peer and `fsockopen` to it from
|
|
`queue-provisioning`. A handshake alone does not prove routed TCP.
|
|
6. Clean up with `removePeer()` and delete the container.
|
|
|
|
Verified end-to-end on 2026-07-25: handshake established, hub→peer TCP banner
|
|
received, peer persisted to `wg0.conf` by `wg-quick save` and removed again.
|
|
|
|
## Before production
|
|
|
|
- The endpoint becomes the production VM's address; regenerate the keypair there
|
|
(the dev private key must never travel).
|
|
- `51820/udp` is the only port that has to be open inbound — the console and the
|
|
Proxmox hosts stay off the public hostnames (see `config/admin_access.php`).
|
|
- Back up `/etc/wireguard/` (the `wireguard` volume): losing the hub private key
|
|
means re-peering every host.
|