Die alte Hetzner-DNS-API ist abgeschaltet (301 auf die Weboberflaeche).
Jede Bereitstellung starb in ConfigureDnsAndTls, nachdem der Kunde bezahlt
hatte. HttpHetznerDnsClient und DnsTokenCheck sprechen jetzt die Cloud-API:
Bearer statt Auth-API-Token, RRSets ueber {name}/{typ} statt Record-IDs, der
Zonen-Lookup entfaellt. Gegen das Live-Konto gemessen, nicht geraten.
Drei Fallen, die am Konto gemessen wurden: TXT-Werte muessen in
Anfuehrungszeichen (sonst 422, was als read_only gemeldet worden waere), ein
Name mit Zonensuffix wird STILL angenommen (201), und der Fake gab andere IDs
aus als der echte Client -- zwoelf Tests waren gruen ueber einem Abbau, der im
Betrieb geworfen haette.
Bereitschaftspunkte, die nur eine Shell beheben konnte, sind jetzt bedienbar:
SSH-Schluesselpaar erzeugen (Ed25519 ueber phpseclib, privater Teil direkt in
den Tresor), Stripe-Katalog abgleichen (Warteschlange, Trockenlauf, Ausgabe
wortgleich), Mailzustellung als Schalter statt MAIL_MAILER, Neustart der
Arbeiterprozesse. Stripe hat jetzt alle drei Werte in der Konsole: Secret Key,
Signatur-Secret (Tresor, je Modus getrennt) und Publishable Key (Klartext).
Codex-Review (R15), zwei P1 behoben: der Signaturschluessel faellt nicht mehr
vom Live- in den Testplatz, und eine Record-ID aus der alten API macht einen
Host nicht mehr unloeschbar -- was adressierbar ist, wandelt eine Migration um,
der Rest wird beim Loeschen laut uebergangen statt geworfen.
2109 Tests gruen.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Written so the next session can start without rediscovering anything: where the
code stands, the three open pieces in the order they matter, and the two release
traps that cost two version numbers today.
The second piece is the owner's own point, and it is the right one: a readiness
page that flags things only a shell can fix has moved the work rather than done
it. Three places do that today — generating the host keypair, syncing the Stripe
catalogue, and nothing else. The rest already links where it should.
The Stripe sync gets a note about not running it inside a Livewire call: it
talks to Stripe and creates products, so it belongs in a queue with a visible
result, and the dry run belongs in the interface beside it rather than only in a
terminal.
Last section names the pattern the whole session kept turning up: a display more
confident than what it rests on. A green badge over a red measurement, a claim
about somebody's Hetzner account derived from an HTML page, a comment promising
a file the installer never created. Worth reading the code with that in mind.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The root cause of the zone_not_found hunt: the old DNS API is gone. It answers
301 to the web console, and the owner's token already returns 200 against
api.hetzner.cloud — so this is code work, not a credential problem.
The note holds what the next session needs and would otherwise have to rediscover:
the endpoints, taken from cloud.spec.json rather than guessed, the fact that
records are now RRSets addressed by name and type, that a zone can be addressed
by NAME directly, and two traps in the name field — it must not end with the zone
name, which is exactly what the current client sends, and PUT on an rrset sets
labels rather than records.
Recorded rather than started, deliberately. HttpHetznerDnsClient writes the A
record for every customer instance; rewriting it is not something to begin
without room left to test it properly, and a half-migrated DNS client is worse
than a documented broken one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DNS check now requires a real zones array before it says anything about the
account. A 200 carrying an HTML page — a portal, a filter, a proxy — was being
read as "no zones at all", which is what the live server was seeing while the
token had read and write and the zone existed with fifteen records.
An intercepted response is now reported as one, with its status and the first
characters of what actually came back.
2053 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the live server: a token with read AND write, the zone
clupilot.cloud present with fifteen records — and the console insisting the
account held no zones at all. Both screenshots contradict the message, so the
message was wrong, and the previous fix did not go far enough.
successful() is not enough. Something in the middle — a portal, a filter, a
proxy — answers with 200 and an HTML page. That body has no `zones` key, `??
[]` turned it into no zones, and the display concluded the Hetzner account was
empty. A 200 is not a promise about who answered.
So the body has to answer the question, not merely arrive: `{"zones": [...]}`
with an actual array, or it is reported as something else having spoken, with
the status and the first 120 characters of what came back. That last part is
what turns it from a verdict into a diagnosis — an operator who sees "Blocked by
policy" knows in one line what nothing else here could have told them.
A genuine `{"zones": []}` still means what it always meant.
2053 tests pass, assets build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DNS check no longer reports an unclassified HTTP failure as "this account
holds no zones". Anything other than a successful zone list is now named as
what it is, with its status code, and says explicitly that it tells you nothing
about the zones.
2050 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The owner asked whether an empty token was being sent. It was not — an empty one
returns `missing` — but the question was worth following, and it found a fault
in the message I had just added.
The check classified 401 and 403 as `rejected` and then read the body. Every
OTHER unsuccessful response — 404, 429, 500, a cache's status page — has no
`zones` key, `?? []` turned that into no zones, and the console then stated "this
account holds no zones at all. The token probably belongs to a different Hetzner
project." A claim about somebody's account, derived from an error nobody looked
at, delivered with more confidence than the working case gets.
`successful()` is asked first now, and an unexpected status is reported as what
it is, with the number beside it: it says nothing about the zones, and it says
so. That is the distinction this check already draws between `unreachable` and
`read_only` — both are failures, only one of them tells you anything about the
token.
A genuinely empty list still means what it meant: 200 with zero zones is a token
for a project without zones.
The test uses a dataset rather than a loop. Http::fake() ADDS stubs instead of
replacing them, so a loop would have had the first status answer all four
iterations and the test would have proved one case three times over.
2050 tests pass, assets build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The readiness page no longer shows a green badge over a red measurement. A
check that was actually run decides the badge; the passive "something is
configured" answer only fills in where nothing was measured.
zone_not_found now names the zone it looked for and the zones the token can
actually see, so it stops reading like "the token is wrong" — a wrong token
never gets that far.
Also: two traffic tests failed on the 29th, 30th and 31st of long months,
because they built "last month" with subMonth() and Carbon resolves that
calendrically. The production code was never affected.
2045 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three findings from the live readiness page, and the worst of them is the one
that looks like nothing.
Green "Erfüllt" sat directly above red "nicht in Ordnung", in the same row,
twice — for the DNS token and for the VM template. The badge came from
`satisfied` alone, the passive check that only establishes something IS
configured, and the measurement was rendered beside it without being allowed to
overrule it. Somebody scanning that list reads the badge, not the small print,
and walks away with "all green" while a measurement said it does not work. R19
names this exact shape — a call that reads as an assurance and is not one — as
worse than no check at all. The measurement wins now, for the badge, the icon
and the reason line.
zone_not_found was a dead end. It reads like "the token is wrong", so the
operator replaces the token — but a wrong token never gets that far: it comes
back as `rejected` from the 401 above. The token had just successfully listed
the zones. What is missing is the ZONE. The check now returns the zones it did
see, and the page puts them next to the one it wanted: looked for
clupilot.cloud, this account holds clupilot.com. The question answers itself.
An empty list says something else again, and gets its own sentence: the token
belongs to a different Hetzner project.
And two traffic tests were failing on main, unrelated to any of this, which is
why they were checked against a clean checkout before being touched. They build
"last month" as now()->subMonth()->format('Y-m'), and Carbon resolves that
calendrically: on 31 July it lands on 1 July, so the row meant to be last
period lands in the current one. Red on the 29th, 30th and 31st of every long
month, and today is the 31st. The production code does not have the trap —
currentPeriod() is now()->format('Y-m') with no arithmetic, and the two places
that do compute months already guard it — so this is the tests, and only the
tests.
2045 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
IP addresses no longer appear in the certificate overview. 127.0.0.1 comes from
ADMIN_HOSTS, where a bare IP is deliberately allowed as the way back into the
console — but no authority issues certificates for one, so the row was
permanently red and unfixable.
The row already stored is removed on the next sync. A vanished config entry that
holds a live certificate becomes a manual entry instead of disappearing.
2043 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
127.0.0.1 was in the list on the live server, marked as console, showing "no
valid certificate: Connection refused" in red. It came from ADMIN_HOSTS, where a
bare IP is deliberately allowed — it is the way back into the console when a
name does not resolve. My filter only asked for a dot, and 127.0.0.1 has three.
Let's Encrypt does not issue for IP addresses, so that row was permanently red
and nobody could ever fix it. A red line that cannot be acted on is worse than
no line: it teaches the reader to skip the colour, which is the one thing the
overview needs them not to do.
Two checks now, not one: the address test, and a last label that is not numeric.
A TLD is never a number, and that also catches the forms FILTER_VALIDATE_IP lets
through.
The row already in the database goes away on the next sync, because otherwise my
mistake would sit on every installation that has already updated. Vanished
config rows are handled by what they carry: one that never had a certificate was
a mistake and is deleted; one that HAS a live certificate becomes a manual entry
instead, so nothing with a running expiry disappears without the operator
deciding. That is a change of mind from the previous commit, which said config
rows are never removed — this case showed the cost of that rule where the row
should never have existed.
2043 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hostnames page fills itself from the installation instead of starting
blank. Every name this installation answers on — from SITE_HOSTS, APP_HOST,
FILES_HOST and ADMIN_HOSTS — appears with its certificate and the days it has
left, without anybody typing it in.
Four counts at the top: total, valid, expiring soon, without a certificate. A
daily run at 04:17 keeps them current, because the question is not whether a
certificate is valid today but whether renewal is running.
2040 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The page was empty, and that was a design fault rather than a missing button. I
built a register that starts blank — while the installation already serves half
a dozen names whose certificates were exactly what the operator wanted to see.
An overview you have to populate first does not answer "what do I have".
So the names are derived now, from the same configuration routes/web.php builds
its domain bindings from: SITE_HOSTS, APP_HOST, FILES_HOST, ADMIN_HOSTS. If a
name is in the environment, the application answers on it, and then it belongs
in this list without anybody typing it a second time. Opening the page syncs
them; the list is never empty again.
Syncing and measuring are deliberately separate. The sync costs nothing and runs
on page load. The measurement goes out over the network and runs on the button
or on a schedule — doing it on page load would mean waiting through half a dozen
TLS handshakes, and one of them is always the name that currently does not
resolve.
Which is the other half of what was missing: there was no overview because
nothing measured unless somebody pressed a button. A daily run at 04:17 fills
it, because the question that matters is not "is it valid right now" but "is
renewal running" — a certificate expiring in forty days is fine, the same one at
twenty means something has been broken for a week. Only a measurement taken
while nobody is looking can tell those apart.
The page now opens with four counts — total, valid, expiring soon, without a
certificate — and says when it last measured. A name that comes from the
environment is marked as such and cannot be removed here: it would come back at
the next sync, and a button that does nothing is worse than no button, because
it gets believed once.
2040 tests pass, assets build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
install-agent.sh now creates /etc/caddy/clupilot-proxy-hosts.conf and adds the
import line itself, validating afterwards and taking the line back out if Caddy
rejects it. The previous release told the operator to add that line by hand
while nothing created the file — and an import of a missing file makes the whole
proxy configuration invalid, which would have taken the portal down with it.
A hostname already written by hand in the Caddyfile is skipped when the managed
file is rendered. Two blocks for one name make Caddy reject everything.
Run once per server:
sudo bash /opt/clupilot/deploy/install-agent.sh
2037 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment in Caddyfile.example said install-agent.sh creates
/etc/caddy/clupilot-proxy-hosts.conf before anything imports it. It did not. I
wrote the claim and never built the thing, and the release notes repeated it as
an instruction.
Following that instruction would have taken the site down. An import of a
missing file makes the WHOLE Caddy configuration invalid — app.clupilot.com,
the portal, everything — and the operator would have been holding a step I told
them to take.
So the installer now does both halves itself: it creates the file, and it adds
the import line, then runs `caddy validate` and takes the line straight back out
if Caddy rejects it. Doing it in a script rather than in a comment is the
point — an instruction gets followed in the wrong order, a script does not.
And a second trap alongside it: a hostname written by hand in the Caddyfile is
now skipped when the generated file is rendered. Two site blocks for one name
make Caddy reject the entire configuration, so an operator who managed an
already-hand-written name in the console as well would have taken the public
site off the air with no visible connection between the two actions.
2037 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hostnames page is now in the navigation under Betrieb — it had a route and
no entry, which meant it existed only for whoever knew the URL.
And files.… no longer redirects to app.… when called without a path. It answers
404, like every other address in this installation that has nothing to offer:
the redirect told anyone who tried the name where the portal lives and that both
share a machine.
2037 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two faults, both mine, both reported from the live server.
The page had a route and no navigation entry. A page reachable only by typing
its URL does not exist as far as the operator is concerned, and "Betrieb →
Hostnamen und Zertifikate" was exactly as findable as I had made it: not at all.
It sits under Betrieb rather than System because what is set there decides
whether an address ANSWERS — that is operations, not configuration.
And calling files.… without a path redirected to app.… The host-bound group only
claimed /bootstrap.tar.gz and /{file}; a bare / matches neither, so the request
fell through to the host-agnostic routes and landed on the portal. Two holes,
because / and a multi-segment path miss the placeholder for different reasons,
and both are closed now.
404 rather than a redirect, and that is the point rather than a detail. The
redirect told anybody who tried the name where the portal lives and that both
sit on the same machine — the one thing every other hostname in routes/web.php
is careful not to say. An address with nothing to offer has nothing to tell
either.
2037 tests pass, assets build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hostnames and certificates from the console: add a name, see whether it has a
certificate and for how long, and apply the list to the reverse proxy without
logging into the server.
Two numbers are skipped and both were my mistake, so they are named here rather
than left as gaps somebody has to work out later. v1.3.68 was cut while the
working directory was still inside the feature worktree, so it landed on the
branch instead of on main; this release carries it. v1.3.69 was pushed pointing
at v1.3.67's commit. Deleted rather than moved, both times: a published tag that
changes what it points at is worse than a number nobody uses.
This release needs one thing done once per server, because the root-owned helper
gained a verb:
sudo bash deploy/install-agent.sh
and one line in /etc/caddy/Caddyfile:
import /etc/caddy/clupilot-proxy-hosts.conf
The installer creates that file before anything imports it — an import of a
missing file makes the whole proxy configuration invalid, and that takes the
public site with it.
2035 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One conflict, in VERSION, and it is worth naming rather than resolving quietly.
v1.3.68 was cut on the FEATURE branch, not on main: the working directory was
still inside .worktrees/host-bootstrap when the merge-and-tag ran, so the merge
was a no-op against itself, the version commit landed on the branch, and the tag
went with it. main stayed on 1.3.67 while a tag claimed otherwise.
This merge brings that commit onto main, so v1.3.68 becomes reachable from the
line it always claimed to be on, and the version continues at 1.3.69 rather than
pretending the number was never used.
The project's own memory records this exact failure — check which branch you are
on before every commit — from the time two releases landed on feat/granted-plans
while live stayed where it was. It was a parallel session then and a stale
working directory now; the rule is the same one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
files.clupilot.com is why this exists. DNS pointed at the right machine, the
env var was set, the release was deployed — and there was still no certificate,
because /etc/caddy/Caddyfile is maintained by hand and nobody thought of it as a
second, separate step. Nothing in the console would have said so. Three settings
looked correct and the address did not answer.
So the page holds two things side by side. The WISH — which names should be
served — and the REALITY: whether the name has a certificate and for how much
longer. The second is measured by opening a TLS connection and reading the
expiry, not by reading configuration, because the configuration is exactly what
looked right while the address was dead. verify_peer stays on: a certificate
that fails validation is not a certificate for this question, and a display that
called it valid would be the fake R19 records.
Applying goes through the existing agent, not a new channel. The console writes
a request, the path unit wakes the agent within a second, and the agent calls one
fixed command line of the root-owned helper.
What that helper is allowed to do is the careful part. It fetches the list
ITSELF rather than being handed one, and the list is HOSTNAMES, never Caddy
blocks — `php artisan clupilot:proxy-hosts` prints `<name> <purpose>` and nothing
else. Each name is matched against a strict pattern before it is used, and the
template around it lives in the helper, which the service account cannot touch.
install-agent.sh already states the principle for the sudoers grant: a grant is
only worth anything if the holder cannot change what it grants. A service account
that could write proxy configuration would have everything the proxy can do —
redirects anywhere, files from any directory.
Purpose is a column rather than a habit. A console name gets the network lock,
a public one does not, and a console name published without it looks exactly
like a working page.
Removing takes the name out of the list and NOT out of the running proxy. Two
decisions in one click, and the second one takes a site off the air.
There is deliberately no "renew" button. Caddy renews on its own at two thirds
of the lifetime; what an operator actually needs is a second attempt after an
issuance has failed, and that is a reload — which is what Apply does. Thirty days
is treated as a problem rather than a warning: at ninety days' lifetime, a
renewal should long since have run, so anything under it is not a tight
certificate but a renewal that is not happening.
The ACME contact falls back to the owner's address, because a contact nobody
reads is the step before expired customer certificates.
CONTRACT and HOST_STEP_NEEDS both move to 2, which is what tells a server
carrying the older helper to run the installer again — caught by the guard test
that compares the two halves.
2035 tests pass, assets build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The reference Caddyfile now carries a block for FILES_HOST. Setting the variable
alone was never enough: the proxy has to terminate TLS for that name and forward
it, and without a site block Caddy never even asks for a certificate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FILES_HOST tells Laravel which hostname to bind the routes to. It does not make
the reverse proxy terminate TLS for that name or forward it anywhere, and the
live server showed exactly that gap: DNS correct, the right machine, port 80
answering — and a TLS handshake that fails because no certificate exists for the
name.
The block carries two things the other public names do not have to think about.
No console allowlist. The archive is fetched by a server in a RESCUE SYSTEM: a
machine with no tunnel and a public address that is in nobody's allow list. The
console's network lock here would be a lock against the only caller that needs
it. The protection lives in the application instead — no valid one-time code,
404 — and the legal documents are public by intent.
And no `http://… { abort }` either, unlike admin and ws below it. Those two hide
that they exist; this one is meant to be known, and killing port 80 takes the
path away from an ACME check on the day Caddy's own challenge handler is not the
first thing to answer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add host rebuilt as a page with one job: a sticky rail of six short steps, the
two provider prerequisites in a panel above the form, and after saving the
command line as the page itself.
Downloads now pass PublicSiteGate. While the site is hidden, a server in a
rescue system would have received the 503 placeholder and piped it into tar — an
unpack error on a machine only reachable through the provider's console.
The number skips 1.3.66. That tag was pushed pointing at v1.3.65's commit,
because the merge behind it had not actually happened, and a server pinning it
would have installed the older tree. It was deleted rather than moved: a
published tag that changes what it points at is worse than a number nobody uses.
2026 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No conflicts: the branch had already taken main up to v1.3.65, and the only
commit main carried on its own was that release itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The page was a wall. Six paragraphs of procedure stacked above a form, so the
thing the page actually asks for — four fields — sat underneath an essay about
what would happen afterwards. It answered everything and showed nothing.
It is now built the way the settings page is built, because that page was
rebuilt for the same reason and there is no case for a second idiom: an eyebrow,
a title, a sticky rail on the left and panels of rows on the right.
The rail carries the six steps as two-word labels rather than paragraphs. That
is what the question at this moment actually is — where am I, how much is left —
and it fits in 232 pixels. The numbers carry the state; a tick beside them would
be a second sign for one statement, and a number can be counted.
What a step MEANS now appears in the row where it is due, not six times in
advance. The two provider steps get a panel of their own above the form, because
they have to be done before you save: the one-time code starts expiring the
moment you do. Everything after the form waits until there is something to say.
After saving the command becomes the page. Full width, its own framed block with
the warning in the header strip rather than floating above, and the two
remaining steps below it as ordinary rows.
Two token bugs went with it. `bg-canvas` does not exist — it was a class that
compiled to nothing, which is part of why the block looked wrong. And
`text-accent` on white is 2.9:1; the config says in as many words to use
accent-text for anything read, so the current step number does.
Also fixed, and it would have broken every takeover in production: PublicSiteGate
is appended to the whole web group, so while the site is hidden a server in a
rescue system fetching the archive would have received the 503 placeholder and
piped it into tar. The operator would have seen an unpack error on a machine
only reachable through the provider's console, with nothing pointing at a switch
in the admin area — and they are by definition neither on a management network
nor signed in, since leaving that state is the whole point. Exempted by ROUTE
NAME, not by hostname: the docblock rightly warns that exempting by host means
trusting a header the caller picks, but that warning is about the entire portal.
Behind these two routes are documents that are public anyway and an archive that
404s without a valid one-time code.
2026 tests pass, assets build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A hostname of its own for downloads. Public: the terms, the DPA and the TOM,
anonymous and indexable, under versioned filenames so an address in a signed
contract keeps meaning what it meant. Private: the bootstrap archive, gated by
the one-time enrolment code that is already in the copied line.
Set FILES_HOST to switch it on. Left empty, the archive stays on the portal
host and nothing changes.
2025 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A domain that exists only to serve one installer is not worth a certificate.
One that also carries the AGB, the AV and the TOM is — those addresses go into
contracts and onto invoices and have to still resolve in three years, which an
address that moves with the next rebuild of the portal cannot promise. That
reason is what changed the answer.
files.… over cdn./storage./archiv.: a CDN is an edge network and this is not
one, so the name would be a lie the day a real CDN goes in front of it.
"storage" reads like object storage or customer data, and a customer seeing it
will wonder whether their files live there. "archiv" says superseded, which the
terms currently in force are not — and R13 keeps paths and names English
anyway.
Two halves on that host, with opposite rules, and that is the whole point of
giving it its own name:
Public — storage/app/files/public/, served to anyone, indexable, because
somebody looking for the terms should find them. Versioned filenames:
agb-2026-01.pdf, never agb.pdf, so a contract signed in January cannot come to
point at conditions written in July. The rule is written where somebody will
look for it rather than enforced, because a upload that rejects a filename
helps nobody.
Private — the installer, and it is not a file in that directory at all: it is
built from deploy/bootstrap on demand. The gate is the one-time enrolment code
that is ALREADY in the pasted line, resolved without being consumed, because
the code is still needed for every progress report and for the registration at
the end. No second secret: a dedicated download token would never expire, would
sit in shell histories forever, and would travel in the same line as the
WireGuard private key — protecting the least sensitive thing with the exposure
of the most sensitive one. 404 rather than 403 on a bad code, and noindex on
the response.
Path traversal is answered before it starts: basename() only, no directory tree
under public/ by design, and dotfiles refused. The test walks ../../.env three
different ways.
Empty FILES_HOST keeps the archive on the portal host exactly as before, so
nothing breaks in the window between setting the variable and the DNS record
existing.
The hostname had to move into phpunit.xml rather than a config()->set(): routes
are bound at boot, so a test that sets it afterwards is setting it too late.
2025 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two features reach a tag at once, because the host-takeover branch was cut from
the operating-mode branch and carries all of it.
The operating mode: a test and a live setting for the whole installation, every
credential with a test slot beside its live one, a stored key that says which
mode it belongs to, and a Stripe catalogue that refuses to act on objects from
the other account.
The host takeover: the bootstrap script that turns a machine in a rescue system
into a finished Proxmox host, the archive that serves it, the one-time enrolment
code, and the six-step guide in the console under Hosts and Add host. None of
the script's steps has been accepted on real hardware yet — every hardware step
in its plan is still unticked, and the console work is what this release makes
visible.
Also in here, found while writing the takeover: host names were being built from
the CUSTOMER zone. RegisterHostDns says clupilot.com in its own docblock, and on
this installation a host was actually called fsn-01.node.clupilot.cloud. There
is a platform_zone now and the step uses it.
2017 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One real conflict, in SyncStripeCatalogue::handle(), and both sides were right.
Main's parallel work resolves the adoption singletons and takes "before" counts
so a run can tell created from adopted. The operating-mode work refuses a
catalogue whose stored objects belong to the other mode, and records which mode
the catalogue now belongs to.
Kept both, with the mode guard first. It REFUSES, so it must not run behind
anything that has already built state; the counters only need to be in place
before the create loop, and they still are. Reversing that order would have the
command resolve singletons and take counts on a catalogue it is about to reject.
2017 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The owner caught me putting the archive on clupilot.cloud. That is the customer
zone, and looking into it turned up the same confusion already sitting in the
code — in the one place that matters most.
RegisterHostDns says "fsn-01.node.clupilot.com" in its own docblock. The
validation comment in Datacenters says it. ServicesTest writes it out verbatim.
The step itself built the name from config('provisioning.dns.zone') — the
CUSTOMER zone — so on this installation a host was actually called
fsn-01.node.clupilot.cloud. Three places asserting one thing and the code doing
another.
OfficialDomains explains why that matters and is worth not weakening: two
registrable domains by design, the company's for site, portal and console, the
instance zone for customer workloads. A Nextcloud is third-party software that
strangers sign into, and on the same registrable domain as the portal it shares
cookie scope with it. A host name in that zone does not break the separation,
but it puts it in question, and the next slip is more expensive.
So there is now a platform_zone, derived from APP_URL when unset, and
RegisterHostDns uses it.
My own archiveUrl was broken for a second reason. It fell back to url() when
APP_HOST is empty — and APP_HOST is empty on most installations, because empty
means "the portal answers on any hostname" and that is the default. Called from
the console, url() would have produced the CONSOLE hostname, and the line would
have 404'd on a machine that is not allowed to reach the admin area at all. It
takes the host from APP_URL now.
The script no longer guesses its own name. It used reverse DNS, then the tunnel
address, then a hard-coded clupilot.net — a third domain that appears nowhere
else in this project and was simply invented. PrepareBaseSystem has the same
invention. A guessed name does not stay guessed: it ends up in /etc/hostname, in
/etc/hosts, in every log line and in every certificate request the machine ever
makes. CluPilot knows the name because it just assigned it, so it passes --fqdn
and the script refuses without it. The value now also survives the reboot in the
arguments file, which it would not have.
The ACME contact moved to .com for the same reason it was wrong: the operator
does not live in the customer zone.
Open, and NOT decided here: the owner also wants the host to get a public DNS
record and a certificate on the .com name. RegisterHostDns deliberately writes
host names only into the tunnel's dnsmasq, and says why — publishing them hands
every scanner the internal subnet and roughly how many hosts sit behind it.
Nothing in the current design needs a public certificate for a host's own name;
Traefik serves customer domains, not this one. Reversing that is a security
decision and belongs to the owner, not to this commit.
1992 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit shipped the guide AFTER the host was created. Its first two
steps happen at the provider — order the machine, boot the rescue system — and
whoever reads them there has the one-time code already in the clipboard with its
24-hour window running. The instructions were correct and in the wrong place,
which is its own kind of wrong.
Six steps now, not three, and the two provider ones are named rather than
assumed. They are marked "first" so the list does not read as something you can
start at the top of and work down: step 2 is half an hour of waiting, and doing
it inside a running code is exactly the mistake the page should prevent.
All six are visible the whole time, including the done ones and the ones still
to come. A guide that shows only the current step cannot answer "how much is
left", which is the question somebody has while a server they are paying for
sits in a rescue system.
It is on the hosts list too, collapsed. Somebody standing there may not have
ordered the machine yet — and that is step 1. Requiring them to click "add host"
to find out what the procedure is puts the answer behind the action it describes.
One component, two places, so the archive address in the instructions is the same
string the command line will carry. Two copies of that would drift, and the
difference would surface on a server that has already been paid for.
The step-4 command box moved into the guide rather than sitting beside it, so
after creating a host the whole procedure stays on screen. Somebody stuck at
step 5 should not be looking at a page that has become a single box.
1991 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The console could describe a takeover it had no way to start. This is the
vertical slice that closes that: a one-time code, the archive the rescue system
fetches, and the page that says what to do with both.
The command carries EVERYTHING the script needs before the tunnel exists,
because there is nothing to fetch — that is the whole point of spec §5. Which
means CluPilot generates the WireGuard keypair and admits the peer at the hub
before the machine has ever booted, and hands the private half over in the line.
It is worthless within minutes: task 9 of the script replaces it with one
generated on the machine.
Shown exactly once. The database holds only the code's hash and never the
private key, so leaving the page does not bring it back — it mints a new code,
which invalidates the old one. That is deliberate: a glance at somebody's screen
should be worth nothing an hour later.
Which is also why save() no longer redirects. Sending the operator to the host
detail page sends them away from the only value they need, and an existing test
asserted that redirect — it now asserts the opposite, with the reason written
next to it.
SHA-256 rather than bcrypt for the code, and the reason is not speed. Both
endpoints have to FIND the host by the code; with bcrypt that means trying every
row. The code is 32 characters of CSPRNG output, so it has the entropy that
stretching exists to manufacture.
resolve() and claim() are separate because progress reports arrive BEFORE
registration. If reporting consumed the code, a host could never register after
its first message.
The archive URL is always the public hostname. The console runs under admin.…,
but this line executes on a machine that must not reach the admin area — it is
locked down for exactly that reason — so route() from the console would emit a
hostname that 404s on a server only reachable through the provider's console.
The page warns about missing tunnel settings BEFORE the host is created, not
after. An empty hub key produces a line that looks clean, copies fine, runs, and
ends in a tunnel that never handshakes — discovered on the machine, after
somebody has already paid for it.
The three steps lead with the rescue system, because that is the one nobody
knows by heart, and it says enabling is not the same as booting into it — the
script refuses a running production machine, which is what a half-done switch
looks like from the inside.
1986 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One entry per section: how you notice, where to look, what it usually is. Above
all of them the rule the plan already states — a half installed machine is
reinstalled, not repaired — with the reason, which is not convenience. The
script is built so a second run on a freshly imaged machine always comes out the
same; a run on a machine somebody straightened out by hand is no longer that.
Repairing trades a repeatable result for a one-off, and the difference only
shows up on the next host.
rescue_checked is the stated exception, because nothing has been written yet. It
is the only section where fixing in place is right, and the runbook says so
rather than leaving the reader to infer it.
Two things that are easy to get wrong are answered up front. Whether the reboot
has happened is one `findmnt -no FSTYPE /` — tmpfs or overlay means still in the
rescue system, zfs means the first-boot hook already resumed. And when a host is
unreachable, the first question is whether the firewall had even run: it is
applied last, so if `registered` is not done, the cause is the network and not
the rules.
The installer can be watched while it runs. If QEMU is still up, forwarding 5900
over SSH shows what the ISO is doing, and an input mask there means the answer
file was not accepted and the installer has dropped into interactive mode where
it will sit until the hour expires. That failure otherwise arrives as a timeout
saying nothing.
The last heading is the honest one: everything here is written from the script
and the findings behind it, not from incidents. What actually stops on the first
real run belongs in this file afterwards, quoted as it appeared in the console.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The last section carries three things, and none of them could sit anywhere
else: the registration and key swap, Traefik's token plus the proof that a route
table actually arrives, and the nftables lockdown that ran last in the old
pipeline for a reason.
`|| true` is gone from the pveum calls, and not by deleting it. `role add ||
true` applied the privilege list only on the run that first created the role, so
a privilege added later reached new hosts and no existing one — which is exactly
how Sys.Modify was missing everywhere, surfacing much later and somewhere else
as a 403 on POST /cluster/backup that failed a paying customer at
register_backup. So: add, and on failure modify, as two separate commands so a
failure can be attributed. For the user and the ACL the tolerated `|| true` is
replaced by looking: try, and if it fails, check whether the desired state holds
anyway. A `|| true` hides "already there" and "went wrong" equally well.
The privilege list is copied verbatim from config/provisioning.php and checked
against it — 18 privileges, byte-identical, Sys.Modify included. It is granted on
/ because both endpoints that need it check / and nothing narrower satisfies
them.
The key swap follows the four steps this plan was amended to require. Register
first and hold the answer, switch wg0 to the new private key, PROVE a fresh
handshake, and only then let the old key go. If the handshake does not come, it
puts the old key back and fails loudly. Ordering alone says when to discard, not
whether the new key carries — and a host that locks itself out on the final step
is the one failure nobody fixes remotely.
Traefik's token is written here because here is where it exists, and the route
table is then fetched directly with that token and that URL. Section 7 could not
give this proof; it is not skipped, it is given where it can be.
The lockdown comes dead last, with the full ruleset from SecureHostFirewall
including the ICMP correction — ICMPv6 neighbour discovery and packet-too-big,
IPv4 fragmentation-needed — because a bare policy drop takes IPv6 down within
minutes and black-holes large transfers instead of failing them. The ruleset is
validated with `nft -c` before it is applied, since a partially loaded ruleset
means policy drop is in place and the exceptions are not. There is no automatic
reopening: a firewall that reopens itself under failure is not a firewall, so the
emergency release is a script for the provider's console.
Verified without hardware: all seven files dash-clean; json_field reads
full-tokenid, value and host_token without jq; the privilege list is identical to
the config's; and the rendered ruleset carries policy drop, both ICMP families,
80/443, the DHCP rebind rule and the tunnel-only 22/8006. Step 2 unticked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All three Block A traps are CHECKED here rather than assumed, because each one
has already cost a paid order and none of them shows a symptom until it is too
late.
Trap 2 is checked before anything is built: virt-filesystems is asked whether
the image uses LVM and whether root is the last partition, and the section
refuses if either is wrong. Debian's cloud image satisfies both by construction,
which is exactly why it is the base — but "satisfies it by construction" is a
claim about an artefact somebody else builds, and it gets verified rather than
trusted.
Traps 1 and 3 are checked AFTER the image is customised, because virt-customize
reports success even when apt quietly did not install something. qemu-guest-agent
has to be in /usr/bin or every provisioning run hangs in WaitForGuestAgent until
it times out, and the compose file inside the image has to carry `user:
www-data` or every occ call fails — including the acceptance check that decides
whether a customer's instance is usable.
The template is verified by its ATTRIBUTE, not its existence. VerifyVmTemplate
checks only that 9000 is there, which a VM that merely happens to be numbered
9000 also passes; `template: 1` is passed only by a template.
Docker comes from Docker's own repository, not Debian's. docker.io ships no
compose plugin and Debian's docker-compose is the old Python one, which does not
read this file at all.
The compose file is deliberately customer-independent: no password, no name, no
domain. Everything variable arrives in /opt/nextcloud/.env, written into the
guest by cloud-init at clone time. Baking a password into an image copies it
onto every host and into every instance, and leaves it there long after the
customer has changed it.
OVERWRITEPROTOCOL, OVERWRITEHOST and TRUSTED_PROXIES are set because TLS ends at
Traefik on the host. Without them Nextcloud builds its own URLs with http://,
the login loops, and WebDAV clients get handed an address that does not exist.
Storage is discovered rather than named: local-zfs is what the PVE installer
creates on ZFS, local-lvm on ext4, and `local` frequently cannot hold disks at
all — an importdisk there fails only after the copy.
Verified without hardware: dash-clean; the compose file parses, carries `user:
www-data` on the app service, publishes 80, and every credential is a ${...}
reference rather than a literal. Step 2 unticked, and it is the one the plan is
most insistent about: the template must actually be cloned, started, and asked
`occ status` as www-data. Without that clone it is not proven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nexxo