Three console pages were fiction. The front page claimed 42 customers, 39
instances, four hosts named pve-fsn-1..3 and €7,842 a month, over a twelve-month
growth curve; the instance list held seven invented machines; the revenue page
reported churn and a trend for a business with no recorded history. All of it
was hard-coded. It read like a running company and measured nothing.
They now read the database. Two figures are gone rather than approximated —
the revenue trend, which needs a monthly history nobody records, and churn,
which needs a base the data cannot supply. ARR stays, labelled as the
projection it is. The green "all systems normal" badge is computed from the
notice list instead of asserted, and the notices themselves come from failed
runs, hosts reporting errors or gone quiet, and monitoring that is down.
Host load is the one number that had to agree with something else: placement
counts the VM disk allocation, ignores a failed instance that never got a VM,
and subtracts the host's reserve. A dashboard doing its own arithmetic would
show a host as comfortable while orders were already being refused on it, so it
uses the host's own accounting — with the filter moved into a scope both share,
and the sum preloaded so listing hosts stays one query.
The instance list drops the Nextcloud version column: that version is not
recorded anywhere, and a column filled with a plausible number is worse than no
column. Statuses the lifecycle writes but nobody had translated no longer
render as "admin.status.failed".
The price sheet also gains the included traffic, which the catalogue has always
carried and the page simply never printed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The marketing page was a generic centred hero in the system font stack: it
neither used the design system the console is built on nor looked like the
product it sells. Rebuilt around what CluPilot actually promises — security you
can produce evidence for — so the page reads as a controlled technical
document: a specification plate instead of a hero image, numbered sections, an
audit register with rhythm and proof per measure, and a price sheet rather than
floating cards. IBM Plex Serif joins Sans and Mono as the display voice, self
hosted as static files so the public page renders even without the Vite build.
The prices were written into the page by hand, and the page and the catalogue
had already drifted apart on three of four plans: the site advertised 249 € for
a plan the checkout charges 399 € for. The sheet now reads the catalogue, like
every other caller — including the currency, which is configurable and was also
hard-coded here.
The catalogue fails loudly on purpose; a public website must not. A broken or
overlapping catalogue is caught in the controller alone: the page still
renders, the sheet is replaced by "on request", and nobody is quoted a number
the checkout would not honour.
The sign-in and registration panels shared a copy-pasted orange gradient. They
now share one component and the same ink plate as the site, so the two halves
of the product no longer look like two companies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The allowlist is managed in the console, which is fine until the address you
manage it from changes — and then the page that would fix the problem is the
page the problem blocks. Every gate needs a door that does not depend on
itself, and on a server that door is a shell:
php artisan clupilot:console-access show
php artisan clupilot:console-access allow 203.0.113.7
php artisan clupilot:console-access open
The address check moved to RestrictConsoleNetwork::isNetwork() so the command
and the console apply the same rule. Codex caught the version that did not: a
typo like 203.0.113.9/99 was stored, reported as success, and matched nothing —
leaving whoever was recovering still locked out, now believing they were not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RestrictAdminHost answers under which NAME the console responds — and the
caller picks the Host header, so it can never answer who is asking. This does:
a network gate on the client address, which behind a trusted proxy is not
something the client chooses.
The management VPN is always in the list and cannot be removed — it is the one
way in that survives a bad entry. Beyond that the owner keeps their own
addresses in the console, because being away from the VPN should not mean being
locked out, and the person who needs to change that list is the person sitting
in front of it.
Two refusals rather than warnings, since by the time a warning renders the
request that would show it has already been rejected: switching the restriction
ON is refused unless the address doing the switching is already covered, and
removing the entry you are sitting behind is refused. Entries are validated as
an address or CIDR — a typo that matches nothing is how someone locks
themselves out while believing they have not.
404, never 403.
Registered as persistent Livewire middleware, and verified in a browser that it
holds there: with the settings page open, narrowing the list turned the next
action from 200 into 404. Codex read the path guard as skipping
/livewire/update; Livewire in fact replays middleware against a duplicate of
the request carrying the original component's path, so the guard matches and
the client address is preserved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Livewire re-applies only the middleware on its persistent list when an action
posts to /livewire/update. Its defaults cover auth; ours were not on the list —
only RestrictAdminHost had been added. So EnsureAdmin and EnsureCustomerActive
guarded the PAGE and not the actions, and the page is not where the actions run:
a signed-in non-operator who could reach the admin host could drive console
components, and a suspended customer could keep driving the portal.
Found by Codex while reviewing the domain-separation design, and confirmed
against Livewire's own default list in vendor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
VERSION at the repository root is the release number and the only place it is
written down — a file rather than `git describe`, because a source archive, a
shallow CI checkout and a container built without .git have no tags to
describe. Cutting a release is editing that file and merging it; CI creates the
annotated v<VERSION> tag once green, only on the commit that raised the number,
and never moves it. Servers are pinned to these.
Two modes, and the difference is the point. RELEASE=v1.0.0 pins a server to an
immutable tag, checked out detached: it moves when someone decides it moves.
BRANCH=main follows the edge, where CI tags a commit only after it is pushed.
The mode is remembered, so re-running the updater on a pinned box does not
quietly walk it back onto main.
Going backwards is refused, by commit ancestry rather than by comparing version
strings — a tag can be cut from anywhere, and only ancestry says whether history
is going back. The schema has already moved forward by then, and the migrations
needed to reverse it are not in the older checkout at all.
What the console reports comes from a manifest written atomically after every
step succeeded, never from live git: git says what the files are, not whether
the deployment came up. So a failed update keeps reporting the version that is
actually serving — including its version number, not the newer one the checkout
has already moved to. A deployment pinned to the tag reads "1.0.0 (abc1234)";
anything else reads "1.0.0-dev (abc1234) · main", because every commit after the
tag still carries VERSION=1.0.0 and is not that release. Reporting it as one is
how a bug gets filed against the wrong code.
Codex reviewed the design before it was written and the result six times after.
Its findings, all real: the version had to come from the manifest too, not just
the commit; branch names need JSON escaping or the manifest silently vanishes; a
`case` glob does not anchor and accepted 1x.2y.3garbage; pinning to the commit a
server already sits on skipped recording the mode, and then skipped detaching;
a manifest that failed to write would never be repaired; an unparseable
timestamp would 500 every admin page; and an empty tag list read as a
connectivity failure.
466 tests green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two findings from reviewing all five phases together rather than one at a time.
The webhook held every event a handler answered `null` to — but `null` also
means "already recorded" and "deliberately skipped", which is every checkout's
own invoice and every redelivered renewal. Their contract is right there, so
`replayHeldFor()` could never come back for them, and ordinary Stripe traffic
silted up the holding area until the weekly prune. It now holds only what it
genuinely cannot match — and if the contract appeared in the meantime, applies
the event instead of dropping it: the creation race is narrow, but losing a
cancellation to it would leave us serving someone who had left.
Also fixes a rollback that predates this work and blocks `migrate:fresh` on
MariaDB entirely: dropping the unique index on customers.user_id fails while
the foreign key added one migration earlier still depends on it. Verified by
building all 33 migrations from nothing on MariaDB, rolling every one of them
back, and building them again.
458 tests green. Codex clean on the full branch diff.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Product per plan family and a Price per priced row of a published version,
plus the four subscribed webhook events that were arriving and being ignored.
`stripe:sync-catalogue` mirrors the catalogue. Idempotent twice over: a stored
id is skipped, and each call carries an idempotency key derived from our own
row, so a crash between Stripe creating a Price and us recording its id gives
back the same object rather than a second one. That matters more here than
usual — a Stripe Price cannot be edited or deleted, so a duplicate is
permanent. Drafts are not synced at all: a version that has promised nothing
has no business in a price list. --dry-run shows what would be created.
The webhook now handles invoice.paid, invoice.payment_failed,
customer.subscription.updated and customer.subscription.deleted. It never
re-derives an amount: the invoice is the authority for what was charged, and
recomputing it from our catalogue would produce a second, disagreeing answer.
A failed payment is recorded and nothing else — Stripe runs the dunning
schedule, and cutting a customer off on the first failure would punish an
expired card as though it were a refusal to pay.
Only a cycle renewal moves the term on. Stripe also sends paid invoices for
prorations and manual charges, and the checkout's own invoice — which is
already in the register as the purchase, and would otherwise double every
customer's first payment.
Stripe does not guarantee delivery order, which is the source of most of the
care here:
- state changes are judged and written in one transaction with the row held,
so two deliveries cannot both pass a check made on a stale copy;
- an older event never overwrites a newer one, with a rank breaking ties
between events sharing a second — their timestamps have one-second
resolution, and a failed attempt and its successful retry routinely do;
- a term is never shortened, and an ended contract is never revived;
- an event for a contract that does not exist yet is HELD and replayed the
moment it appears, rather than acknowledged and forgotten. A cancellation
dropped that way would leave us serving someone who had already left. Held
rows that never match are pruned after a week.
Register entries are keyed by what they are about — the invoice, the attempt,
the subscription — with a unique index doing the work, because a check two
concurrent deliveries can both pass is not idempotency.
PlanChange is now documented as the preview shown before someone confirms, not
the invoice: Stripe's proration accounts for tax, existing credit and the exact
second the change lands, and ours cannot.
Not run against the live account — `stripe:sync-catalogue` creates objects that
cannot be deleted, so that is the owner's call. The dry run lists 12.
457 tests green. Codex review clean after seven rounds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things the contract could not answer on its own: what happened, and what
the customer's whole bill is.
`subscription_records` is append-only, one row per commercial event. Flat
columns for everything searched or relied on as evidence — event, customer,
subscription, plan family and version, term, net/tax/gross, currency, tax rate,
reverse charge, Stripe ids — PLUS a versioned JSON copy of the whole snapshot.
Not JSON alone: you cannot query it, and "the amount is in there somewhere" is
poor evidence. Copied, not joined, so a row still answers after the customer,
the plan or the version have gone.
The flat columns describe the TRANSACTION. Gross is what was actually taken;
net and tax are that gross split by the rate that applied on the day. A
discount lowers the taxable amount rather than creating negative VAT, and a
free checkout is recorded as free instead of as paid in full. What was agreed
sits beside it in the snapshot, so a charge that differs from the catalogue is
preserved as a question rather than reconciled away.
The register refuses to be rewritten, in bulk as well as one row at a time —
model events do not fire for `query()->update()`, which is exactly the shape a
careless data fix takes.
`subscription_addons` carries the owner's rule: a customer's total is their
subscription plus their modules, and all of it is frozen at what was agreed.
Price protection covering only the plan would let a module quietly double for
someone who booked it two years ago. A module they have NOT booked is a sale
still to be made, at today's price — so the catalogue is consulted only for
what is not in this table. Several bookings of one module are summed rather
than reduced to one arbitrary row, or the page and the bill would disagree.
Concurrency: one order books one module, enforced by a unique index rather than
a lookup two retries can both pass; cancellation is an atomic claim; and each
booking commits with its own register entry, so a failure cannot leave a sale
without evidence and a retry that finds the booking cannot skip the event.
Verified in the browser: with the module raised from 29,00 € to 59,00 € in the
catalogue, the customer who booked it still sees 29,00 € and a total of
208,00 €, while a new customer is quoted 59,00 €.
436 tests green. Codex review clean after seven rounds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The owner can now do from the console what only a config edit could do before:
create a plan line, draft a version, price it, publish it into a window, and
pull a plan out of the shop.
Two pages and one modal, because the catalogue has exactly two levels. The
plan list is a name, a rank and a kill switch. Everything that can be got
wrong — capabilities, prices, windows — lives on the versions, where the page
is built around the one rule that matters: a draft is freely editable, a
published version is not touchable at all. So drafting and publishing are
separate acts, and publishing says plainly that it is final.
The console refuses everything the catalogue refuses, on the form rather than
as a stack trace: an overlapping window, a window that ends before it starts,
a version with no price for a term we sell on, and — new here — a version with
no VM template, which would be bought and then fail provisioning every time.
Numbers are bounded on both sides, because a mistyped price otherwise
overflows the column and answers with a 500 instead of saying which field was
wrong. Performance class and features are picked from the keys we have labels
for; free text would be frozen at publication and shown to customers as a raw
translation key forever.
Concurrency, since two admins share one catalogue: draft numbers are allocated
under a lock on the family, publication is an atomic conditional claim, and
discarding a draft is one statement conditional on it still being a draft —
otherwise a draft published in the meantime could be deleted out from under
the customers now contracted to it.
`plans.manage` (Owner and Admin) guards the pages themselves, not only the
buttons, and the modal authorises itself — prices, drafts and unreleased plans
are not something to leave readable to anyone who types the URL.
Also fixes the shared checkbox, which ticked in the browser's own blue on
every page that used one: a native checkbox ignores text colour and needs
accent-color.
Verified in the browser: withdrawing a plan in the console removes it from the
customer's billing page immediately, and restoring it brings it back. The
public landing page carries no catalogue-driven plan list, so there is nothing
there to hide.
421 tests green. Codex review clean after six rounds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Plans lived in config/provisioning.php, where the owner cannot reach them and
where a plan has no history. They now live in plan_families / plan_versions /
plan_prices — a name that never moves, what the plan WAS at a point in time,
and a price per version and term with its own Stripe Price id.
- Availability is computed on every read (available_from <= now < until,
half-open, UTC) plus a per-family sales kill switch. Nothing schedules a plan
into or out of sale: a job that fails to run is a plan that silently
misbehaves.
- Overlaps crash rather than resolve. currentVersion() uses sole(), and
scheduling takes a lock on the family and rejects an overlapping window —
two versions on sale at once would decide a customer's price by row order.
- A version is frozen from publication, not from first sale, and so is its
price: a checkout is not instant, and an amount edited between the session
opening and the webhook landing would contract someone at a price they were
never quoted. Repricing publishes a new version, as Stripe requires anyway.
- Neither a published version, its price, nor a family with customers can be
deleted, and a family key cannot be renamed. All of those would null the
provenance off existing contracts.
- Subscriptions and orders record plan_version_id. A historical reference
resolved by plan NAME hands back today's terms, which is the split-brain one
level up. A checkout that carried its version is honoured even after the
window closes — they paid for what they were shown.
Switched atomically and failing closed: config('provisioning.plans') and
plan_features are gone, and nothing falls back to them. A fallback would
resurrect a plan the owner had just switched off. The seed lives in the
migration, and PlanCatalogueTest pins what the config catalogue sold as the
shadow comparison. `php artisan plans:check` reports overlaps, gaps and
missing prices before a customer finds them.
Contract-backed displays, which were reading the live catalogue:
seat limits, the cloud card, the plan card, and admin MRR — the last three now
divide a yearly contract down via Subscription::monthlyPriceCents(), since all
three label the figure per month.
Two recovery gaps closed along the way: the order now commits before the
contract is opened (so a contract failure can never erase the record of a
payment), a Stripe retry repairs a missing contract and restarts a run stranded
with no_subscription, and TickProvisioning sweeps runs left pending by a crash
rather than only running/waiting ones.
402 tests green. Codex review clean after thirteen rounds. Verified in the
browser: portal, billing and console render unchanged off the new catalogue.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pipeline re-resolved config('provisioning.plans') by order.plan, so the
subscription snapshot protected a customer's price but not their machine:
shrinking a plan resized an existing customer's VM on its next run. Nothing
created a subscription either, so closing this meant opening the contract at
purchase and pointing provisioning at it.
- OpenSubscription freezes the catalogue onto a subscription when a checkout
is paid; StartCustomerProvisioning calls it inside the order transaction.
- CustomerStep::plan() reads the frozen snapshot. ValidateOrder and
ReserveResources fail closed with no_subscription rather than falling back
to the catalogue, which is the bug itself.
- template_vmid joins the snapshot so a re-clone cannot pick up a blueprint
published after the sale. Deliberately outside FROZEN: it is how we build
the machine, not a term the customer is owed, and a dead template must be
replaceable without cancelling a contract.
- TrafficMeter reads the allowance off the contract too — cutting a plan's
traffic was otherwise enough to start throttling someone who bought more.
- The migration backfills contracts for orders that already bought something,
reconstructed from what was actually delivered where an instance exists,
and adopts an existing order-less contract instead of opening a second.
Orders paid in a currency the catalogue cannot price get none, matching the
checkout path.
price_cents stays the catalogue's NET price, which is what PlanChange
prorates against — not Order::amount_cents, which holds Stripe's GROSS total.
Reconciling the two belongs to the proof register and Stripe (phases 4/5).
Also pins STRIPE_WEBHOOK_SECRET blank in phpunit.xml: the operator's real
secret was reaching the suite from .env and rejecting every unsigned test
payload, which is why 7 webhook tests failed before any of this.
Verified in the browser: with team traffic cut from 3000 to 500 GB in the
catalogue, the customer's portal still shows 3 TB.
373 tests green. Codex review clean after three rounds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A cancelled subscription still reported an upgrade as allowed and priced it, so
a caller trusting that would have provisioned and billed a customer who had
already left. And any term string other than the two we support was silently
priced monthly while keeping the unknown value — a subscription whose price and
billing period disagree. Both are refused now.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flooring the remaining time made an upgrade with under a day left cost nothing,
and an upgrade requested after the period had ended cost nothing while being
applied immediately — the bigger plan for free. Remaining time now rounds up
while any service is left, and an expired period defers the change to the next
term instead of pricing it at zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Once the term is over, the downgrade has to be allowed — otherwise the job that
is supposed to carry it out never can, and the change waits forever for a date
that has already passed.
And a grandfathered plan can be cheaper than the smaller plan costs today, which
made the goodwill credit negative: an invoice for the privilege of downgrading.
Clamped to zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With grandfathered prices the comparison inverts: a business plan bought when
it cost less than today's team plan would have treated a move to team as an
upgrade — charged immediately, while the customer loses resources. The plan's
rank is frozen with the rest of the snapshot and decides the direction; prices
only decide the amount.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The plan catalogue describes what we sell today; a customer who signed up last
year bought last year's terms. Every commercially relevant condition — price,
quotas, seats, the hardware behind the plan — is now frozen onto a subscription
at signup, and the model refuses to let any of it be rewritten afterwards. A
price rise applies to new subscriptions and cannot reach back into an existing
contract.
PlanChange holds the two rules, computed against the frozen price rather than
today's catalogue:
- Upgrading is immediate and pro rata: the new plan for the days left in the
paid term, minus what the old plan was worth over those same days. On the
last day of a month that is one day's difference, not a month's.
- Downgrading waits for the end of the term. A yearly customer bought a year
and can move down when it is up; a month is a month. A mid-term downgrade is
a goodwill decision, not a self-service button, and its credit covers only
the unused part of the difference.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A verifier that returns the number in display form ("DE 811 907 980") would
have failed the comparison against the normalised current value and silently
switched a genuine reverse-charge customer back to domestic VAT. Both sides are
normalised now, and vat_id_verified_at is a real datetime rather than a string.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A timestamp alone said "some number was checked once": editing the field left
it intact, so a customer could swap a verified foreign VAT ID for any
plausible-looking one and keep zero-VAT pricing. The verified value is stored
and compared, which makes the rule self-enforcing — no writer has to remember
to clear a flag, and there are several writers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Any non-empty string starting with two characters other than AT switched the
customer to reverse charge — typing "XX123" was a 20 % discount. Reverse charge
now requires a VAT ID that is verified, belongs to an EU member state other than
ours, and looks like a VAT number at all. Unverified is the normal state and
means the domestic rate: over-collecting is correctable, under-collecting is a
tax liability.
Changing the number clears its verification. Verification itself (VIES) is not
built yet, so reverse charge stays off until someone confirms a number — which
is the safe direction to be wrong in.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex was right that this could misstate real charges: an EU business with a
VAT ID registered in another country is billed under reverse charge, and we
were adding 20 % Austrian VAT to their total anyway. TaxTreatment resolves it
from the customer's VAT ID, and the whole page — cart, plan cards, add-on
cards — now states one treatment instead of contradicting itself.
Explicitly NOT handled: cross-border sales to private individuals, which are
taxed at the buyer's national rate under OSS. That needs a maintained rate
table and a tax adviser, not a guess, so those fall back to the domestic rate —
over-collecting rather than under-collecting.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The note rounded VAT after aggregating while the total rounded per order, so
cent-level prices made the two disagree — and a customer who spots that stops
trusting every other number on the page.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two contradicting plan upgrades sat in the cart at 799 and 399 € — no checkout
could resolve which one the customer meant. Choosing another now replaces the
pending one and says so; add-ons still stack, because buying 200 GB as two packs
is a sensible thing to want.
Every price now states net or gross and how often. The cart shows net per line
with "pro Monat" or "einmalig", then subtotal, VAT and gross — and separates
the monthly recurring amount from a one-off traffic top-up sharing the same
cart, because those are two different commitments. The rate is configurable
(CLUPILOT_TAX_PERCENT, 20 % default) since it follows the seller's country.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The handoff stayed in the cache for its full ten minutes, so any replayed
component request could fetch the plaintext again — "shown once" was a figure
of speech. It is consumed on the first read and the token dropped; the payload
reaches the view and nothing else.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two from Codex:
- The job accepted any stored instance with a host. A closed one's VMID can have
been reused on the same machine, and the reset would then land on another
customer's VM. It now requires a live instance, and the action is not offered
for anything else.
- An unreachable Proxmox or guest agent throws rather than returning an exit
code, so nothing was ever written to the handoff and the modal polled forever.
The throw is caught, and a failed() handler covers anything that still escapes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Impersonation borrows the customer's portal session; this is access to their
installation, which is a different thing and now a different button.
Nextcloud has no passwordless admin jump, so this does the only honest thing it
offers: it resets OUR managed admin account inside that installation and hands
the credentials over once. The customer's own accounts are untouched, and the
next request sets a new password again.
- New capability instances.adminlogin, Owner and Admin only — stronger than
impersonation, because it hands over control rather than a session.
- The operator's own password is required every time, rate-limited: taking over
a customer's installation is not something an unattended browser should manage
on its own.
- The reset runs on the provisioning worker (it owns the tunnel and the Proxmox
credentials); the console polls a handoff token, so the credentials never
enter a Livewire snapshot.
- A silent instance is reported as such instead of appearing to succeed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"5 purchases pending" told nobody what they had ordered and offered no way to
change their mind. The billing page now lists each pending purchase by name,
what it costs, when it was added, and the total — with a remove button per row
behind a confirmation, like every other destructive action in the console.
The wording lives on the Order model rather than in the view, so the cart, the
invoice list and any later confirmation mail cannot each invent their own name
for the same row. Removal is scoped to the customer's own still-pending orders,
checked in the modal itself: modals are reachable without the page's guards.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jobs that pin themselves to the provisioning connection bypass
QUEUE_CONNECTION, so two tests opened a Redis connection — fine on a machine
running the stack, a RedisException in CI. A test run must not depend on
infrastructure being up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI has no build, so every page rendering through @vite threw a view exception —
dozens of failures with one cause. withoutVite() makes the suite test the
application rather than the state of the asset pipeline; a broken build shows up
in the build job, which is where it belongs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Once a key is free, someone can legitimately create a new access with it. The
forced revocation looked the key up and would have deleted that person's access
and pulled their key off the hub. It now acts only on an adoption artifact — a
system peer with no owner and no creator — and leaves anything else alone.
Re-issuing also threw when VPN_CONFIG_KEY was missing, which is exactly the
situation the console tells people to fix by re-issuing. It now hands the new
config over once instead of storing it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex found that a sync landing between the key swap and its removal would see
the old key as unknown. Writing the test showed something worse than the
duplicate he predicted: the adoption cannot even happen — the address still
belongs to the live access — so the insert violated the unique index and took
the ENTIRE reconciliation down with it. One stale key would have stopped every
peer's state from updating.
The sync now recognises that case and queues the removal instead of adopting,
and a key with no row of its own is revoked with force: it can only ever be
removed, never kept, so the removal cannot be talked out of it by a row that
should not exist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An access created without storing its config had no download button and no
explanation — it looked broken. It now shows a dimmed button that says why
(the private key only exists on your device), and every staff access gets
"Re-issue": a new keypair, old key off the hub before the new one goes on,
so two peers never claim the same tunnel address at once. A stored config is
re-encrypted in step, or the owner would download a key the hub no longer
accepts.
Host peers are excluded: their key belongs to the machine's own wg0, and
swapping it here would cut the host off with nothing left to repair it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A registration finishing while the purge waited on the run locks wrote
dns_record_id after the purge had already loaded the host, so it skipped the
deletion and then deleted the row holding the only id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The failure may be a lost response to a request that did create the record.
Advancing straight away stored no id, so PurgeHost could never remove it and the
host's management address stayed published. The upsert is idempotent, so a retry
recovers the id; only after the attempts run out does the onboarding continue
without a name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two legacy codes can normalise to the same label (eu_west and eu-west), and
separate counters then handed both of them eu-west-01 — a unique-constraint
failure inside the reservation, which blocks onboarding rather than being a DNS
problem the step can shrug off. Lock, counter and the in-use check all key off
the label now.
(The helper was also called label(), which HostStep already declares as the
step's display name — renamed.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
alpha_dash accepted eu_west, -edge and edge-, none of which are valid DNS
labels — every host in such a datacenter would have failed registration and,
because DNS is non-fatal, silently ended up without a name. The console now
requires a proper label, and the step normalises anything created before that
rule so existing rows still get a usable name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A lost response, or a retry after a later step failed, made every further
attempt a 404 — and the host impossible to purge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Swallowing it and carrying on deleted the row that held the only reference to
that record, publishing the machine's management address for good. The purge now
fails instead — and is retried, since it re-reads the host and its other steps
are idempotent — so a broken DNS provider becomes a visible failed job rather
than a silent leak.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex was right that my never-reuse claim did not hold:
- The lock was released with only a candidate in hand, so two concurrent
onboardings in one datacenter could pick the same name and overwrite each
other's record. The name is now persisted on the host while still locked.
- The number was derived from the hosts that happen to exist, so removing the
highest one handed that number straight back out — a cached name would then
resolve to a different machine. The counter is stored per datacenter, floored
by what is actually in use so a wiped settings store cannot reissue live names.
- PurgeHost deleted the row that carried the record id, leaving the machine's
management address published with nothing left to clean it up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fsn-01.node.clupilot.com, numbered per datacenter and never reused — removing a
host must not renumber its neighbours onto its name, so the number is stored
rather than derived.
The record points at the host's WireGuard address, not its public IP: the name
exists so an operator can reach a host by name over the VPN, and publishing a
Proxmox host's public address would hand every scanner a target, which is the
one thing this network design avoids.
DNS is convenience, not a prerequisite: a failure logs an event and lets the
onboarding finish rather than stranding a host that is otherwise ready.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Defaulting missing netin/netout to zero reset the baseline, so the next real
sample added the entire cumulative counter again — a customer throttled for
traffic that was counted twice. Such a sample is skipped and logged now.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tying the release to the creation of the new month's row meant a single failed
call was never retried: the row exists from then on, and the current period
looks unthrottled, so nothing could detect the stale NIC limit. A customer would
have stayed slow for a whole month they had paid for. Checked on every sample
now, with a test that makes the first release fail.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two from Codex:
- A throttled instance kept its NIC limit into the new month: the new period
row starts unthrottled, so neither branch of enforce() fired and nothing ever
took the limit off. A customer would have stayed slow through a month they
had already paid for.
- PublicSiteGate exempted livewire/* wholesale. That endpoint is shared by the
console and the portal, so a signed-in customer could keep driving portal
components — billing included — while the portal was supposed to be offline.
Operators pass the normal check anyway, which is all the console needed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Customers can now see what they have used and what is left, and the service
slows down rather than stopping when the allowance is gone — a slow Nextcloud
gets a top-up, a dead one gets a cancellation.
- instance_traffic keeps one row per instance per month. Proxmox counters are
cumulative since the VM last started, so usage is the difference between two
samples, and a counter that went backwards means a restart, not a refund.
- Outbound is what counts: inbound is free at our providers and egress is what
Hetzner's 20 TB per server applies to.
- CollectInstanceTraffic samples every 15 minutes, warns once per threshold
(80/95 %) rather than on every run, and at 100 % limits the VM's NIC via
Proxmox — released again as soon as the customer tops up or the month rolls
over.
- The dashboard gets a full-width band (it throttles the service, so it is not
a tile among tiles) with the top-up offer right next to the warning.
- Bytes are formatted in SI units now: allowances are computed in SI, so
dividing by 1024 made a 1000 GB plan read as "931 GB".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
While the product is still being built, the marketing site and the customer
portal should not be reachable — but they must stay reachable for us.
- A toggle in the console (site.manage, Owner/Admin) stored in a new
app_settings table, because this has to be flippable without a deploy.
- Outsiders get a placeholder with 503 + noindex, not 200: a 200 invites search
engines to index the placeholder as the site's content, which is far harder
to undo than to prevent.
- Anyone on the management VPN, and any signed-in operator, sees the real site.
The console, Livewire's endpoint, the Stripe webhook and the health check are
always reachable — otherwise the switch could only be flipped once.
- robots.txt is generated by the app and follows the switch. It had to stop
being a static file: nginx short-circuited it and Laravel's stock file said
'Disallow:' with an empty value, so crawlers were never told anything.
- Settings reads fall back to the default when the table is unavailable. The
gate reads one on every request, so a deploy running new code before migrate
would otherwise answer the entire site with a 500.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The action labels wrapped to two lines and crushed the column — now icon
buttons with tooltips, like every other admin table. The form column was too
narrow for its content: the datetime fields silently clipped their own value,
so they are stacked and full width now, with duration chips because typing an
end timestamp by hand is the fiddliest part of the form. Host rows show a real
selected state and a per-datacenter count, and the impact column says
"1 Host · 1 Kunde" instead of "Host(s) · Kunde(n)".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ownership alone kept download, block and delete reachable for someone whose
roles were removed by any path other than revokeStaff() — a direct role edit,
say. Being staff is now part of ownership, so revocation closes these doors
without depending on whoever removed the role also remembering the tunnel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The download counter was read-modify-written, so two concurrent retrievals
recorded one. It is an audit trail — under-reporting is the failure mode it
must not have.
The creation-race test broke when issuing moved into a transaction: its
simulated competitor writes inside our transaction, so the rollback removes
that row too. It now asserts what actually matters (the loser is told, and
creates nothing) and states what a single-connection sqlite test cannot show.
This test was red in the previous commit — my mistake for committing before
reading the run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A revocation committing between the operator check and the insert would find no
peer to remove and still leave the revoked colleague with a brand-new tunnel.
Issuing now runs in a transaction that locks the owner row — the same row
revokeStaff() locks — so the two cannot interleave.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every five-second poll re-rendered the page, putting the private key into
another HTTP response — the handoff kept it out of the snapshot but not out of
the responses. Traffic figures can wait the minute it takes to copy a key.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Creating a second access with a supplied key skipped the config branch, so the
first access's private configuration stayed on screen under the new name — and
someone would have handed it out.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Once revealed, the plaintext sits in the handoff cache for ten minutes, and the
modal read it from the token alone. A former owner could keep pulling the key
out of an open modal after revocation — the exact window purgeSecret() exists to
close. Every request now re-checks the policy and drops the handoff when it no
longer holds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The sidebar still gated the VPN entry on vpn.manage, which the split deleted —
so the page would have vanished from the console for every role, Owner
included. Every operator can hold their own access, so the entry is shown to
all of them and the page itself shows only what the policy allows. Guarded by a
test across all five roles, which is the assertion my rewrite had dropped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A worker could pick the job up mid-transaction, still see the peer as active,
leave it on the hub — and never be asked again, because the committed state is
only a soft delete. A revoked colleague would keep their tunnel until some
later reconciliation noticed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex spotted the migration labelling sync-adopted peers as staff with no
owner, which breaks the invariant the same migration introduces. The live sync
had the identical hole — it never set kind at all, so every adopted peer landed
on the 'staff' default. Both now use system for a peer nobody is behind, and
promote it to host once the link is known.
.env.example carried ADMIN_HOSTS twice; phpdotenv keeps the first, so a fresh
checkout would have got the list without localhost and 404'd its own console.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reworked after a design consultation with Codex, which pushed back on my first
proposal in three useful ways.
Ownership and rights:
- vpn.manage split into vpn.view.all and vpn.manage.all. Seeing an access is
not managing it, and neither is holding its private key.
- Record-level rules live in VpnPeerPolicy, not in permissions: an access
belongs to a person, and ownership is what grants sight of it. Every operator
reaches the page, but sees only their own unless they may see all.
- Issuing an access is not self-service — it reaches the management network, so
it needs vpn.manage.all even for oneself.
- New Developer role: sees everything, manages nothing. Writing code does not
imply authority over other people's access.
- kind (staff|host|system) replaces a null user_id that had to mean two
different things at once.
Config storage, opt-in per access:
- Downloading is owner-only — explicitly NOT for view.all or manage.all. An
admin who needs access revokes this one and issues their own, which keeps the
record of who holds what honest.
- The password is asked on EVERY retrieval, rate-limited. Laravel's
password.confirm keeps a 15-minute session stamp, which would authorise
unlimited later downloads from an unattended browser.
- Stored under VPN_CONFIG_KEY, not APP_KEY: a leaked application key must not
also hand over the management network. Purged on revocation and when a staff
member is revoked — console taken away, tunnel left open was the worst case.
- The plaintext never enters a Livewire snapshot (the page polls every five
seconds); the component carries an opaque handle instead.
Also: copy button works without HTTPS again (navigator.clipboard only exists in
a secure context, so over http it silently did nothing), plus config download
as a file and a QR code for the mobile apps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The FK only nulled host_id, so the adopted peer stayed enabled-but-absent: shown
as "wird angewendet" forever, and re-enabling it would have put the deleted
host's key and address back on the hub. PurgeHost now tombstones that row, which
hands it to the same revocation machinery as everything else.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- A host's wg_pubkey may not be in vpn_peers yet, so the duplicate check passed
and an operator could create an access with it. ApplyVpnPeer would then run
wg set with a freshly allocated address, rewriting that host's allowed-ip and
cutting CluPilot's management tunnel to a live machine. Creation now rejects
keys already held by a host, checked under the allocation lock.
- A sync landing between addPeer() and the step storing wg_pubkey adopted the
peer as "unknown", and a later sync filled in host_id but left the name, so
the page showed that host as unknown forever. The name is now filled in when
the link becomes known — only for sync-adopted rows, so an access an operator
named keeps its name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ConfigureWireguard and RemoveWireguardPeer mutate the hub too, and were not
taking the lock, so a sync could still read the interface around them and
write stale state. Both now hold wireguard:hub, which is what makes the
read/mutation guarantee actually hold.
- The duplicate-key check ran before the allocation lock, so two concurrent
creations with the same key both passed it and the loser hit the unique index
as an unhandled 500. The check moved inside the lock, with the index kept as
a caught backstop for any writer that does not take it — reproduced in a test
by inserting a colliding row from within the allocation call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A sync could read the hub snapshot just before ApplyVpnPeer removed a peer and
purged its tombstone, then adopt what it had seen as a fresh enabled access —
restoring a revoked one. Both jobs now hold Cache::lock('wireguard:hub') around
read-decide-mutate, so a reconciliation can never straddle a removal. Retry
removals are dispatched after the lock is released, because the sync queue
driver runs them inline and they take the same lock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
If an add job was retried after the access had been revoked and its tombstone
purged, it found no row and trusted its own enabled=true payload — re-adding a
key that no longer had a row anyone could revoke it with. A missing row can only
mean the access is gone, so the captured payload may now remove but never
enable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two more from Codex:
- Addresses come from one subnet but live in two tables (hosts.wg_ip and
vpn_peers.allowed_ip), so a concurrent host onboarding and console creation
could each see the same address as free and both insert — neither unique
index sees the other. Creation now waits on Cache::lock('wireguard:allocate'),
the same lock ConfigureWireguard already holds while reserving a host address.
- mount() runs once, so revoking vpn.manage left an open tab polling every five
seconds and still receiving fresh peer state. Authorization now also runs on
hydration, which is what the poll goes through.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two lifecycle holes Codex found in the new VPN management:
1. Delete removed the row before the hub knew. If the removal job was delayed
or failed, SyncVpnPeers saw a peer it did not recognise and adopted it back
as a live access — silently restoring what an operator had just revoked, and
with no row left to revoke it again. Deletion is now a soft-delete
tombstone: the row survives until the hub confirms the peer is gone, the
sync retries the removal instead of adopting, and the tombstone is purged
only once the peer is really absent. It also keeps the key and the tunnel
address reserved meanwhile, so neither is handed out twice.
2. ApplyVpnPeer applied the state captured at dispatch. A retried block job
could therefore undo a later unblock, and nothing would repair it — the sync
only observes state, it never re-applies intent. The job now resolves the
current desired state from the row and treats its payload as a fallback for
the case where the row is already gone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The console can now create, block and remove VPN accesses, and shows who is
actually connected — traffic counters, source endpoint and last handshake.
Design notes:
- The web container has no wg0, so the page never talks to WireGuard. Live
state is copied in by SyncVpnPeers on the provisioning queue (the only worker
whose container owns the interface); the page polls the database and merely
nudges that job, throttled so many open tabs cannot flood the queue.
- enabled (operator intent) and present (observed on the hub) are stored
separately, so a sync can never quietly undo a block and drift stays visible
as "wird angewendet".
- The sync adopts peers the host pipeline registered, naming them after their
host — otherwise "who is on the VPN" would have blind spots.
- Keypairs are generated in PHP via libsodium rather than shelling out to
wg genkey (no interface in this container, and it keeps generation testable).
The private key is shown once and never stored.
- allocateIp() now also considers vpn_peers, which would otherwise be handed a
tunnel address a host already holds.
- vpn.manage is a new capability held by Owner/Admin only, and it hides the nav
entry too — a VPN access reaches the management network.
Verified end to end against the real hub, not just mocks: created an access in
the browser, connected with the generated config, watched the console flip to
"Verbunden" with real counters, then blocked it and confirmed the peer was
gone from wg0 and the client could no longer handshake.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DNS names are case-insensitive and Symfony's getHost() always returns lowercase,
so ADMIN_HOSTS=Admin.Example.com rejected every request and locked operators
out. Normalised in the config AND at the comparison, so the value is safe
whichever route it took in (env, cached config, runtime set).
operator()/admin() moved from RbacTest/HostManagementTest to tests/Pest.php:
they were only loaded when those files happened to be in the run, so a targeted
single-file run died on 'undefined function operator()'.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Admin actions post to /livewire/update, which a path-scoped guard skips — an
operator session could drive admin components through a PUBLIC hostname despite
ADMIN_HOSTS. The restriction is now registered as Livewire-persistent middleware
and listed on the admin route group, so Livewire re-applies it from the
component snapshot.
Proven by replaying a snapshot taken from the real rendered page: identical
payload -> 404 on a public host, 200 on the allowed host (positive control, so
the test cannot pass for the wrong reason).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Proxmox fleet and the operator console must never be publicly reachable. The
primary control is the reverse proxy, but nginx here is a catch-all
(server_name _), so /admin was served on EVERY hostname — a proxy
misconfiguration would expose it. ADMIN_HOSTS pins it; any other host gets 404
(not 403: a public domain must not disclose that a console exists).
Prepended to the group instead of the admin route group on purpose: route
middleware is reordered by Laravel's priority list, which runs first — a
guest would then be redirected to /login and learn the console is there. Covered
by a test for exactly that case. Empty ADMIN_HOSTS = unrestricted, so nobody is
locked out by upgrading.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- MONITORING_ATTEMPTS is capped by the run's max_attempts, so a large value can
no longer burn the retry budget and fail the very provisioning the degradation
exists to protect (covered by a new test at the budget boundary)
- /health is now dependency-free (200 in ~2ms with Kuma absent, verified: the
container healthcheck stays healthy); Kuma state is exclusively in /ready
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- attempt is 0-based, so compare one-based: ATTEMPTS=2 really means two tries
- bridge accepts add_monitor returning {monitorID}|{id}|int (v1.2.1 returns a
dict — re-verified: first create on a fresh Kuma is HTTP 200 with the id)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kuma has no write REST API (monitor CRUD is Socket.IO only), so docker/kuma-bridge
translates the exact generic REST contract HttpMonitoringClient already speaks —
no PHP client change needed. Opt-in compose profile 'monitoring'.
Verified end-to-end against a real Uptime Kuma 1.x:
create -> id, create again -> same id (idempotent), list, status, delete
and PHP MonitoringClient -> bridge -> Kuma round-trip.
The e2e run caught a real bug: add_monitor takes 'maxretries', not 'retries'.
Monitoring is now observability, not a delivery gate:
- RegisterMonitoring retries MONITORING_ATTEMPTS times on an outage, then
continues degraded with a visible 'info' event
- RunAcceptanceChecks no longer fails when monitoring isn't green (a fresh Kuma
monitor legitimately reports 'pending' — confirmed in the e2e run); it records
the gap instead. MONITORING_REQUIRED=true restores strict gating.
- docs/monitoring-uptime-kuma.md documents setup, failure behaviour, alternatives
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ResolvesCustomer trait replaces 5 duplicated customer() lookups; portal actions
now SAY why nothing happened (operator accounts have no Customer) instead of
returning silently — the actual cause of 'dead buttons', '0/0 seats' and
'settings not saving' when signed in as admin
- portal layout: explicit notice for a login without a linked customer
- /cloud: per-instance maintenance badge + window details; seat note 'owner = seat 1'
- maintenance form: sectioned card, hints, placeholder, styled datetime fields,
grouped host picker with per-datacenter select-all + selection counter
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- isOperator()/EnsureAdmin/broadcast fall back to is_admin so a legacy admin is
never locked out by a stale permission cache
- published modal: centered max-w-lg card (dynamic modalWidth class was purged
by Tailwind → full-width)
- datacenter edit moved to a modal (no row-height jump); location is now a
country dropdown (config/countries.php) instead of free text
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 5 operator roles (Owner/Admin/Support/Billing/Read-only) seeded via migration
with a capability catalogue; app checks capabilities via Gate, never role names
- every mutating admin action authorizes server-side (hosts/datacenters/customers/
impersonate/provisioning); is_admin reads migrated to console.view / isOperator()
- admin /settings: own account + Owner-only staff invite/role/revoke with
last-owner, self-role and customer-collision guards (transactional)
- sidebar 'zum Kundenportal' link replaced with Settings
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- seat limit follows the active/cancelling package, not a newer inactive record
- suspend/reactivate toggle refuses closed accounts (terminal); closed shown in list
- factory emails use a large unique numeric space (safeEmail pool could collide
late in a full suite run)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- seats table + model; Users page manages team members (owner auto-created)
- invite respects plan seat allowance + dedupe; role change + revoke guarded
so the last owner can never be removed or demoted
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- customers gain profile + branding + closed_at; instances gain cancellation
fields; branding resolver (NULL -> CluPilot defaults) snapshotted into the
provisioning run context
- cancel package: term-end, irreversible, typed-confirm modal (R5)
- close account: guarded (no active package), typed-confirm modal (R5)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- customers.user_id unique so impersonation/billing can't cross customers
- HostCreate validates datacenter is active (exists rule + active=1)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- migration seeds fsn/hel + any host-referenced code so existing installs
keep selectable datacenters after upgrade
- Billing::customer() resolves via user_id, email only as legacy fallback
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude
nexxo