lz/agent-nexus

Fork 0

fix(db): renumber api_tokens to 0019 so production can boot #131

Merged

lz merged 1 commit from fix/migration-0017-collision into main

2026-09-05 17:25:05 +02:00

lz commented

2026-09-05 17:10:28 +02:00

Owner

Production is down on main. It exits at boot with:

corrupted migrations: expected previously executed migration 0017_provision_detail
to be at index 16 but 0017_api_tokens was found in its place.

What happened

Two branches both took the number 0017:

0017_provision_detail — landed on main via #104, "migration 0017 ahead of the RC"
0017_api_tokens — landed on main 188 commits later via #123, from a branch cut earlier

Production was running a build from the window between those two merges, so its ledger has 0017_provision_detail at index 16 and has never seen 0017_api_tokens. When #123 merged, its migration inserted itself alphabetically ahead of one production had already executed. kysely requires the executed ledger to be a prefix of the sorted migration list, so it refuses to start.

Nothing is corrupted. The schema is intact; the migrator declines to run.

There is an irony worth recording: #104 existed specifically to claim 0017 early and avoid a collision. #123's branch had already claimed it.

The fix

Move 0017_api_tokens → 0019_api_tokens, after 0018_session_attention. Every migration has its own number again and kysely's ordering check stays strict.

Renumbering is safe here for one specific reason, checked rather than assumed: production has never executed 0017_api_tokens. Its ledger has no such row and the database has no api_tokens table. It is also the only live instance, so no database anywhere holds the old spelling.

That window is narrow, and worth stating plainly: once two databases have run the same number in different orders, no renaming can satisfy both — renaming to fix one trips a different, unrelaxable check on the other (previously executed migration ... is missing). This was recoverable only because nothing had run the loser of the collision.

Verification

Ran the real runMigrations against a copy of the production database:

applies 0018_session_attention, then 0019_api_tokens
resulting schema byte-identical to a fresh install's (full sqlite_master comparison)
all 4 workers and 5 sessions preserved
lint clean, typecheck clean (0 errors), full suite green 1501/1501

Guard

migrations/migration-numbers.test.ts fails on any duplicate migration number, so the next collision is caught in CI rather than at boot. Proven to fail by reintroducing the exact 0017 collision — it reports both filenames.

This is deliberately the weaker of two guards, and the one that needs no git context. The stronger rule — every migration a branch adds must sort after the newest migration already on the base branch — catches the whole family, including a stale branch merging late under any naming scheme. Measured against real history, it flags #123 at the merge that broke production and passes this PR. It needs the base ref in CI (fetch-depth: 0), so it is a follow-up rather than a change to the PR that has to merge to end the outage.

Note for whoever deploys

Merge, rebuild, redeploy. The first boot will apply 0018_session_attention and 0019_api_tokens to the existing database; no manual database work is needed.

Production is down on `main`. It exits at boot with: ``` corrupted migrations: expected previously executed migration 0017_provision_detail to be at index 16 but 0017_api_tokens was found in its place. ``` ## What happened Two branches both took the number `0017`: - `0017_provision_detail` — landed on main via #104, *"migration 0017 ahead of the RC"* - `0017_api_tokens` — landed on main 188 commits later via #123, from a branch cut earlier Production was running a build from the window between those two merges, so its ledger has `0017_provision_detail` at index 16 and has never seen `0017_api_tokens`. When #123 merged, its migration inserted itself *alphabetically ahead* of one production had already executed. kysely requires the executed ledger to be a prefix of the sorted migration list, so it refuses to start. Nothing is corrupted. The schema is intact; the migrator declines to run. There is an irony worth recording: #104 existed specifically to claim `0017` early and avoid a collision. #123's branch had already claimed it. ## The fix Move `0017_api_tokens` → `0019_api_tokens`, after `0018_session_attention`. Every migration has its own number again and kysely's ordering check stays strict. Renumbering is safe here for one specific reason, checked rather than assumed: **production has never executed `0017_api_tokens`.** Its ledger has no such row and the database has no `api_tokens` table. It is also the only live instance, so no database anywhere holds the old spelling. That window is narrow, and worth stating plainly: once two databases have run the same number in *different orders*, no renaming can satisfy both — renaming to fix one trips a different, unrelaxable check on the other (`previously executed migration ... is missing`). This was recoverable only because nothing had run the loser of the collision. ## Verification Ran the real `runMigrations` against a copy of the production database: - applies `0018_session_attention`, then `0019_api_tokens` - resulting schema **byte-identical** to a fresh install's (full `sqlite_master` comparison) - all 4 workers and 5 sessions preserved - `lint` clean, `typecheck` clean (0 errors), full suite green 1501/1501 ## Guard `migrations/migration-numbers.test.ts` fails on any duplicate migration number, so the next collision is caught in CI rather than at boot. Proven to fail by reintroducing the exact `0017` collision — it reports both filenames. This is deliberately the *weaker* of two guards, and the one that needs no git context. The stronger rule — every migration a branch adds must sort after the newest migration already on the base branch — catches the whole family, including a stale branch merging late under any naming scheme. Measured against real history, it flags #123 at the merge that broke production and passes this PR. It needs the base ref in CI (`fetch-depth: 0`), so it is a follow-up rather than a change to the PR that has to merge to end the outage. ## Note for whoever deploys Merge, rebuild, redeploy. The first boot will apply `0018_session_attention` and `0019_api_tokens` to the existing database; no manual database work is needed.

lz added 1 commit

2026-09-05 17:10:29 +02:00

fix(db): let the two 0017 migrations coexist so production can boot

ci / images (pull_request) Has been cancelled

Details

ci / nexus (pull_request) Has been cancelled

Details

df8c447a12

Production ran the issue-96 RC, so its ledger holds 0017_provision_detail at
index 16. main later merged 0017_api_tokens, which sorts alphabetically ahead
of it, and kysely's positional check refuses to start against any ledger that
is not a prefix of the sorted list.

Renaming cannot fix this. A database that ran api_tokens first and one that
ran provision_detail first are in genuinely different orders, so no single
naming is a prefix of both -- and renaming to satisfy one trips the separate
"previously executed migration is missing" check on the other, which
allowUnorderedMigrations does not relax. Verified against a copy of the
production database: it now applies 0017_api_tokens and 0018_session_attention
and lands on a schema byte-identical to a fresh install, rows intact.

The flag makes a number collision non-fatal, which also makes it invisible, so
migration-numbers.test.ts fails on any new one. The 0017 pair is grandfathered
and can never be renumbered: both spellings are in live ledgers.

lz force-pushed fix/migration-0017-collision from df8c447a12

ci / images (pull_request) Has been cancelled

Details

ci / nexus (pull_request) Has been cancelled

Details

to ab7feae3b6

ci / nexus (pull_request) Successful in 10m24s

Details

pr-image-cleanup / delete-pr-images (pull_request) Successful in 10s

Details

ci / nexus (push) Successful in 10m8s

Details

ci / images (pull_request) Successful in 10m31s

Details

ci / images (push) Successful in 10m59s

Details

2026-09-05 17:20:57 +02:00

Compare

lz changed title from ~~fix(db): let the two 0017 migrations coexist so production can boot~~ to fix(db): renumber api_tokens to 0019 so production can boot

2026-09-05 17:21:08 +02:00

lz merged commit ab7feae3b6 into main

2026-09-05 17:25:05 +02:00

lz deleted branch fix/migration-0017-collision

2026-09-05 17:25:06 +02:00