fix: crash recovery after server reboot (#16) #43

Merged

lz merged 17 commits from fix/crash-recovery into main 2026-06-23 16:12:52 +02:00

Conversation 0 Commits 17 Files changed 23 +2191 -326

lz commented 2026-06-22 23:41:59 +02:00

Owner

Copy link

Closes #16.

Problem

After a host reboot, workers came back broken in two ways:

1. Crash-loop / lockout — the entrypoint waited 30s for Nexus to inject secrets into tmpfs, then exit 1. On a restart Nexus never re-injects, so with RestartPolicy: unless-stopped the container looped every ~30s and the operator couldn't even docker exec in.
2. Sessions lost, then reaped — the tmux server dies with the container; the old reconciler then hard-deleted the sessions rows, making clean re-attach impossible.

The hard constraint: the master key lives only in memory, so the vault is locked on a fresh Nexus boot — secrets can't be auto-re-derived.

Solution

Worker self-heals (no Nexus, no vault):

• The entrypoint persists the secret block to /workspace/.nexus/secrets.env (per-worker volume, chmod 600) after the clone, and re-sources it on restart. tmpfs stays as the first-spawn transport only (the clone needs /workspace empty).
• A provisioned workspace with no secrets boots degraded (skips git-creds + forgejo-mcp, which survive on the container layer) and never crash-loops — writing /run/nexus/degraded, surfaced as a degraded badge in the UI.
• .nexus/ is now excluded via the untracked /workspace/.git/info/exclude, so a fresh clone shows zero Nexus-induced changes (ensureNexusGitignore removed).

Nexus restores sessions (convergeSessions):

• The strike-based reaper is replaced by one convergeSessions() — the worktree is the source of truth: a missing window with an intact worktree is restored (relaunch claude resuming the pinned claude_session_id); a row is deleted only when the worktree is positively confirmed gone. An in-flight guard prevents duplicate spawns; "couldn't check" never deletes.
• Invoked on boot (restoreSessionsOnBoot), on startWorker, and on the health poll. Restored windows get NEXUS_URL re-injected (a degraded-boot tmux server lacks it).
• Conversation resume: each session pins a claude_session_id (migration 0012) and restores with --resume <uuid>.

How it was built

Designed via brainstorm → spec (docs/superpowers/specs/2026-06-22-crash-recovery-design.md) → plan (docs/superpowers/plans/2026-06-22-crash-recovery.md), implemented in 8 reviewed units (spec + code-quality review each; final whole-branch review passed).

Test status

• pnpm run typecheck → 0 errors; pnpm test → 541 passed; sh worker/secret-acquire.test.sh → PASS (first-spawn / restart / degraded / fatal / info-exclude). New tests: migration 0012, resume-flag composition, convergeSessions (restore / confirmed-orphan-delete / never-delete-on-uncertainty / creating-skip / workspace_shell).

Pre-merge checklist (cannot run here — no Docker/worker)

• Resume-flag spike (required before relying on conversation resume): in a real worker run claude --remote-control "<id>" --session-id <uuid>, take a turn, kill it, relaunch with --resume <uuid>, confirm the conversation returns AND remote control re-registers (watch upstream caveat #60790). If --remote-control ignores --resume, the fix is a one-function edit in resumeFlag (already isolated + flagged in-code).
• Smoke: docker restart <worker> → no crash-loop, window returns with prior conversation, git push works, forgejo-mcp present, no degraded badge. Delete /workspace/.nexus/secrets.env + restart → degraded badge, no loop. docker compose restart (reboot sim) → sessions auto-restore, notify-preview works (NEXUS_URL injected).

Notes

• Out of scope (deferred): re-injecting secrets on operator unlock (would also heal a container recreate, not just a restart).
• RestartPolicy: unless-stopped is unchanged — it now does the right thing once the entrypoint stops crash-looping.

Closes #16. ## Problem After a host reboot, workers came back broken in two ways: 1. **Crash-loop / lockout** — the entrypoint waited 30s for Nexus to inject secrets into tmpfs, then `exit 1`. On a restart Nexus never re-injects, so with `RestartPolicy: unless-stopped` the container looped every ~30s and the operator couldn't even `docker exec` in. 2. **Sessions lost, then reaped** — the tmux server dies with the container; the old reconciler then hard-deleted the `sessions` rows, making clean re-attach impossible. The hard constraint: the master key lives only in memory, so the vault is **locked** on a fresh Nexus boot — secrets can't be auto-re-derived. ## Solution **Worker self-heals (no Nexus, no vault):** - The entrypoint persists the secret block to `/workspace/.nexus/secrets.env` (per-worker volume, `chmod 600`) **after** the clone, and re-sources it on restart. tmpfs stays as the first-spawn transport only (the clone needs `/workspace` empty). - A provisioned workspace with no secrets boots **degraded** (skips git-creds + forgejo-mcp, which survive on the container layer) and never crash-loops — writing `/run/nexus/degraded`, surfaced as a `degraded` badge in the UI. - `.nexus/` is now excluded via the untracked `/workspace/.git/info/exclude`, so a fresh clone shows zero Nexus-induced changes (`ensureNexusGitignore` removed). **Nexus restores sessions (`convergeSessions`):** - The strike-based reaper is replaced by one `convergeSessions()` — the **worktree** is the source of truth: a missing window with an intact worktree is **restored** (relaunch `claude` resuming the pinned `claude_session_id`); a row is deleted **only** when the worktree is positively confirmed gone. An in-flight guard prevents duplicate spawns; "couldn't check" never deletes. - Invoked on boot (`restoreSessionsOnBoot`), on `startWorker`, and on the health poll. Restored windows get `NEXUS_URL` re-injected (a degraded-boot tmux server lacks it). - Conversation resume: each session pins a `claude_session_id` (migration 0012) and restores with `--resume <uuid>`. ## How it was built Designed via brainstorm → spec (`docs/superpowers/specs/2026-06-22-crash-recovery-design.md`) → plan (`docs/superpowers/plans/2026-06-22-crash-recovery.md`), implemented in 8 reviewed units (spec + code-quality review each; final whole-branch review passed). ## Test status - `pnpm run typecheck` → 0 errors; `pnpm test` → **541 passed**; `sh worker/secret-acquire.test.sh` → PASS (first-spawn / restart / degraded / fatal / info-exclude). New tests: migration 0012, resume-flag composition, `convergeSessions` (restore / confirmed-orphan-delete / never-delete-on-uncertainty / creating-skip / workspace_shell). ## Pre-merge checklist (cannot run here — no Docker/worker) - [ ] **Resume-flag spike** (required before relying on conversation resume): in a real worker run `claude --remote-control "<id>" --session-id <uuid>`, take a turn, kill it, relaunch with `--resume <uuid>`, confirm the conversation returns AND remote control re-registers (watch upstream caveat #60790). If `--remote-control` ignores `--resume`, the fix is a one-function edit in `resumeFlag` (already isolated + flagged in-code). - [ ] **Smoke**: `docker restart <worker>` → no crash-loop, window returns with prior conversation, `git push` works, forgejo-mcp present, no degraded badge. Delete `/workspace/.nexus/secrets.env` + restart → degraded badge, no loop. `docker compose restart` (reboot sim) → sessions auto-restore, `notify-preview` works (NEXUS_URL injected). ## Notes - Out of scope (deferred): re-injecting secrets on operator unlock (would also heal a container *recreate*, not just a restart). - `RestartPolicy: unless-stopped` is unchanged — it now does the right thing once the entrypoint stops crash-looping.

lz added 15 commits 2026-06-22 23:41:59 +02:00

docs(crash-recovery): design spec for issue #16 ... c7a1ef0cf1

Worker self-heals from a persisted secret cache and never crash-loops;
Nexus converges sessions (restore-or-orphan-delete) on boot/start/poll,
replacing the strike-based reaper. .nexus/ ignored via info/exclude.

docs(crash-recovery): add conversation resume + degraded-boot observability ... 5961ddd35f

Resume: pin sessions.claude_session_id, relaunch resume-capable, exact
remote-control flags settled by an implementation spike (docs unreliable).
Degraded: entrypoint marks /run/nexus/degraded, surfaced via worker health + UI.

docs(crash-recovery): implementation plan for issue #16 ... 26f593ce37

12 TDD tasks: entrypoint persist/degraded/info-exclude, claude_session_id
migration, resume-capable launch, convergeSessions (boot/start/poll),
degraded health+UI, drop ensureNexusGitignore, docs. Resume spike gates flags.

fix(worker): persist secrets + degraded boot so reboots don't crash-loop (#16) bba717d401

fix(worker): ignore .nexus/ via info/exclude so fresh clones stay clean (#16) 9210a2c4e1

feat(db): add sessions.claude_session_id for conversation resume (#16) fcadd78c85

feat(sessions): resume-capable launch + shared launchSessionWindow with pinned session id (#16) ee142d6e31

fix(sessions): don't --resume a legacy/empty session id; backfill the pin (#16) ... dfcb1b9b14

A legacy workspace_shell row has claude_session_id === '' (pre-migration).
openWorkspaceSession minted a fresh uuid for it but set resume=Boolean(existing),
emitting --resume <fresh-uuid> for a conversation that never existed, and the
respawn UPDATE didn't persist the minted id — so every respawn re-minted and the
pin was never stored. Gate resume on a real pinned id and backfill the id on the
existing-row UPDATE. Extract the shared resume-flag fragment into resumeFlag() so
the pending Task-0 spike changes one place. Adds two empty-id regression tests.

docs: crash-recovery — persisted secrets, info/exclude, convergeSessions, degraded boot (#16) da59dac27a

feat(sessions): convergeSessions restores windows, drops strike reaper (#16) 034433d5c9

feat(workers): health poll converges sessions + reports degraded boot (#16) c6038ee76e

feat(ui): degraded-boot badge on the workspace card (#16) fecb681ede

fix(sessions): in-flight guard on convergeSessions; cover creating + workspace_shell rows (#16) 55ac661105

feat(sessions): boot pass + startWorker converge (#16) ... 219f2aaf9e

- Add restoreSessionsOnBoot (nexus/src/lib/server/sessions/restore.ts):
  iterates all worker rows and calls convergeSessions per-worker; per-worker
  errors are non-fatal, mirroring restorePreviewsOnBoot.
- Wire restoreSessionsOnBoot into server.ts main(), immediately after
  restorePreviewsOnBoot, so sessions are restored on every Nexus boot.
- Extend startWorker to accept config and fire-and-forget convergeSessions
  after the container starts, so sessions restore promptly on operator
  start rather than waiting for the next 10s health poll.
- Update POST /api/workers/:id/start route to pass config to startWorker.

Claude-Session: https://claude.ai/code/session_012wTT6SMZA8roWXvuwdagcd

refactor(sessions): drop ensureNexusGitignore (covered by info/exclude) (#16) 5e7ce99bc2

lz added 2 commits 2026-06-23 15:50:00 +02:00

fix(nexus): unbreak Windows build (better-sqlite3 v12 + esbuild JS API) ... bccd91bcbe

Two Windows-only build failures:

- better-sqlite3 ^11.7.0 has no prebuilt binary for Node 24, so pnpm
  compiled from source and failed: V8 headers now require C++20, which
  VS 2019 Build Tools cannot target. Bump to ^12.11.1, which ships a
  prebuilt win32-x64/Node-24 binary -- no local compiler needed.

- The build:server script passed the $server esbuild alias as an inline
  CLI flag escaped for POSIX sh. Under pnpm's Windows shell the backslash
  survives, so esbuild rejected it ("Invalid alias name"). Move the
  bundling into scripts/build-server.mjs and use esbuild's JS API, where
  the alias is a plain string no shell touches. Identical behavior on
  Linux and Windows.

fix(nexus): copy scripts/ into the Docker build so CI build succeeds ... 829f160d3e

vite.config.ts imports ./scripts/gen-migrations.mjs, but the Dockerfile
copied src/static/migrations and the config files only -- never scripts/.
So `pnpm run build` failed in CI when Vite loaded its config
("Could not resolve ./scripts/gen-migrations.mjs"). Add COPY scripts.

This also covers scripts/build-server.mjs, which build:server now runs
directly.

lz merged commit bae52a0c42 into main 2026-06-23 16:12:52 +02:00

lz referenced this pull request from a commit 2026-06-23 16:12:53 +02:00

Merge pull request 'fix: crash recovery after server reboot (#16)' (#43) from fix/crash-recovery into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

wontfix

This won't be fixed

No labels

bug

duplicate

enhancement

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lz/agent-nexus!43

No description provided.