Hand container health to the daemon #188

Merged
lz merged 14 commits from feat/daemon-container-health into main 2026-09-22 11:18:50 +02:00
Owner

Closes #184. Milestone 27. Cut from main at 9ede718 (after #180 and #185 merged).

Workspace health stops being something Nexus infers from a docker exec on a 10s browser poll and becomes something the daemon reports. checkWorkerHealth reads inspect().State.Health.Status; the boot grace period moves into the healthcheck's own StartPeriod; and the four helpers that existed only because Nexus had to guess are deleted.

This ships while the browser still drives the poll, so it is independent of #177 — it takes the largest deletion out of that PR and leaves it to be about ownership rather than about health.

What changed

  • workers/healthcheck.ts (new leaf): WORKER_HEALTHCHECK (CMD-SHELL tmux has-session -t nexus), hasHealthcheck, and two mapping functions. Every timing field is stated — Interval 10s, Timeout 5s, StartPeriod 10 min, StartInterval 2s, Retries 3.
  • provisionWorker sets the healthcheck in the create config.
  • startWorker inspects the container and, if it carries no healthcheck, replaces it — routed through provisionWorker with mounts from workers.mounts_json. claimForRecreate (guarded by notBeingRemoved) claims the row before the remove.
  • checkWorkerHealth maps the daemon's vocabulary: healthy → healthy, starting → unknown, unhealthy → unhealthy, no healthcheck → unknown with a reason.
  • Deleted: settleBootingPhase, BOOT_GRACE_MS, the tmux exec probe, three dead WorkerHealthDTO fields (containerState, tmuxRunning, recentLogs), and one duplicate execProbe.
  • ensureSessionTrust now checks for zero-byte input (test -s) before moving the file.
  • worker/entrypoint.sh now validates clone completion via .nexus instead of just checking if /workspace is populated.

Two deliberate departures from the issue

1. lastBootMarker is kept. It answers "what is the entrypoint doing right now", which Health.Status: 'starting' cannot express over the 30-90s boot window. It feeds the provision detail rendered by ProvisionProgress.svelte, RailHoverCard.svelte, and the Ctrl-K palette. Deleting it blanks all three during boot with nothing taking their place.

2. A volume-safety fix in the recreate. teardownContainer was called by both success and failure paths with different semantics — success can delete the volume, but failure must keep it (since the row stays in error with a Retry). The two paths now call teardownContainer (success, deletes volume) and removeProvisionedContainer (failure, keeps volume).

Boot latency

booting used to clear on the first poll whose docker exec found tmux — expected ~5s, worst case 10s. It now clears when Health.Status: 'healthy' is observed. With StartInterval: 2s the daemon observes tmux within 2s, so expected ~6s, worst case ~12s. On a daemon older than Docker 25 the field is ignored and probing falls back to Interval (10s) — bounded and never worse than the status quo.

StartPeriod replaces BOOT_GRACE_MS one-for-one: 10 minutes in which a failing probe does not count toward Retries.

Behavior changes

Starting a workspace whose container predates this now makes one provider call (listRepos) before the recreate to rebuild the authenticated clone URL — the same as Retry. It happens once per container. If the connection is unreachable, Start fails with the provider's message.

Sessions survive the restart the way they survive any restart: the worktrees live on the named workspace volume, which the recreate does not touch, so convergeSessions restores each window against the pinned claude_session_id.

The clone guard fix

An earlier revision deferred this. worker/entrypoint.sh now asks whether a previous boot completed a clone (by checking for .nexus) rather than just whether /workspace is non-empty. The three arms:

  • Present → skip
  • Empty → clone
  • Populated without .nexus → clear and re-clone (safe because no boot reached the line creating .nexus, so there are no worktrees)

workspace_cloned() tests the same predicate, so a half-cloned volume refuses to boot instead of booting degraded over a workspace that never finished cloning.

Verified in real containers: a bare repo and a real workspace volume, once per arm:

pre-seeded /workspace entrypoint result
empty ==> Cloning .git, README.md, commit present
.nexus/worktrees/alpha ==> Workspace already cloned, skipping no clone, .nexus intact
.git + src/, no .nexus discarding unfinished clone, then ==> Cloning debris gone, clean clone

The negative control (old guard against the same debris) adopted it and created .nexus, leaving the volume reading as provisioned forever with no valid clone — the exact bug this fixes.

This needs a worker image release. The entrypoint.sh change is baked into the image layer and is not bind-mounted. The test -s fix to the jq is in the Nexus TS copy and ships live immediately.

Review rounds

First round — six defects, all fixed with mutation-tested guards:

  1. A failed docker exec jq on a zero-byte file exited 0, landing an empty /root/.claude.json. Added test -s check before the move.
  2. Widening the converge gate cost ~4 execs/sec per unhealthy workspace. Narrowed to healthy || undefined.
  3. A legacy workspace stopped mid-boot was permanently unstartable (no clear of booting). stopWorker now clears the phase.
  4. Failed restart left the row stopped, which is invisible. Status is now written before the launch.
  5. lastBootMarker comment accuracy: corrected explanations in three places + pinned by real-daemon test.
  6. Smaller fixes: 404 handling in inspect(), last_seen_at comment accuracy, and identity-matched chip comparison.

Second round — seven real defects, all fixed with proven guards:

  1. The recreate destroyed per-worktree workspace trust. ~/.claude.json is container-layer state keyed by exact path, so the first recreate had convergeSessions restore sessions into a cwd claude no longer trusts. Fixed by asserting trust inside launchSessionWindow.
  2. Gating convergence on healthy silently stopped restoring sessions for pre-#184 containers. Now gated on "not still starting".
  3. claimForRecreate had no phase guard, so two Starts could each claim the same container. Now scoped to NULL phase.
  4. A failed remove parked the row in error, whose Retry 409s (old container holds the name). Now releaseRecreateClaim puts the row back to running so Start is the action.
  5. A swallowed inspect() made "we could not ask the daemon" read as "this container can never answer". Now throws.
  6. healthReason output reached no surface — only unhealthy renders it. Workers.svelte now chips it amber ("needs restart"), gated on NULL provision phase.
  7. The volume rule was too blunt — a first spawn dying mid-clone left a partial tree. Now keyed on workers.last_seen_at: never came up → volume goes; has come up → it stays.

Testing & verification

Each gate run as its own invocation (not chained, not piped):

gate result
pnpm lint clean
pnpm vitest run 206 files, 2293 passed / 10 skipped, 0 failed
pnpm typecheck 0 errors
pnpm build clean

Real daemon testing (NEXUS_DOCKER_IT=1 against Docker 29.8.0 / API 1.56):

  • WORKER_HEALTHCHECK round-trips through the daemon unchanged (including StartInterval)
  • hasHealthcheck matches real inspect output (present, absent, Test: ['NONE'])
  • Container with no healthcheck has no State.Health at all → unknown, never healthy
  • Inside StartPeriod, failing probe stays starting, FailingStreak 0
  • Probing during StartPeriod uses StartInterval (≥2 probes in 7s vs 1 without)
  • An older daemon drops StartInterval rather than refusing the create (API v1.43 tested)
  • remove({ v: true }) keeps named volumes, reaps anonymous ones
  • Real CMD-SHELL tmux has-session -t nexus container: starting → healthy ~1s after tmux starts

Negative controls — 31 mutations in the battery, 31 KILLED, re-verified after simplification:

  • Each guard was re-broken and the named test demanded to fail
  • Each mutation diffed against a cp backup before and after
  • Covers: healthcheck fields, health status mapping, recreate logic, claim guards, volume retention, trust assertion, clone guard (3 more mutations, all killed)

Simplification pass — four deletions proven not to lose guards:

  • Docker's open-ended health status mapped to a closed set once, at the boundary (eliminates drifted predicates)
  • resolveProvisionInputs + provisionContextFromRow unified, catching one omitted field immediately
  • hasHealthcheck is now the single authority for "can this container report"
  • hasEverRun rides the provisioning context, fixing an undefined state

The pass found two real defects the gates missed: a claudeVersion regression (pre-#184 containers silently stop reporting it) and zero coverage on the volume-deletion path (fixed with proven guards).

Notes for merge

Worker image release is required for the entrypoint changes (clone guard + test -s) to take effect. The healthcheck itself is set in the create config (live) and needs no image change. No socket-proxy endpoint is added — inspect is already covered.

Sessions are restored the way they survive any restart: the worktrees live on the named volume, which the recreate does not touch.

All 13 guard tests remain, proven to fail when their guard is removed. All three simplifications are verified not to lose guards. Ready to merge.

Closes #184. Milestone 27. Cut from `main` at `9ede718` (after #180 and #185 merged). Workspace health stops being something Nexus infers from a `docker exec` on a 10s browser poll and becomes something the daemon reports. `checkWorkerHealth` reads `inspect().State.Health.Status`; the boot grace period moves into the healthcheck's own `StartPeriod`; and the four helpers that existed only because Nexus had to guess are deleted. This ships while the browser still drives the poll, so it is independent of #177 — it takes the largest deletion out of that PR and leaves it to be about *ownership* rather than about health. ## What changed - **`workers/healthcheck.ts`** (new leaf): `WORKER_HEALTHCHECK` (`CMD-SHELL tmux has-session -t nexus`), `hasHealthcheck`, and two mapping functions. Every timing field is stated — `Interval` 10s, `Timeout` 5s, `StartPeriod` 10 min, `StartInterval` 2s, `Retries` 3. - **`provisionWorker`** sets the healthcheck in the create config. - **`startWorker`** inspects the container and, if it carries no healthcheck, replaces it — routed through `provisionWorker` with mounts from `workers.mounts_json`. `claimForRecreate` (guarded by `notBeingRemoved`) claims the row before the remove. - **`checkWorkerHealth`** maps the daemon's vocabulary: `healthy` → healthy, `starting` → unknown, `unhealthy` → unhealthy, no healthcheck → unknown with a reason. - **Deleted:** `settleBootingPhase`, `BOOT_GRACE_MS`, the tmux exec probe, three dead `WorkerHealthDTO` fields (`containerState`, `tmuxRunning`, `recentLogs`), and one duplicate `execProbe`. - **`ensureSessionTrust`** now checks for zero-byte input (`test -s`) before moving the file. - **`worker/entrypoint.sh`** now validates clone completion via `.nexus` instead of just checking if `/workspace` is populated. ## Two deliberate departures from the issue **1. `lastBootMarker` is kept.** It answers "what is the entrypoint doing right now", which `Health.Status: 'starting'` cannot express over the 30-90s boot window. It feeds the provision detail rendered by `ProvisionProgress.svelte`, `RailHoverCard.svelte`, and the Ctrl-K palette. Deleting it blanks all three during boot with nothing taking their place. **2. A volume-safety fix in the recreate.** `teardownContainer` was called by both success and failure paths with different semantics — success can delete the volume, but failure must keep it (since the row stays in `error` with a Retry). The two paths now call `teardownContainer` (success, deletes volume) and `removeProvisionedContainer` (failure, keeps volume). ## Boot latency `booting` used to clear on the first poll whose `docker exec` found tmux — expected ~5s, worst case 10s. It now clears when `Health.Status: 'healthy'` is observed. With `StartInterval: 2s` the daemon observes tmux within 2s, so expected ~6s, worst case ~12s. On a daemon older than Docker 25 the field is ignored and probing falls back to `Interval` (10s) — bounded and never worse than the status quo. `StartPeriod` replaces `BOOT_GRACE_MS` one-for-one: 10 minutes in which a failing probe does not count toward `Retries`. ## Behavior changes Starting a workspace whose container predates this now makes one provider call (`listRepos`) before the recreate to rebuild the authenticated clone URL — the same as Retry. It happens once per container. If the connection is unreachable, Start fails with the provider's message. Sessions survive the restart the way they survive any restart: the worktrees live on the named workspace volume, which the recreate does not touch, so `convergeSessions` restores each window against the pinned `claude_session_id`. ## The clone guard fix An earlier revision deferred this. `worker/entrypoint.sh` now asks whether a previous boot **completed** a clone (by checking for `.nexus`) rather than just whether `/workspace` is non-empty. The three arms: - Present → skip - Empty → clone - Populated without `.nexus` → clear and re-clone (safe because no boot reached the line creating `.nexus`, so there are no worktrees) `workspace_cloned()` tests the same predicate, so a half-cloned volume refuses to boot instead of booting degraded over a workspace that never finished cloning. **Verified in real containers:** a bare repo and a real workspace volume, once per arm: | pre-seeded /workspace | entrypoint | result | |---|---|---| | empty | `==> Cloning` | `.git`, `README.md`, commit present | | `.nexus/worktrees/alpha` | `==> Workspace already cloned, skipping` | no clone, `.nexus` intact | | `.git` + `src/`, no `.nexus` | discarding unfinished clone, then `==> Cloning` | debris gone, clean clone | The negative control (old guard against the same debris) adopted it and created `.nexus`, leaving the volume reading as provisioned forever with no valid clone — the exact bug this fixes. > **This needs a worker image release.** The `entrypoint.sh` change is baked into the image layer and is not bind-mounted. The `test -s` fix to the jq is in the Nexus TS copy and ships live immediately. ## Review rounds **First round — six defects, all fixed with mutation-tested guards:** 1. A failed `docker exec` jq on a zero-byte file exited 0, landing an empty `/root/.claude.json`. Added `test -s` check before the move. 2. Widening the converge gate cost ~4 execs/sec per unhealthy workspace. Narrowed to `healthy || undefined`. 3. A legacy workspace stopped mid-boot was permanently unstartable (no clear of `booting`). `stopWorker` now clears the phase. 4. Failed restart left the row `stopped`, which is invisible. Status is now written before the launch. 5. `lastBootMarker` comment accuracy: corrected explanations in three places + pinned by real-daemon test. 6. Smaller fixes: 404 handling in `inspect()`, `last_seen_at` comment accuracy, and identity-matched chip comparison. **Second round — seven real defects, all fixed with proven guards:** 1. The recreate destroyed per-worktree workspace trust. `~/.claude.json` is container-layer state keyed by exact path, so the first recreate had `convergeSessions` restore sessions into a cwd claude no longer trusts. Fixed by asserting trust inside `launchSessionWindow`. 2. Gating convergence on `healthy` silently stopped restoring sessions for pre-#184 containers. Now gated on "not still `starting`". 3. `claimForRecreate` had no phase guard, so two Starts could each claim the same container. Now scoped to NULL phase. 4. A failed remove parked the row in `error`, whose Retry 409s (old container holds the name). Now `releaseRecreateClaim` puts the row back to `running` so Start is the action. 5. A swallowed `inspect()` made "we could not ask the daemon" read as "this container can never answer". Now throws. 6. `healthReason` output reached no surface — only `unhealthy` renders it. `Workers.svelte` now chips it amber ("needs restart"), gated on NULL provision phase. 7. The volume rule was too blunt — a first spawn dying mid-clone left a partial tree. Now keyed on `workers.last_seen_at`: never came up → volume goes; has come up → it stays. ## Testing & verification Each gate run as its own invocation (not chained, not piped): | gate | result | |---|---| | `pnpm lint` | clean | | `pnpm vitest run` | 206 files, 2293 passed / 10 skipped, 0 failed | | `pnpm typecheck` | 0 errors | | `pnpm build` | clean | **Real daemon testing** (`NEXUS_DOCKER_IT=1` against Docker 29.8.0 / API 1.56): - `WORKER_HEALTHCHECK` round-trips through the daemon unchanged (including `StartInterval`) - `hasHealthcheck` matches real inspect output (present, absent, `Test: ['NONE']`) - Container with no healthcheck has **no** `State.Health` at all → `unknown`, never `healthy` - Inside `StartPeriod`, failing probe stays `starting`, `FailingStreak` 0 - Probing during `StartPeriod` uses `StartInterval` (≥2 probes in 7s vs 1 without) - An **older daemon drops `StartInterval`** rather than refusing the create (API v1.43 tested) - `remove({ v: true })` keeps named volumes, reaps anonymous ones - Real `CMD-SHELL tmux has-session -t nexus` container: `starting` → `healthy` ~1s after tmux starts **Negative controls** — 31 mutations in the battery, 31 KILLED, re-verified after simplification: - Each guard was re-broken and the named test demanded to fail - Each mutation diffed against a `cp` backup before and after - Covers: healthcheck fields, health status mapping, recreate logic, claim guards, volume retention, trust assertion, clone guard (3 more mutations, all killed) **Simplification pass** — four deletions proven not to lose guards: - Docker's open-ended health status mapped to a closed set once, at the boundary (eliminates drifted predicates) - `resolveProvisionInputs` + `provisionContextFromRow` unified, catching one omitted field immediately - `hasHealthcheck` is now the single authority for "can this container report" - `hasEverRun` rides the provisioning context, fixing an undefined state The pass found two real defects the gates missed: a `claudeVersion` regression (pre-#184 containers silently stop reporting it) and zero coverage on the volume-deletion path (fixed with proven guards). ## Notes for merge **Worker image release is required** for the entrypoint changes (clone guard + `test -s`) to take effect. The healthcheck itself is set in the create config (live) and needs no image change. No socket-proxy endpoint is added — `inspect` is already covered. Sessions are restored the way they survive any restart: the worktrees live on the named volume, which the recreate does not touch. All 13 guard tests remain, proven to fail when their guard is removed. All three simplifications are verified not to lose guards. Ready to merge.
lz added this to the MCP support (#140) milestone 2026-09-21 20:04:09 +02:00
lz added 9 commits 2026-09-21 20:04:09 +02:00
lz added 2 commits 2026-09-21 20:39:07 +02:00
lz added 2 commits 2026-09-21 22:02:55 +02:00
Merge main (#189 test unification) into feat/daemon-container-health
All checks were successful
ci / nexus (pull_request) Successful in 7m23s
ci / images (pull_request) Successful in 26m16s
pr-image-cleanup / delete-pr-images (pull_request) Successful in 14s
e24846844f
lz merged commit efccf87afe into main 2026-09-22 11:18:50 +02:00
lz deleted branch feat/daemon-container-health 2026-09-22 11:18:51 +02:00
Sign in to join this conversation.
No reviewers
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
lz/agent-nexus!188
No description provided.