Hand container health to the daemon #188
No reviewers
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
lz/agent-nexus!188
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/daemon-container-health"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #184. Milestone 27. Cut from
mainat9ede718(after #180 and #185 merged).Workspace health stops being something Nexus infers from a
docker execon a 10s browser poll and becomes something the daemon reports.checkWorkerHealthreadsinspect().State.Health.Status; the boot grace period moves into the healthcheck's ownStartPeriod; and the four helpers that existed only because Nexus had to guess are deleted.This ships while the browser still drives the poll, so it is independent of #177 — it takes the largest deletion out of that PR and leaves it to be about ownership rather than about health.
What changed
workers/healthcheck.ts(new leaf):WORKER_HEALTHCHECK(CMD-SHELL tmux has-session -t nexus),hasHealthcheck, and two mapping functions. Every timing field is stated —Interval10s,Timeout5s,StartPeriod10 min,StartInterval2s,Retries3.provisionWorkersets the healthcheck in the create config.startWorkerinspects the container and, if it carries no healthcheck, replaces it — routed throughprovisionWorkerwith mounts fromworkers.mounts_json.claimForRecreate(guarded bynotBeingRemoved) claims the row before the remove.checkWorkerHealthmaps the daemon's vocabulary:healthy→ healthy,starting→ unknown,unhealthy→ unhealthy, no healthcheck → unknown with a reason.settleBootingPhase,BOOT_GRACE_MS, the tmux exec probe, three deadWorkerHealthDTOfields (containerState,tmuxRunning,recentLogs), and one duplicateexecProbe.ensureSessionTrustnow checks for zero-byte input (test -s) before moving the file.worker/entrypoint.shnow validates clone completion via.nexusinstead of just checking if/workspaceis populated.Two deliberate departures from the issue
1.
lastBootMarkeris kept. It answers "what is the entrypoint doing right now", whichHealth.Status: 'starting'cannot express over the 30-90s boot window. It feeds the provision detail rendered byProvisionProgress.svelte,RailHoverCard.svelte, and the Ctrl-K palette. Deleting it blanks all three during boot with nothing taking their place.2. A volume-safety fix in the recreate.
teardownContainerwas called by both success and failure paths with different semantics — success can delete the volume, but failure must keep it (since the row stays inerrorwith a Retry). The two paths now callteardownContainer(success, deletes volume) andremoveProvisionedContainer(failure, keeps volume).Boot latency
bootingused to clear on the first poll whosedocker execfound tmux — expected ~5s, worst case 10s. It now clears whenHealth.Status: 'healthy'is observed. WithStartInterval: 2sthe daemon observes tmux within 2s, so expected ~6s, worst case ~12s. On a daemon older than Docker 25 the field is ignored and probing falls back toInterval(10s) — bounded and never worse than the status quo.StartPeriodreplacesBOOT_GRACE_MSone-for-one: 10 minutes in which a failing probe does not count towardRetries.Behavior changes
Starting a workspace whose container predates this now makes one provider call (
listRepos) before the recreate to rebuild the authenticated clone URL — the same as Retry. It happens once per container. If the connection is unreachable, Start fails with the provider's message.Sessions survive the restart the way they survive any restart: the worktrees live on the named workspace volume, which the recreate does not touch, so
convergeSessionsrestores each window against the pinnedclaude_session_id.The clone guard fix
An earlier revision deferred this.
worker/entrypoint.shnow asks whether a previous boot completed a clone (by checking for.nexus) rather than just whether/workspaceis non-empty. The three arms:.nexus→ clear and re-clone (safe because no boot reached the line creating.nexus, so there are no worktrees)workspace_cloned()tests the same predicate, so a half-cloned volume refuses to boot instead of booting degraded over a workspace that never finished cloning.Verified in real containers: a bare repo and a real workspace volume, once per arm:
==> Cloning.git,README.md, commit present.nexus/worktrees/alpha==> Workspace already cloned, skipping.nexusintact.git+src/, no.nexus==> CloningThe negative control (old guard against the same debris) adopted it and created
.nexus, leaving the volume reading as provisioned forever with no valid clone — the exact bug this fixes.Review rounds
First round — six defects, all fixed with mutation-tested guards:
docker execjq on a zero-byte file exited 0, landing an empty/root/.claude.json. Addedtest -scheck before the move.healthy || undefined.booting).stopWorkernow clears the phase.stopped, which is invisible. Status is now written before the launch.lastBootMarkercomment accuracy: corrected explanations in three places + pinned by real-daemon test.inspect(),last_seen_atcomment accuracy, and identity-matched chip comparison.Second round — seven real defects, all fixed with proven guards:
~/.claude.jsonis container-layer state keyed by exact path, so the first recreate hadconvergeSessionsrestore sessions into a cwd claude no longer trusts. Fixed by asserting trust insidelaunchSessionWindow.healthysilently stopped restoring sessions for pre-#184 containers. Now gated on "not stillstarting".claimForRecreatehad no phase guard, so two Starts could each claim the same container. Now scoped to NULL phase.error, whose Retry 409s (old container holds the name). NowreleaseRecreateClaimputs the row back torunningso Start is the action.inspect()made "we could not ask the daemon" read as "this container can never answer". Now throws.healthReasonoutput reached no surface — onlyunhealthyrenders it.Workers.sveltenow chips it amber ("needs restart"), gated on NULL provision phase.workers.last_seen_at: never came up → volume goes; has come up → it stays.Testing & verification
Each gate run as its own invocation (not chained, not piped):
pnpm lintpnpm vitest runpnpm typecheckpnpm buildReal daemon testing (
NEXUS_DOCKER_IT=1against Docker 29.8.0 / API 1.56):WORKER_HEALTHCHECKround-trips through the daemon unchanged (includingStartInterval)hasHealthcheckmatches real inspect output (present, absent,Test: ['NONE'])State.Healthat all →unknown, neverhealthyStartPeriod, failing probe staysstarting,FailingStreak0StartPeriodusesStartInterval(≥2 probes in 7s vs 1 without)StartIntervalrather than refusing the create (API v1.43 tested)remove({ v: true })keeps named volumes, reaps anonymous onesCMD-SHELL tmux has-session -t nexuscontainer:starting→healthy~1s after tmux startsNegative controls — 31 mutations in the battery, 31 KILLED, re-verified after simplification:
cpbackup before and afterSimplification pass — four deletions proven not to lose guards:
resolveProvisionInputs+provisionContextFromRowunified, catching one omitted field immediatelyhasHealthcheckis now the single authority for "can this container report"hasEverRunrides the provisioning context, fixing an undefined stateThe pass found two real defects the gates missed: a
claudeVersionregression (pre-#184 containers silently stop reporting it) and zero coverage on the volume-deletion path (fixed with proven guards).Notes for merge
Worker image release is required for the entrypoint changes (clone guard +
test -s) to take effect. The healthcheck itself is set in the create config (live) and needs no image change. No socket-proxy endpoint is added —inspectis already covered.Sessions are restored the way they survive any restart: the worktrees live on the named volume, which the recreate does not touch.
All 13 guard tests remain, proven to fail when their guard is removed. All three simplifications are verified not to lose guards. Ready to merge.