feat(fleet): server-owned, event-driven fleet status (#177a) #192
No reviewers
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
lz/agent-nexus!192
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/fleet-status"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes part of #177 (milestone 27). Targets
feat/daemon-container-health(#188), notmain— it consumeshasHealthcheck,workspaceHealthStateand the boot machinery that #188 introduces, none of which exist on main. Merge #188 first.What this does
Fleet status stops being a side effect of somebody having a browser open.
Before,
GET /api/workers/:id/healthwas the only thing keeping fleet state true: the browser polled it every 10s per workspace, and that handler was what ranconvergeSessions, wrote boot progress, and refreshed health. Navigate to/settingsand reconciliation stopped. Create a workspace with no browser open and it never leftbooting— the reason MCP could not create a usable workspace.nexus/src/lib/server/fleet/replaces it with a server-owned layer:store.ts— oneFleetStoreonglobalThisbehindSymbol.for('agent-nexus.fleet'). Its own slot rather than aSingletonsfield, becausefleet/imports fromworkers/andsessions/and the reverse edge is a cycle (AGENTS.md #31). CarrieswaitFor(predicate, timeoutMs), the in-process await #155/#156 need — foundation, deliberately without a caller in this PR.runner.ts—docker.getEvents()filtered by the daemon tonexus-managed=true, with reconnect backoff, a 10s probe of the workspaces that cannot announce their own change, and a 60s resync.intent.ts— Docker event →observe | forget | ignore, pure and daemon-free.observe.ts— the relocatedcheckWorkerHealthbody, now writing the store.reconcile.ts— re-derives the fleet from DB + Docker. AbsorbsrestoreSessionsOnBoot, which is deleted rather than left beside it: two boot paths that both converge is how the doubleclaude --resumethatconvergingWorkersguards against gets back in.GET /api/workersandGET /api/workers/:id/healthnow read the store. The health route became a pure read, so it is now inTOKEN_ROUTES(workspaces:read).EVENTS: 1added to the socket proxy — it was default-denied, and without it the stream 403s at runtime with every gate green, because the tests mock dockerode.fleet/socket-proxy.test.tsreads the compose file so that omission is red instead of silent.Two things measured that contradict the design spec
intentFormust be an ALLOWLIST. Docker emitsexec_create/exec_start/exec_dieas container events carrying the container's labels, and bothobserveWorkspace's probes and the daemon's own healthcheck are execs. The spec's "every action butdestroymeans re-inspect" therefore closed a loop: observe → exec → events → observe. WideningintentForreopens it.hooks.server.ts'sinitdoes fire in production.build/handler.jsawaitsserver.init()at module top level, so importing it fromsrc/server.tsrunsinitbefore that file'smain()— measured, withinit's log line ahead of the firstmigration applied. The spec's "Starting the loop — the hazard" section and fact #14's old parenthetical were both wrong. There is one call site, not two.Code review found seven defects; all seven are fixed
Five were silent by construction. Three change what the design says, and reviewers should read these rather than the spec:
convergeSessionshad been left with no trigger for the case it exists for: a session whoseclaudeexits closes a tmux window inside a container that stays healthy, and a healthy container emits no events at all (measured: zerohealth_statusin 45s of steady state, and zero non-exec events in a 59s window on a live daemon). The probe set also deliberately excluded every healthchecked workspace. So such a session was never restored — the browser poll used to do it. The resync restores fact #15's promise and also retries a boot reconcile that threw, which a healthy stream would otherwise never do. #177c'snexus-report-exitis the proper edge; when it lands the resync becomes redundancy rather than the mechanism.store.silent()—!reporting || state === 'starting', at 10s. Boot progress was otherwise dead: no events fire during the healthcheck'sStartPeriod, so the entrypoint's==>markers never reached the card. The set is empty once a workspace is up, which keeps that cadence off the steady-state path.unobservedis a newWorkspaceHealthState. The health route answeredunreportablefor a workspace nothing had observed — the stateWorkers.svelterenders as a "needs restart" chip. Every workspace carried that advice from boot, and permanently if the stream never opened.FleetStore.serialize). Two overlapping observes commit in completion order, so a slowstoplanded on top of thediebehind it and the card readrunninguntil the next signal. It also stops two observes exec'ing into one container at once.destroy()ed. Anerrorin that window is an unhandled'error'event, which Node rethrows and which takes the process down; aclosein it parks the connect loop on a promise nothing can resolve — no reconnect, nothing logged.Verification
Gates, each run as its own command (never chained — see AGENTS.md "Running the gates without lying to yourself"):
pnpm lintpnpm typecheckpnpm vitest runpnpm buildEvery guard was re-broken and proven to fail — seven mutations, seven kills, each file restored from a
cpbackup and diffed byte-identical afterwards. One of the new tests survived its mutation on the first pass (aPassThroughresolves a destroyed-stream case through an unrelated path); it was rewritten against a deterministic double and now dies correctly.On a real daemon, with a labelled healthchecked container and a matching worker row, running the built artifact (
node build/server.js):fleet reconciled (workspaces: 1),fleet: watching Docker events;exec_createflat at 5–7 per 10s, no growth. A/B with the server stopped: 30/60s healthcheck-only vs 37/60s running. The extra 7 is exactly 6 silent probes plus the one resync observe.Notes for review
WorkspaceEntry.containerId/lastSignalAt/lastSignalKindare written and not yet read — the shape #177b's SSE payload wants. Delete them if #177b does not use them.observeWorkspacestill costs one uncacheddocker execper observation for the degraded marker, whereprobeClaudeVersionbeside it is cached. Caching it keyed oncontainerId + State.StartedAt(never the id alone, which survives a stop/start) is the open optimisation.provision_phase, so therow ?? storehazard (fact #27) cannot arise here. It arises in #177b.