Server-owned, event-driven fleet status layer #177
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
lz/agent-nexus#177
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Tracks
docs/superpowers/specs/2026-09-17-fleet-status-layer-design.md.Nexus has no server-side reconciliation. The loop is a
setIntervalinroutes/+page.svelte— the home-page component — andcheckWorkerHealthhas exactly one caller, the health route it drives. So fleet state is only true while a browser sits on/.Navigate to
/settingsand it stops. With no browser at all,provision_phasenever leavesbooting, which means a workspace created or started over MCP is permanently unusable. That is what blocks the whole MCP lifecycle arc.This does not make the codebase smaller — an audit measured it. Justify it on correctness: server-owned reconciliation,
/api/workers/:id/healthbecoming side-effect-free and thereforeTOKEN_ROUTES-eligible for the first time, and MCP having something true to await.Shape
Docker's own
/eventsstream plus a containerHealthcheckset at create, feeding one process-wide store onglobalThis; SSE to the UI;fleet.waitFor()in-process for MCP. Reconcile is edge-triggered — on boot, on stream reconnect, and for the workspaces that structurally cannot report — never periodic.Full design in the spec. The parts most likely to be got wrong:
EVENTSis in the socket proxy's default-deny list.docker-compose.ymlnames it explicitly among the flags left at 0. WithoutEVENTS: 1the stream 403s at runtime with the build and suite green, because the tests mock dockerode.src/server.tsis the production entry andpnpm devnever evaluates it; SvelteKit'sinitruns in dev but fact #14 records it does not fire from the custom entry. Both, idempotently, or it runs in exactly one environment while looking correct in the other.Healthcheckis fixed at create,docker updatecannot add one, and nothing forces a recreate — so legacy containers never emithealth_statusandsettleBootingPhase,lastBootMarker,BOOT_GRACE_MSand the tmux probe all survive for them.startfires before the entrypoint's first line./run/nexus/degradedis written atentrypoint.sh:84andtmux new-sessionis line 255 of 256. Degraded state and the claude version must hang offhealth_status: healthy, or an exec onstartreads "not degraded" permanently.provision_phaseassembly isrow ?? store, never the reverse. Backwards, a Remove-during-pull renders as still pulling andisRunningreturns true for a container that is gone — the incident fact #27 was written about.EventSourceexposes no status code — a 401 arrives asonerror— and the stream never passes throughcall(), sounlock-redirect.tsdoes not cover it. Thesession-statspoll is what detects expiry today (fact #25).Depends on
The declaration collapse, and the two bugs below, all of which are strictly cheaper before this than after.
Done when
A workspace created or started with no browser open reaches
readyon its own,convergeSessionsis driven by Nexus rather than by a client, and/api/workers/:id/healthis inTOKEN_ROUTES.Unblocked once #188 (issue #184) merges — and it changes what this issue has to build.
#184 shipped as PR #188. What #177 inherits:
Put the discriminator on the stream, not Docker's raw string.
workspaceHealthState()(workers/healthcheck.ts) maps the daemon's open-ended status to a closed set —healthy | unhealthy | starting | unreportable— andWorkerHealthDTOnow carries it asstatealongsidehealth. It exists because four consumers each re-read the raw value with their own predicate and had drifted: the version probe kept=== 'healthy'while convergence was widened, so a pre-healthcheck container silently stopped reportingclaudeVersion. A fifth consumer on an event stream would have been the next one to drift.The degraded probe is the cache this issue should build, and #184 deliberately did not. It is still one
docker execper poll per workspace — measured, ~21 Docker round trips/min/workspace — for a marker written once atentrypoint.sh:84and never removed. It was left alone because this issue moves it ontohealth_status: healthy, where it is computed once; building a cache first would mean building it and then deleting it. When you do: key it oncontainerId + State.StartedAt, never on the container id alone — the id survives a stop/start, so it would carry a stalefalseacross a boot that went degraded.Convergence is gated on
healthy || unreportable, never on!== 'starting'.convergeSessionsopens withwaitForTmux(container, 60_000), which execs every 250ms for a full minute. Converging a workspace whose tmux is known dead burns ~4 execs/sec against it indefinitely — the in-flight guard only stops the passes overlapping, not recurring. Theunreportablearm is load-bearing: gating onhealthyalone silently stops restoring a legacy container's sessions, which is the promise AGENTS.md fact #15 makes.Already deleted, so the spec's residual-prober section is moot:
settleBootingPhase,BOOT_GRACE_MS, the tmuxexecProbeand the log-marker boot inference.lastBootMarkerwas deliberately KEPT — the daemon saysstartingfor the whole 30-90s window and has no opinion on the stage, and that marker feedsprovision_detailon three surfaces.restoreSessionsOnBootis still present and is this issue's to delete (item 5 of the roadmap).A worker image release is owed before item 7. #188 changed
worker/entrypoint.sh(the clone guard, andtest -sbefore the jqmv), and the entrypoint is baked into the image —COPY entrypoint.shatDockerfile:61,ENTRYPOINTat:83. A container recreate does not pick it up: it builds a new container from the same image tag.nexus-report-exitneeds a release too, so release once for both, and check whether #183 is close enough to join.Two things worth not rediscovering:
last_seen_atis not a clone proxy.finishProvisioningwrites it right aftercontainer.start(), before the entrypoint clones, so it means "a container for this row started once". #188 carries it asProvisionContext.hasEverRunand the entrypoint now does the real check via/workspace/.nexus.EVENTS: 1still has to be added todocker-compose.yml, ordocker.getEvents()403s at runtime while the suite stays green — the tests mock dockerode.