feat(fleet): server-owned, event-driven fleet status (#177a) #192

Merged
lz merged 25 commits from feat/fleet-status into main 2026-09-23 10:36:49 +02:00
Owner

Closes part of #177 (milestone 27). Targets feat/daemon-container-health (#188), not main — it consumes hasHealthcheck, workspaceHealthState and the boot machinery that #188 introduces, none of which exist on main. Merge #188 first.

What this does

Fleet status stops being a side effect of somebody having a browser open.

Before, GET /api/workers/:id/health was the only thing keeping fleet state true: the browser polled it every 10s per workspace, and that handler was what ran convergeSessions, wrote boot progress, and refreshed health. Navigate to /settings and reconciliation stopped. Create a workspace with no browser open and it never left booting — the reason MCP could not create a usable workspace.

nexus/src/lib/server/fleet/ replaces it with a server-owned layer:

  • store.ts — one FleetStore on globalThis behind Symbol.for('agent-nexus.fleet'). Its own slot rather than a Singletons field, because fleet/ imports from workers/ and sessions/ and the reverse edge is a cycle (AGENTS.md #31). Carries waitFor(predicate, timeoutMs), the in-process await #155/#156 need — foundation, deliberately without a caller in this PR.
  • runner.ts — docker.getEvents() filtered by the daemon to nexus-managed=true, with reconnect backoff, a 10s probe of the workspaces that cannot announce their own change, and a 60s resync.
  • intent.ts — Docker event → observe | forget | ignore, pure and daemon-free.
  • observe.ts — the relocated checkWorkerHealth body, now writing the store.
  • reconcile.ts — re-derives the fleet from DB + Docker. Absorbs restoreSessionsOnBoot, which is deleted rather than left beside it: two boot paths that both converge is how the double claude --resume that convergingWorkers guards against gets back in.

GET /api/workers and GET /api/workers/:id/health now read the store. The health route became a pure read, so it is now in TOKEN_ROUTES (workspaces:read). EVENTS: 1 added to the socket proxy — it was default-denied, and without it the stream 403s at runtime with every gate green, because the tests mock dockerode. fleet/socket-proxy.test.ts reads the compose file so that omission is red instead of silent.

Two things measured that contradict the design spec

  • intentFor must be an ALLOWLIST. Docker emits exec_create/exec_start/exec_die as container events carrying the container's labels, and both observeWorkspace's probes and the daemon's own healthcheck are execs. The spec's "every action but destroy means re-inspect" therefore closed a loop: observe → exec → events → observe. Widening intentFor reopens it.
  • hooks.server.ts's init does fire in production. build/handler.js awaits server.init() at module top level, so importing it from src/server.ts runs init before that file's main() — measured, with init's log line ahead of the first migration applied. The spec's "Starting the loop — the hazard" section and fact #14's old parenthetical were both wrong. There is one call site, not two.

Code review found seven defects; all seven are fixed

Five were silent by construction. Three change what the design says, and reviewers should read these rather than the spec:

  1. The watch is paired with a 60s resync, and the issue's "never periodically" does not mean what it looks like — it is about the 10s fleet poll, which is gone. convergeSessions had been left with no trigger for the case it exists for: a session whose claude exits closes a tmux window inside a container that stays healthy, and a healthy container emits no events at all (measured: zero health_status in 45s of steady state, and zero non-exec events in a 59s window on a live daemon). The probe set also deliberately excluded every healthchecked workspace. So such a session was never restored — the browser poll used to do it. The resync restores fact #15's promise and also retries a boot reconcile that threw, which a healthy stream would otherwise never do. #177c's nexus-report-exit is the proper edge; when it lands the resync becomes redundancy rather than the mechanism.
  2. The probe set is store.silent() — !reporting || state === 'starting', at 10s. Boot progress was otherwise dead: no events fire during the healthcheck's StartPeriod, so the entrypoint's ==> markers never reached the card. The set is empty once a workspace is up, which keeps that cadence off the steady-state path.
  3. unobserved is a new WorkspaceHealthState. The health route answered unreportable for a workspace nothing had observed — the state Workers.svelte renders as a "needs restart" chip. Every workspace carried that advice from boot, and permanently if the stream never opened.
  4. Observations are serialised per workspace (FleetStore.serialize). Two overlapping observes commit in completion order, so a slow stop landed on top of the die behind it and the card read running until the next signal. It also stops two observes exec'ing into one container at once.
  5. Stream listeners are attached before the connect reconcile is awaited, and the abandoned stream is destroy()ed. An error in that window is an unhandled 'error' event, which Node rethrows and which takes the process down; a close in it parks the connect loop on a promise nothing can resolve — no reconnect, nothing logged.

Verification

Gates, each run as its own command (never chained — see AGENTS.md "Running the gates without lying to yourself"):

gate result
pnpm lint clean
pnpm typecheck 0 errors / 5199 files
pnpm vitest run 2353 passed / 10 skipped
pnpm build exit 0

Every guard was re-broken and proven to fail — seven mutations, seven kills, each file restored from a cp backup and diffed byte-identical afterwards. One of the new tests survived its mutation on the first pass (a PassThrough resolves a destroyed-stream case through an unrelated path); it was rewritten against a deterministic double and now dies correctly.

On a real daemon, with a labelled healthchecked container and a matching worker row, running the built artifact (node build/server.js):

  • stream opens, fleet reconciled (workspaces: 1), fleet: watching Docker events;
  • resync ticks at +59.9s and +60.0s across three reconciles;
  • zero non-exec events in 59s from a steady container — finding 1's premise, re-measured;
  • exec_create flat at 5–7 per 10s, no growth. A/B with the server stopped: 30/60s healthcheck-only vs 37/60s running. The extra 7 is exactly 6 silent probes plus the one resync observe.

Notes for review

  • WorkspaceEntry.containerId / lastSignalAt / lastSignalKind are written and not yet read — the shape #177b's SSE payload wants. Delete them if #177b does not use them.
  • observeWorkspace still costs one uncached docker exec per observation for the degraded marker, where probeClaudeVersion beside it is cached. Caching it keyed on containerId + State.StartedAt (never the id alone, which survives a stop/start) is the open optimisation.
  • The store deliberately holds no provision_phase, so the row ?? store hazard (fact #27) cannot arise here. It arises in #177b.
  • AGENTS.md fact #42 documents the layer; #5, #14, #15, #24, #40 were amended because this change made their premises false.
Closes part of #177 (milestone 27). **Targets `feat/daemon-container-health` (#188), not `main`** — it consumes `hasHealthcheck`, `workspaceHealthState` and the boot machinery that #188 introduces, none of which exist on main. Merge #188 first. ## What this does Fleet status stops being a side effect of somebody having a browser open. Before, `GET /api/workers/:id/health` was the only thing keeping fleet state true: the browser polled it every 10s per workspace, and that handler was what ran `convergeSessions`, wrote boot progress, and refreshed health. Navigate to `/settings` and reconciliation stopped. Create a workspace with no browser open and it never left `booting` — the reason MCP could not create a usable workspace. `nexus/src/lib/server/fleet/` replaces it with a server-owned layer: - **`store.ts`** — one `FleetStore` on `globalThis` behind `Symbol.for('agent-nexus.fleet')`. Its own slot rather than a `Singletons` field, because `fleet/` imports from `workers/` and `sessions/` and the reverse edge is a cycle (AGENTS.md #31). Carries `waitFor(predicate, timeoutMs)`, the in-process await #155/#156 need — **foundation, deliberately without a caller in this PR**. - **`runner.ts`** — `docker.getEvents()` filtered by the daemon to `nexus-managed=true`, with reconnect backoff, a 10s probe of the workspaces that cannot announce their own change, and a 60s resync. - **`intent.ts`** — Docker event → `observe | forget | ignore`, pure and daemon-free. - **`observe.ts`** — the relocated `checkWorkerHealth` body, now writing the store. - **`reconcile.ts`** — re-derives the fleet from DB + Docker. Absorbs `restoreSessionsOnBoot`, which is deleted rather than left beside it: two boot paths that both converge is how the double `claude --resume` that `convergingWorkers` guards against gets back in. `GET /api/workers` and `GET /api/workers/:id/health` now read the store. The health route became a pure read, so it **is now in `TOKEN_ROUTES`** (`workspaces:read`). `EVENTS: 1` added to the socket proxy — it was default-denied, and without it the stream 403s at runtime with every gate green, because the tests mock dockerode. `fleet/socket-proxy.test.ts` reads the compose file so that omission is red instead of silent. ## Two things measured that contradict the design spec - **`intentFor` must be an ALLOWLIST.** Docker emits `exec_create`/`exec_start`/`exec_die` as *container* events carrying the container's labels, and both `observeWorkspace`'s probes and the daemon's own healthcheck **are execs**. The spec's "every action but `destroy` means re-inspect" therefore closed a loop: observe → exec → events → observe. Widening `intentFor` reopens it. - **`hooks.server.ts`'s `init` does fire in production.** `build/handler.js` awaits `server.init()` at module top level, so importing it from `src/server.ts` runs `init` before that file's `main()` — measured, with `init`'s log line ahead of the first `migration applied`. The spec's "Starting the loop — the hazard" section and fact #14's old parenthetical were both wrong. There is one call site, not two. ## Code review found seven defects; all seven are fixed Five were silent by construction. Three change what the design says, and reviewers should read these rather than the spec: 1. **The watch is paired with a 60s resync, and the issue's "never periodically" does not mean what it looks like** — it is about the 10s *fleet poll*, which is gone. `convergeSessions` had been left with no trigger for the case it exists for: a session whose `claude` exits closes a tmux window inside a container that stays healthy, and **a healthy container emits no events at all** (measured: zero `health_status` in 45s of steady state, and zero non-exec events in a 59s window on a live daemon). The probe set also deliberately excluded every healthchecked workspace. So such a session was never restored — the browser poll used to do it. The resync restores fact #15's promise and also retries a boot reconcile that threw, which a healthy stream would otherwise never do. **#177c's `nexus-report-exit` is the proper edge; when it lands the resync becomes redundancy rather than the mechanism.** 2. **The probe set is `store.silent()`** — `!reporting || state === 'starting'`, at 10s. Boot progress was otherwise dead: no events fire during the healthcheck's `StartPeriod`, so the entrypoint's `==> ` markers never reached the card. The set is empty once a workspace is up, which keeps that cadence off the steady-state path. 3. **`unobserved` is a new `WorkspaceHealthState`.** The health route answered `unreportable` for a workspace nothing had observed — the state `Workers.svelte` renders as a **"needs restart"** chip. Every workspace carried that advice from boot, and permanently if the stream never opened. 4. **Observations are serialised per workspace** (`FleetStore.serialize`). Two overlapping observes commit in *completion* order, so a slow `stop` landed on top of the `die` behind it and the card read `running` until the next signal. It also stops two observes exec'ing into one container at once. 5. **Stream listeners are attached before the connect reconcile is awaited**, and the abandoned stream is `destroy()`ed. An `error` in that window is an unhandled `'error'` event, which Node rethrows and which takes the process down; a `close` in it parks the connect loop on a promise nothing can resolve — no reconnect, nothing logged. ## Verification Gates, each run as its own command (never chained — see AGENTS.md "Running the gates without lying to yourself"): | gate | result | |---|---| | `pnpm lint` | clean | | `pnpm typecheck` | **0 errors / 5199 files** | | `pnpm vitest run` | **2353 passed / 10 skipped** | | `pnpm build` | exit 0 | **Every guard was re-broken and proven to fail** — seven mutations, seven kills, each file restored from a `cp` backup and diffed byte-identical afterwards. One of the new tests *survived* its mutation on the first pass (a `PassThrough` resolves a destroyed-stream case through an unrelated path); it was rewritten against a deterministic double and now dies correctly. **On a real daemon**, with a labelled healthchecked container and a matching worker row, running the built artifact (`node build/server.js`): - stream opens, `fleet reconciled (workspaces: 1)`, `fleet: watching Docker events`; - resync ticks at **+59.9s and +60.0s** across three reconciles; - **zero non-exec events in 59s** from a steady container — finding 1's premise, re-measured; - `exec_create` **flat at 5–7 per 10s**, no growth. A/B with the server stopped: **30/60s healthcheck-only vs 37/60s running**. The extra 7 is exactly 6 silent probes plus the one resync observe. ## Notes for review - `WorkspaceEntry.containerId` / `lastSignalAt` / `lastSignalKind` are written and not yet read — the shape #177b's SSE payload wants. Delete them if #177b does not use them. - `observeWorkspace` still costs one uncached `docker exec` per observation for the degraded marker, where `probeClaudeVersion` beside it **is** cached. Caching it keyed on `containerId + State.StartedAt` (never the id alone, which survives a stop/start) is the open optimisation. - The store deliberately holds no `provision_phase`, so the `row ?? store` hazard (fact #27) cannot arise here. It arises in #177b. - AGENTS.md fact **#42** documents the layer; **#5, #14, #15, #24, #40** were amended because this change made their premises false.
lz added 24 commits 2026-09-22 08:08:34 +02:00
Task 5 renamed the function; these comments still named the old one,
including two that justified `isRunning` by a browser-poll premise the
fleet layer's event stream has since made false. Restate the mcp/projection
reasoning against the fleet layer instead of deleting it.
Facts 5, 15, 24 and 40 stated the pre-fleet arrangement as current: a
health poll, an unmapped health route, and convergence driven from a
browser tab. All three are now false in a way a reader would build on.
Fact 42 records the fleet layer itself — the store's globalThis slot,
the event-filtered stream, the residual probe, and why the old boot
path had to be deleted rather than left beside the new one.
Docker emits exec_create/exec_start/exec_die as container events, and both
observeWorkspace's probes and the daemon's own healthcheck are execs — so
admitting every non-destroy action made each observation produce the events
that trigger the next one. intentFor now takes an allowlist.

Measured alongside it: hooks.server.ts's init runs in production too, because
build/handler.js awaits server.init() at module top level. src/server.ts's
startFleet call was therefore dead on every boot, and its comment said the
reverse; both are gone, leaving the one call site that always runs. The
ordering that puts the fleet layer ahead of runMigrations is safe because
migrations are synchronous while the first fleet query is behind an await.

Also drops listManagedContainers, whose last caller went when the roster
started reading the store, and a re-export nothing imported.
fix(fleet): pair the watch with a resync, and stop trusting completion order
Some checks failed
ci / nexus (pull_request) Successful in 4m0s
ci / images (pull_request) Failing after 56s
72b23d4a80
The code review found seven defects in the event-driven fleet layer. Five of
them are silent by construction.

convergeSessions had been left with no trigger for the case it exists for. A
session whose claude exits closes a tmux window inside a container that stays
healthy, and a healthy container emits no events at all — measured, zero
health_status in 45s of steady state — while residual() deliberately excluded
every healthchecked workspace from the probe. So such a session was never
restored. reconcileFleet now also runs on a 60s timer, which is the resync any
watch-based controller owes its watch; that also retries a boot reconcile that
threw, which a healthy stream would otherwise never do.

The stream's error/end/close listeners were attached only after the connect
reconcile was awaited. An error in that window is an unhandled 'error' event,
which Node rethrows and which takes the process down; a close in it parks the
connect loop on a promise nothing can resolve, with no reconnect and nothing
logged. They are wired before the await now, and the stream a reconnect
abandons is destroyed rather than left feeding apply beside its successor.

Observations of one workspace ran concurrently and committed in completion
order, so a slow observe of a `stop` landed on top of the fast observe of the
`die` behind it and the workspace read `running` until the next signal. They
are serialised per workspace, which also stops two observes exec'ing into one
container at once.

The health route answered `unreportable` for a workspace nothing had observed
yet — the state Workers.svelte renders as a "needs restart" chip. Every
workspace carried that advice from boot until its first observation, and
permanently if the stream never opened. `unobserved` is now its own state.

Boot progress was effectively dead: no events fire during the healthcheck's
start period, so the entrypoint's `==> ` markers never reached the card. The
probe set is now every workspace that cannot announce its own change — no
healthcheck, or still inside StartPeriod — on 10s rather than 60s. It is empty
once a workspace is up, so the fast cadence stays off the steady-state path.

Every new guard was re-broken and proven to fail: seven mutations, seven kills,
each file restored byte-identical.
lz added this to the MCP support (#140) milestone 2026-09-22 08:08:49 +02:00
lz changed target branch from feat/daemon-container-health to main 2026-09-22 11:18:51 +02:00
Merge main (#188, nexus-plugin SessionStart) into feat/fleet-status
All checks were successful
ci / nexus (pull_request) Successful in 5m38s
pr-image-cleanup / delete-pr-images (pull_request) Successful in 27s
ci / images (pull_request) Successful in 7m45s
18d26fce59
Both conflicts were each side deleting a different line in the same spot:
server.ts drops restoreSessionsOnBoot (absorbed by reconcileFleet here) and
seedAgentInstructionsOnBoot (retired on main); scopes.test.ts drops the
health route (now mapped here) and agent-instructions (deleted on main).
lz merged commit 6184f44206 into main 2026-09-23 10:36:49 +02:00
lz deleted branch feat/fleet-status 2026-09-23 10:36:49 +02:00
Sign in to join this conversation.
No reviewers
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
lz/agent-nexus!192
No description provided.