Server-owned, event-driven fleet status layer #177

Open
opened 2026-09-18 07:48:44 +02:00 by lz · 1 comment
Owner

Tracks docs/superpowers/specs/2026-09-17-fleet-status-layer-design.md.

Nexus has no server-side reconciliation. The loop is a setInterval in routes/+page.svelte — the home-page component — and checkWorkerHealth has exactly one caller, the health route it drives. So fleet state is only true while a browser sits on /.

Navigate to /settings and it stops. With no browser at all, provision_phase never leaves booting, which means a workspace created or started over MCP is permanently unusable. That is what blocks the whole MCP lifecycle arc.

This does not make the codebase smaller — an audit measured it. Justify it on correctness: server-owned reconciliation, /api/workers/:id/health becoming side-effect-free and therefore TOKEN_ROUTES-eligible for the first time, and MCP having something true to await.

Shape

Docker's own /events stream plus a container Healthcheck set at create, feeding one process-wide store on globalThis; SSE to the UI; fleet.waitFor() in-process for MCP. Reconcile is edge-triggered — on boot, on stream reconnect, and for the workspaces that structurally cannot report — never periodic.

Full design in the spec. The parts most likely to be got wrong:

  • EVENTS is in the socket proxy's default-deny list. docker-compose.yml names it explicitly among the flags left at 0. Without EVENTS: 1 the stream 403s at runtime with the build and suite green, because the tests mock dockerode.
  • The loop cannot be started from one place. src/server.ts is the production entry and pnpm dev never evaluates it; SvelteKit's init runs in dev but fact #14 records it does not fire from the custom entry. Both, idempotently, or it runs in exactly one environment while looking correct in the other.
  • The healthcheck does not delete the boot machinery. Healthcheck is fixed at create, docker update cannot add one, and nothing forces a recreate — so legacy containers never emit health_status and settleBootingPhase, lastBootMarker, BOOT_GRACE_MS and the tmux probe all survive for them.
  • Docker's start fires before the entrypoint's first line. /run/nexus/degraded is written at entrypoint.sh:84 and tmux new-session is line 255 of 256. Degraded state and the claude version must hang off health_status: healthy, or an exec on start reads "not degraded" permanently.
  • provision_phase assembly is row ?? store, never the reverse. Backwards, a Remove-during-pull renders as still pulling and isRunning returns true for a container that is gone — the incident fact #27 was written about.
  • Deleting the UI poll deletes the session-expiry detector. EventSource exposes no status code — a 401 arrives as onerror — and the stream never passes through call(), so unlock-redirect.ts does not cover it. The session-stats poll is what detects expiry today (fact #25).

Depends on

The declaration collapse, and the two bugs below, all of which are strictly cheaper before this than after.

Done when

A workspace created or started with no browser open reaches ready on its own, convergeSessions is driven by Nexus rather than by a client, and /api/workers/:id/health is in TOKEN_ROUTES.

Tracks `docs/superpowers/specs/2026-09-17-fleet-status-layer-design.md`. Nexus has no server-side reconciliation. The loop is a `setInterval` in `routes/+page.svelte` — the home-page component — and `checkWorkerHealth` has exactly one caller, the health route it drives. So fleet state is only true while a browser sits on `/`. Navigate to `/settings` and it stops. With no browser at all, `provision_phase` never leaves `booting`, which means a workspace created or started over MCP is permanently unusable. That is what blocks the whole MCP lifecycle arc. **This does not make the codebase smaller** — an audit measured it. Justify it on correctness: server-owned reconciliation, `/api/workers/:id/health` becoming side-effect-free and therefore `TOKEN_ROUTES`-eligible for the first time, and MCP having something true to await. ## Shape Docker's own `/events` stream plus a container `Healthcheck` set at create, feeding one process-wide store on `globalThis`; SSE to the UI; `fleet.waitFor()` in-process for MCP. Reconcile is edge-triggered — on boot, on stream reconnect, and for the workspaces that structurally cannot report — never periodic. Full design in the spec. The parts most likely to be got wrong: - **`EVENTS` is in the socket proxy's default-deny list.** `docker-compose.yml` names it explicitly among the flags left at 0. Without `EVENTS: 1` the stream 403s at runtime with the build and suite green, because the tests mock dockerode. - **The loop cannot be started from one place.** `src/server.ts` is the production entry and `pnpm dev` never evaluates it; SvelteKit's `init` runs in dev but fact #14 records it does not fire from the custom entry. Both, idempotently, or it runs in exactly one environment while looking correct in the other. - **The healthcheck does not delete the boot machinery.** `Healthcheck` is fixed at create, `docker update` cannot add one, and nothing forces a recreate — so legacy containers never emit `health_status` and `settleBootingPhase`, `lastBootMarker`, `BOOT_GRACE_MS` and the tmux probe all survive for them. - **Docker's `start` fires before the entrypoint's first line.** `/run/nexus/degraded` is written at `entrypoint.sh:84` and `tmux new-session` is line 255 of 256. Degraded state and the claude version must hang off `health_status: healthy`, or an exec on `start` reads "not degraded" permanently. - **`provision_phase` assembly is `row ?? store`, never the reverse.** Backwards, a Remove-during-pull renders as still pulling and `isRunning` returns true for a container that is gone — the incident fact #27 was written about. - **Deleting the UI poll deletes the session-expiry detector.** `EventSource` exposes no status code — a 401 arrives as `onerror` — and the stream never passes through `call()`, so `unlock-redirect.ts` does not cover it. The `session-stats` poll is what detects expiry today (fact #25). ## Depends on The declaration collapse, and the two bugs below, all of which are strictly cheaper before this than after. ## Done when A workspace created or started with no browser open reaches `ready` on its own, `convergeSessions` is driven by Nexus rather than by a client, and `/api/workers/:id/health` is in `TOKEN_ROUTES`.
lz added this to the MCP support (#140) milestone 2026-09-18 07:48:44 +02:00
Author
Owner

Unblocked once #188 (issue #184) merges — and it changes what this issue has to build.

#184 shipped as PR #188. What #177 inherits:

Put the discriminator on the stream, not Docker's raw string. workspaceHealthState() (workers/healthcheck.ts) maps the daemon's open-ended status to a closed set — healthy | unhealthy | starting | unreportable — and WorkerHealthDTO now carries it as state alongside health. It exists because four consumers each re-read the raw value with their own predicate and had drifted: the version probe kept === 'healthy' while convergence was widened, so a pre-healthcheck container silently stopped reporting claudeVersion. A fifth consumer on an event stream would have been the next one to drift.

The degraded probe is the cache this issue should build, and #184 deliberately did not. It is still one docker exec per poll per workspace — measured, ~21 Docker round trips/min/workspace — for a marker written once at entrypoint.sh:84 and never removed. It was left alone because this issue moves it onto health_status: healthy, where it is computed once; building a cache first would mean building it and then deleting it. When you do: key it on containerId + State.StartedAt, never on the container id alone — the id survives a stop/start, so it would carry a stale false across a boot that went degraded.

Convergence is gated on healthy || unreportable, never on !== 'starting'. convergeSessions opens with waitForTmux(container, 60_000), which execs every 250ms for a full minute. Converging a workspace whose tmux is known dead burns ~4 execs/sec against it indefinitely — the in-flight guard only stops the passes overlapping, not recurring. The unreportable arm is load-bearing: gating on healthy alone silently stops restoring a legacy container's sessions, which is the promise AGENTS.md fact #15 makes.

Already deleted, so the spec's residual-prober section is moot: settleBootingPhase, BOOT_GRACE_MS, the tmux execProbe and the log-marker boot inference. lastBootMarker was deliberately KEPT — the daemon says starting for the whole 30-90s window and has no opinion on the stage, and that marker feeds provision_detail on three surfaces. restoreSessionsOnBoot is still present and is this issue's to delete (item 5 of the roadmap).

A worker image release is owed before item 7. #188 changed worker/entrypoint.sh (the clone guard, and test -s before the jq mv), and the entrypoint is baked into the image — COPY entrypoint.sh at Dockerfile:61, ENTRYPOINT at :83. A container recreate does not pick it up: it builds a new container from the same image tag. nexus-report-exit needs a release too, so release once for both, and check whether #183 is close enough to join.

Two things worth not rediscovering:

  • last_seen_at is not a clone proxy. finishProvisioning writes it right after container.start(), before the entrypoint clones, so it means "a container for this row started once". #188 carries it as ProvisionContext.hasEverRun and the entrypoint now does the real check via /workspace/.nexus.
  • EVENTS: 1 still has to be added to docker-compose.yml, or docker.getEvents() 403s at runtime while the suite stays green — the tests mock dockerode.
**Unblocked once #188 (issue #184) merges — and it changes what this issue has to build.** #184 shipped as PR #188. What #177 inherits: **Put the discriminator on the stream, not Docker's raw string.** `workspaceHealthState()` (`workers/healthcheck.ts`) maps the daemon's open-ended status to a closed set — `healthy | unhealthy | starting | unreportable` — and `WorkerHealthDTO` now carries it as `state` alongside `health`. It exists because four consumers each re-read the raw value with their own predicate and had drifted: the version probe kept `=== 'healthy'` while convergence was widened, so a pre-healthcheck container silently stopped reporting `claudeVersion`. A fifth consumer on an event stream would have been the next one to drift. **The degraded probe is the cache this issue should build, and #184 deliberately did not.** It is still one `docker exec` per poll per workspace — measured, ~21 Docker round trips/min/workspace — for a marker written once at `entrypoint.sh:84` and never removed. It was left alone because this issue moves it onto `health_status: healthy`, where it is computed once; building a cache first would mean building it and then deleting it. When you do: key it on `containerId + State.StartedAt`, **never on the container id alone** — the id survives a stop/start, so it would carry a stale `false` across a boot that went degraded. **Convergence is gated on `healthy || unreportable`, never on `!== 'starting'`.** `convergeSessions` opens with `waitForTmux(container, 60_000)`, which execs every 250ms for a full minute. Converging a workspace whose tmux is known dead burns ~4 execs/sec against it indefinitely — the in-flight guard only stops the passes overlapping, not recurring. The `unreportable` arm is load-bearing: gating on `healthy` alone silently stops restoring a legacy container's sessions, which is the promise AGENTS.md fact #15 makes. **Already deleted, so the spec's residual-prober section is moot:** `settleBootingPhase`, `BOOT_GRACE_MS`, the tmux `execProbe` and the log-marker boot inference. `lastBootMarker` was deliberately KEPT — the daemon says `starting` for the whole 30-90s window and has no opinion on the stage, and that marker feeds `provision_detail` on three surfaces. `restoreSessionsOnBoot` is still present and is this issue's to delete (item 5 of the roadmap). **A worker image release is owed before item 7.** #188 changed `worker/entrypoint.sh` (the clone guard, and `test -s` before the jq `mv`), and the entrypoint is baked into the image — `COPY entrypoint.sh` at `Dockerfile:61`, `ENTRYPOINT` at `:83`. A container recreate does **not** pick it up: it builds a new container from the same image tag. `nexus-report-exit` needs a release too, so release once for both, and check whether #183 is close enough to join. Two things worth not rediscovering: - `last_seen_at` is **not** a clone proxy. `finishProvisioning` writes it right after `container.start()`, before the entrypoint clones, so it means "a container for this row started once". #188 carries it as `ProvisionContext.hasEverRun` and the entrypoint now does the real check via `/workspace/.nexus`. - `EVENTS: 1` still has to be added to `docker-compose.yml`, or `docker.getEvents()` 403s at runtime while the suite stays green — the tests mock dockerode.
Sign in to join this conversation.
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
lz/agent-nexus#177
No description provided.