feat(fleet): push workspace state to the browser over SSE (#177b) #193

Merged
lz merged 25 commits from feat/fleet-sse into main 2026-09-23 10:38:36 +02:00
Owner

Closes the UI half of #177 (milestone 27). Targets feat/fleet-status (#192), which targets feat/daemon-container-health (#188) — not main. This is a three-deep stack; merging out of order will not compile. #188 → #192 → this.

What this does

Workspace state stops being polled.

Before, routes/+page.svelte bumped a tick every 10s and workspaces.refresh() answered it with 1 roster + 1 previews + 2 × N_running requests — sessions and health per running workspace. At five running workspaces that is 12 requests every 10s, in every open tab, forever. A container transition was visible up to 10s late, and later still if the operator happened to be typing a session name (the tick skips itself to avoid disrupting input).

#177a made the server hold live_status and health in a FleetStore. This PR pushes them.

  • lib/server/events/bus.ts — a leaf fan-out. subscribe/touch/forget, behind a Symbol.for slot.
  • lib/server/events/payload.ts — fleetSnapshot (literally listWorkers(db, store) plus the store's health map) and workspaceDelta. No new wire types: the payload is WorkerDTO and WorkerHealthDTO, so none of the repo's declared-on-both-sides parity rules apply.
  • GET /api/events — SSE, cookie-authenticated, snapshot then per-workspace deltas, coalesced per microtask.
  • workspaces.svelte.ts — connect(), applySnapshot/applyDelta/applyForget, and a streamAuthoritative flag that decides which single path owns the roster.

Sessions stay on the tick. attention_state is written by the worker's hooks via POST /api/agent/events, which touches no workspace, so a session going needs-you raises no fleet signal and has nothing to ride. Putting it on the stream needs a server-side session store plus a notify in that callback — unowned work; no open issue in milestone 27 covers it, and #155 is an MCP tool, not an SSE feature.

Measured against the built artifact

Everything here was driven through a browser against node build/server.js with real Docker containers, not reasoned about.

docker stop → DOM mutation 2 ms, no request in between
requests per page load one /api/events (stays open), one /api/workers on mount, zero roster/health after
45s at rest /api/workers 0, /api/workers/:id/health 0; sessions 10 (2 workspaces × 5 ticks), previews 5, quota 5
16 transitions in 4.0s 18 DOM updates, 0 fetches for the whole burst
/api/quota cadence under load 10.000s apart to the millisecond — no delta moved it
tab close 3 sockets → 0 within 6s
boot progress provision_detail advancing every ~10s on SILENT_PROBE_INTERVAL_MS, no HTTP request accompanying any of it
pull progress provision_percent 18 → 27 → 35 → 44 → 52 → 60 → 69 → 78 → 86 → 94 → 99 → 100 across 27 pulling deltas, spaced 760–846 ms
frame order on that run 1 snapshot then 28 workspace — snapshot first, as designed
/api/events unauthenticated 401, and 401 to a bearer token — JSON, not a stream, so the gate fires ahead of the handler

The pull half was driven by spawning against an image absent from the daemon, forcing a real docker pull. It confirms the claim the spec made by reasoning: PULL_WRITE_INTERVAL_MS (750 ms) is inherited by the SSE emit for free, because it already gates the call into setProvisionProgress — the measured 760–846 ms spacing is that throttle, not a second one. The run ended in recordProvisionError ("not a Nexus worker image (missing the org.nexus.worker label)"), so the terminal error phase reached the stream too. That exercises three of the ten phase writers end to end — advanceProvisionPhase, setProvisionProgress, recordProvisionError.

Two transport facts, measured because a streaming response through a custom Node entry is exactly the kind of thing that works in dev and not in production: frames arrive 2–3 ms after enqueue through handle → adapter-node → src/server.ts's createServer, and teardown is socket-close-driven (build/handler.js wires res.on('close') to reader.cancel()), so a closed tab cancels its stream while idle and no heartbeat is needed. Idle is the steady state here by design, so a write-triggered teardown would have leaked a subscriber per closed tab and no test that writes would have caught it.

Five defects found during implementation, not by review

Each of these was in the gap between the design and what the code actually does. None would have been caught by reading the spec.

  1. A phase writer must announce only a write that LANDED. Nine of the ten provisioning.ts writers carry a scoping predicate that can match nothing. notBeingRemoved is the one that bites: it exists so a Remove clicked during a pull is not overwritten, and in exactly that scenario the pull loop calls setProvisionProgress every 750ms with every call refused — each shipping "this workspace changed" to every tab while the card should read "Removing…". An earlier revision split the writers "guarded vs unconditional" and was wrong three times running; there is now one rule and no classification. provisioning-notify.test.ts enforces the form — the line carrying fleetBus().touch( must also carry > 0 — because asserting the call merely exists passes over an unguarded one.

  2. Existence is decided by the row, never by the signal. FleetStore.forget also fires when a container is destroyed during startWorker's recreate (service.ts:851-862), where the row survives. Forwarding that as "drop this workspace" made a card vanish on Stop/Start. Both signal kinds now run workspaceDelta and its null is the single authority.

  3. The snapshot must be written before any delta. Subscribing before the snapshot query starts prevents a signal being missed; it does not order the two frames, which are independent promise chains. A forget arriving first was applied and then the snapshot resurrected the workspace, permanently. Signals now buffer until the snapshot has been sent — including on the snapshot's error path, or a failed query parks every later signal for the life of the connection, silently.

  4. The client reads the roster under untrack(). A tracked read inside refresh(), which is called from an $effect, is not merely an extra fetch per delta — the authoritative branch reads sessionsByWorker with no preceding await and the commit then writes it, which is effect_update_depth_exceeded. $effect is inert under vitest's node environment (Svelte resolves to its SSR build), so no unit test can observe this; it was confirmed in a browser across 16 transitions in 4.0s with a page-level error collector, since Svelte throws that error rather than logging it.

  5. A snapshot must not discard an in-flight refresh's sessions. Bumping generation in applySnapshot superseded the whole commit, including sessionsByWorker and previews — state the stream does not carry — leaving the sidebar's session list empty for a full 10s on every page load. Time to a visible session name: 10409 / 10389 ms before, 97 / 92 / 80 ms after. The streamAuthoritative guard already protects the roster; the bump protected nothing and cost a tick of session state. Restoring the bump kills the new test and leaves a refresh that started before a delta does not clobber it green — which is the proof the bump was never what protected the roster.

    The mechanism differs between environments, same outcome: in the unit-test timing the bump lands before listWorkers resolves, so the session fan-out never fires at all; in a browser the fan-out has already been issued and only its commit is thrown away. The mount race above is unchanged by the fix — what changed is that the mount refresh's results now survive.

Known limits, stated rather than buried

  • The mount refresh() can still issue 0–2 /api/workers/:id/health requests when the snapshot lands mid-fan-out. Bounded to page load, self-correcting, and replacing N every 10s forever. Accepted rather than closed with machinery.
  • A workspace appended by a delta mid-refresh has its empty sessionsByWorker entry pruned at the commit; self-corrects on the next tick, and a just-arrived workspace has no sessions to lose.

Verification

Gates, each run as its own command:

gate
pnpm lint clean
pnpm typecheck 0 errors / 5207 files
pnpm vitest run 2402 passed / 10 skipped
pnpm build exit 0

pnpm build matters here specifically: it is the only gate that validates SvelteKit route-module exports (fact #32), and this PR adds a route.

Fourteen mutations were run against the guards this PR adds, each killed, each file restored from a cp backup and diffed byte-identical afterwards. That is not the same as "every guard was proven to fail", and an earlier revision of this body said so incorrectly — a review subsequently found nine mutations the suite SURVIVES (listed below). Two are worth naming because they nearly recorded the wrong result: one mutation duplicated a block instead of moving it, passing as a survival until the diff showed 46a47,56; and the original ordering test asserted .sort()-ed membership, which cannot distinguish sequence from set, so it would have survived an inverted-order mutation. Both were caught and the test was converted to assert the sequence explicitly.

AGENTS.md facts #42 and #26 amended.


Review findings — all fixed

A five-agent review (correctness, silent failures, tests, comments, type design) found three critical defects and nine mutations the suite survived. All are fixed on this branch; every fix was proven by re-breaking the code it guards.

The root cause was architectural. The 10s poll was the client's only repair mechanism and nothing replaced it. The server reconciles its own store every 60s; the client reconciled never — so every divergence became permanent instead of ≤10s stale. #192's own lesson, that a watch owes its watch a resync, had been applied to the server and not one layer up.

The client now resyncs its roster every 60s even while the stream is authoritative. Roster only: health arrives via the stream and the server's reconcile keeps pushing it, so a missed health delta self-heals; what cannot is existence, because nothing re-announces a row that is gone. A resync overtaken by a delta discards its own result. Cost is roughly one request per minute; the 2 ms transition latency and the removal of the per-workspace health fetch are untouched.

The three criticals

  • A completed removal announced nothing. runTeardown's row delete raised no signal — fleetBus was not imported in service.ts at all — while the container destroy fires seconds earlier and resolves to a workspace delta because the row still exists, and empties the store entry so reconcileFleet's orphan sweep had nothing left to name. The card stuck on "Removing…" in every open tab until a reload, and only when everything went right: with no container there is no destroy event and the sweep cleared it correctly. Measured across a full removeWorker: one signal, beginRemoval's claim.
  • A failed snapshot fabricated an empty fleet, and it was data loss. The client cannot tell that frame from a genuinely empty roster: it set streamAuthoritative, disarmed its poll, and marked every scope dead — and the dock's panel prune is debounced into localStorage, so a reload restored the emptied layout. The connection now ends instead, reaching the client's own onerror. It closes rather than errors because controller.error() discards queued chunks, which would have lost the retry: hint enqueued immediately before it; and it releases its subscription and keepalive by hand, because cancel() runs only when the consumer cancels.
  • The SSE event names were declared twice with nothing tying them together. EventSource silently ignores an event type nobody registered for — no throw, no onerror — so a drifted name left the snapshot listener firing, the poll disarmed and the roster frozen with the page looking healthy and nothing logged at either end. Invisible to every gate. They are now one exported constant in $lib/api/types.ts, imported by both sides.

Also fixed

  • A half-open socket was never noticed. Neither side writes in the steady state, so a NAT/proxy idle timeout, VPN re-key or suspend left the client serving stale state forever. A 25 s comment frame is the write that eventually fails.
  • FleetSignal.kind is gone. Nothing read it, and its forget was a false cognate for the wire's — the bus fires when a container is destroyed, whose row usually survives a recreate.
  • The provisioning guard is behavioural, not syntactic. The old one asked whether the notify line also carried > 0: any unrelated predicate satisfied it (message.length > 0 passed), it rejected a semantically identical guard-clause rewrite, and its enumeration could not see a writer declared in another form — an arrow-function writer with no notify at all passed every case. Every writer is now called against a deleted row and must stay silent, and every export must be a known writer or a named exemption.
  • Nine surviving mutations now die, including: a delta with null health erasing held health (it fires on every spawn); a forget not dropping that workspace's sessions, under a test named for it; the per-connection coalescing buffer hoisted to module scope (two tabs, one silently stale); the buffer never draining.
  • streamLost() on unmount (close() fires no onerror); connect()'s docblock corrected — only a network failure or an aborted established stream reconnects, so a 500 or 401 handshake is terminal; the $effect implicit-return-as-cleanup trap documented.

Correction to this PR's own claims

An earlier revision of this body said "every guard was re-broken and proven to fail." That was true of the fourteen guards tested and false as a general claim. AGENTS.md fact #42 also stated a snapshot bumps generation — the branch's own last commit had removed that deliberately, one commit after the fact was written. Both corrected.

Verification

gate
pnpm lint clean
pnpm typecheck 0 errors / 5207 files
pnpm vitest run 2411 passed / 10 skipped
pnpm build exit 0
Closes the UI half of #177 (milestone 27). **Targets `feat/fleet-status` (#192), which targets `feat/daemon-container-health` (#188) — not `main`.** This is a three-deep stack; merging out of order will not compile. #188 → #192 → this. ## What this does Workspace state stops being polled. Before, `routes/+page.svelte` bumped a `tick` every 10s and `workspaces.refresh()` answered it with **1 roster + 1 previews + 2 × N_running** requests — sessions *and* health per running workspace. At five running workspaces that is 12 requests every 10s, in every open tab, forever. A container transition was visible up to 10s late, and later still if the operator happened to be typing a session name (the tick skips itself to avoid disrupting input). #177a made the server hold `live_status` and health in a `FleetStore`. This PR pushes them. - **`lib/server/events/bus.ts`** — a leaf fan-out. `subscribe`/`touch`/`forget`, behind a `Symbol.for` slot. - **`lib/server/events/payload.ts`** — `fleetSnapshot` (literally `listWorkers(db, store)` plus the store's health map) and `workspaceDelta`. **No new wire types**: the payload is `WorkerDTO` and `WorkerHealthDTO`, so none of the repo's declared-on-both-sides parity rules apply. - **`GET /api/events`** — SSE, cookie-authenticated, snapshot then per-workspace deltas, coalesced per microtask. - **`workspaces.svelte.ts`** — `connect()`, `applySnapshot`/`applyDelta`/`applyForget`, and a `streamAuthoritative` flag that decides which single path owns the roster. Sessions stay on the tick. `attention_state` is written by the worker's hooks via `POST /api/agent/events`, which touches no workspace, so a session going `needs-you` raises no fleet signal and has nothing to ride. Putting it on the stream needs a server-side session store plus a notify in that callback — **unowned work; no open issue in milestone 27 covers it, and #155 is an MCP tool, not an SSE feature.** ## Measured against the built artifact Everything here was driven through a browser against `node build/server.js` with real Docker containers, not reasoned about. | | | |---|---| | `docker stop` → DOM mutation | **2 ms**, no request in between | | requests per page load | one `/api/events` (stays open), one `/api/workers` on mount, **zero** roster/health after | | 45s at rest | `/api/workers` **0**, `/api/workers/:id/health` **0**; sessions 10 (2 workspaces × 5 ticks), previews 5, quota 5 | | 16 transitions in 4.0s | 18 DOM updates, **0 fetches for the whole burst** | | `/api/quota` cadence under load | 10.000s apart to the millisecond — no delta moved it | | tab close | 3 sockets → **0** within 6s | | boot progress | `provision_detail` advancing every ~10s on `SILENT_PROBE_INTERVAL_MS`, no HTTP request accompanying any of it | | pull progress | `provision_percent` **18 → 27 → 35 → 44 → 52 → 60 → 69 → 78 → 86 → 94 → 99 → 100** across 27 `pulling` deltas, spaced **760–846 ms** | | frame order on that run | 1 `snapshot` then 28 `workspace` — snapshot first, as designed | | `/api/events` unauthenticated | **401**, and 401 to a bearer token — JSON, not a stream, so the gate fires ahead of the handler | The pull half was driven by spawning against an image absent from the daemon, forcing a real `docker pull`. It confirms the claim the spec made by reasoning: **`PULL_WRITE_INTERVAL_MS` (750 ms) is inherited by the SSE emit for free**, because it already gates the call into `setProvisionProgress` — the measured 760–846 ms spacing is that throttle, not a second one. The run ended in `recordProvisionError` ("not a Nexus worker image (missing the org.nexus.worker label)"), so the terminal error phase reached the stream too. That exercises three of the ten phase writers end to end — `advanceProvisionPhase`, `setProvisionProgress`, `recordProvisionError`. Two transport facts, measured because a streaming response through a custom Node entry is exactly the kind of thing that works in dev and not in production: frames arrive **2–3 ms** after `enqueue` through `handle` → adapter-node → `src/server.ts`'s `createServer`, and **teardown is socket-close-driven** (`build/handler.js` wires `res.on('close')` to `reader.cancel()`), so a closed tab cancels its stream while idle and **no heartbeat is needed**. Idle is the steady state here by design, so a write-triggered teardown would have leaked a subscriber per closed tab and no test that writes would have caught it. ## Five defects found during implementation, not by review Each of these was in the gap between the design and what the code actually does. None would have been caught by reading the spec. 1. **A phase writer must announce only a write that LANDED.** Nine of the ten `provisioning.ts` writers carry a scoping predicate that can match nothing. `notBeingRemoved` is the one that bites: it exists so a Remove clicked during a pull is not overwritten, and in exactly that scenario the pull loop calls `setProvisionProgress` every 750ms with every call refused — each shipping "this workspace changed" to every tab while the card should read "Removing…". An earlier revision split the writers "guarded vs unconditional" and was wrong three times running; there is now one rule and no classification. `provisioning-notify.test.ts` enforces the **form** — the line carrying `fleetBus().touch(` must also carry `> 0` — because asserting the call merely exists passes over an unguarded one. 2. **Existence is decided by the row, never by the signal.** `FleetStore.forget` also fires when a container is destroyed during `startWorker`'s recreate (`service.ts:851-862`), where the row survives. Forwarding that as "drop this workspace" **made a card vanish on Stop/Start**. Both signal kinds now run `workspaceDelta` and its `null` is the single authority. 3. **The snapshot must be written before any delta.** Subscribing before the snapshot query starts prevents a signal being *missed*; it does not *order* the two frames, which are independent promise chains. A `forget` arriving first was applied and then the snapshot resurrected the workspace, permanently. Signals now buffer until the snapshot has been sent — including on the snapshot's **error** path, or a failed query parks every later signal for the life of the connection, silently. 4. **The client reads the roster under `untrack()`.** A tracked read inside `refresh()`, which is called from an `$effect`, is not merely an extra fetch per delta — the authoritative branch reads `sessionsByWorker` with no preceding await and the commit then writes it, which is `effect_update_depth_exceeded`. **`$effect` is inert under vitest's node environment** (Svelte resolves to its SSR build), so no unit test can observe this; it was confirmed in a browser across 16 transitions in 4.0s with a page-level error collector, since Svelte throws that error rather than logging it. 5. **A snapshot must not discard an in-flight refresh's sessions.** Bumping `generation` in `applySnapshot` superseded the whole commit, including `sessionsByWorker` and `previews` — state the stream does not carry — leaving the sidebar's session list **empty for a full 10s on every page load**. Time to a visible session name: **10409 / 10389 ms before, 97 / 92 / 80 ms after**. The `streamAuthoritative` guard already protects the roster; the bump protected nothing and cost a tick of session state. Restoring the bump kills the new test and leaves `a refresh that started before a delta does not clobber it` **green** — which is the proof the bump was never what protected the roster. The mechanism differs between environments, same outcome: in the unit-test timing the bump lands before `listWorkers` resolves, so the session fan-out never fires at all; in a browser the fan-out has already been issued and only its commit is thrown away. The mount race above is unchanged by the fix — what changed is that the mount refresh's results now survive. ## Known limits, stated rather than buried - **The mount `refresh()` can still issue 0–2 `/api/workers/:id/health` requests** when the snapshot lands mid-fan-out. Bounded to page load, self-correcting, and replacing N every 10s forever. Accepted rather than closed with machinery. - **A workspace appended by a delta mid-refresh** has its empty `sessionsByWorker` entry pruned at the commit; self-corrects on the next tick, and a just-arrived workspace has no sessions to lose. ## Verification Gates, each run as its own command: | gate | | |---|---| | `pnpm lint` | clean | | `pnpm typecheck` | **0 errors / 5207 files** | | `pnpm vitest run` | **2402 passed / 10 skipped** | | `pnpm build` | exit 0 | `pnpm build` matters here specifically: it is the only gate that validates SvelteKit route-module exports (fact #32), and this PR adds a route. Fourteen mutations were run against the guards this PR adds, each killed, each file restored from a `cp` backup and `diff`ed byte-identical afterwards. **That is not the same as "every guard was proven to fail", and an earlier revision of this body said so incorrectly** — a review subsequently found nine mutations the suite SURVIVES (listed below). Two are worth naming because they nearly recorded the wrong result: one mutation *duplicated* a block instead of moving it, passing as a survival until the `diff` showed `46a47,56`; and the original ordering test asserted `.sort()`-ed membership, which cannot distinguish sequence from set, so it would have survived an inverted-order mutation. Both were caught and the test was converted to assert the sequence explicitly. AGENTS.md facts **#42** and **#26** amended. --- ## Review findings — all fixed A five-agent review (correctness, silent failures, tests, comments, type design) found **three critical defects and nine mutations the suite survived**. All are fixed on this branch; every fix was proven by re-breaking the code it guards. **The root cause was architectural.** The 10s poll was the client's only repair mechanism and nothing replaced it. The server reconciles its own store every 60s; the client reconciled **never** — so every divergence became permanent instead of ≤10s stale. #192's own lesson, that a watch owes its watch a resync, had been applied to the server and not one layer up. **The client now resyncs its roster every 60s** even while the stream is authoritative. Roster only: health arrives via the stream and the server's reconcile keeps pushing it, so a missed health delta self-heals; what cannot is *existence*, because nothing re-announces a row that is gone. A resync overtaken by a delta discards its own result. Cost is roughly one request per minute; the 2 ms transition latency and the removal of the per-workspace health fetch are untouched. ### The three criticals - **A completed removal announced nothing.** `runTeardown`'s row delete raised no signal — `fleetBus` was not imported in `service.ts` at all — while the container `destroy` fires seconds earlier and resolves to a `workspace` delta because the row still exists, *and* empties the store entry so `reconcileFleet`'s orphan sweep had nothing left to name. The card stuck on "Removing…" in every open tab until a reload, and **only when everything went right**: with no container there is no destroy event and the sweep cleared it correctly. Measured across a full `removeWorker`: one signal, `beginRemoval`'s claim. - **A failed snapshot fabricated an empty fleet, and it was data loss.** The client cannot tell that frame from a genuinely empty roster: it set `streamAuthoritative`, disarmed its poll, and marked every scope dead — and the dock's panel prune is debounced into `localStorage`, so a reload restored the emptied layout. The connection now ends instead, reaching the client's own `onerror`. It **closes rather than errors** because `controller.error()` discards queued chunks, which would have lost the `retry:` hint enqueued immediately before it; and it releases its subscription and keepalive by hand, because `cancel()` runs only when the *consumer* cancels. - **The SSE event names were declared twice with nothing tying them together.** `EventSource` silently ignores an event type nobody registered for — no throw, no `onerror` — so a drifted name left the snapshot listener firing, the poll disarmed and the roster frozen with the page looking healthy and nothing logged at either end. Invisible to every gate. They are now one exported constant in `$lib/api/types.ts`, imported by both sides. ### Also fixed - **A half-open socket was never noticed.** Neither side writes in the steady state, so a NAT/proxy idle timeout, VPN re-key or suspend left the client serving stale state forever. A 25 s comment frame is the write that eventually fails. - **`FleetSignal.kind` is gone.** Nothing read it, and its `forget` was a false cognate for the wire's — the bus fires when a *container* is destroyed, whose row usually survives a recreate. - **The provisioning guard is behavioural, not syntactic.** The old one asked whether the notify line also carried `> 0`: any unrelated predicate satisfied it (`message.length > 0` passed), it *rejected* a semantically identical guard-clause rewrite, and its enumeration could not see a writer declared in another form — an arrow-function writer with no notify at all passed every case. Every writer is now called against a deleted row and must stay silent, and every export must be a known writer or a named exemption. - **Nine surviving mutations now die**, including: a delta with null health erasing held health (it fires on every spawn); a forget not dropping that workspace's sessions, under a test *named* for it; the per-connection coalescing buffer hoisted to module scope (two tabs, one silently stale); the buffer never draining. - `streamLost()` on unmount (`close()` fires no `onerror`); `connect()`'s docblock corrected — only a network failure or an aborted *established* stream reconnects, so a 500 or 401 handshake is terminal; the `$effect` implicit-return-as-cleanup trap documented. ### Correction to this PR's own claims An earlier revision of this body said *"every guard was re-broken and proven to fail."* That was true of the fourteen guards tested and false as a general claim. AGENTS.md fact #42 also stated a snapshot bumps `generation` — the branch's own last commit had removed that deliberately, one commit after the fact was written. Both corrected. ## Verification | gate | | |---|---| | `pnpm lint` | clean | | `pnpm typecheck` | **0 errors / 5207 files** | | `pnpm vitest run` | **2411 passed / 10 skipped** | | `pnpm build` | exit 0 |
lz added 23 commits 2026-09-22 10:23:40 +02:00
Two things the layer design and the roadmap handoff assert are wrong, and both
were load-bearing for scope.

Deleting the workspace tick does NOT delete the session-expiry detector.
sessionStats is refcounted with its own timer and QuotaBadge subscribes from
Header.svelte, which mounts on every page the operator can browse. Nothing
needs to be built for it.

Sessions on the stream is not #155's territory. #155 is an MCP tool; the work
of giving sessions a server-side store and notifying from the attention
callback is unowned, and no open issue in milestone 27 covers it.

Two constraints derived here rather than inherited. The subscriber list cannot
live on FleetStore: provisioning.ts must notify, and fleet/observe.ts already
takes a value import from workers/provisioning.js, so that placement would wire
the reverse edge fact #31 exists to prevent. And the client gains a second
writer for workspace state, which is the completion-order defect this arc
already shipped once server-side.

The transport was spiked against the built artifact rather than assumed: SSE
survives handle -> adapter-node -> the custom createServer with 2-3ms latency
and no buffering, and teardown is socket-close-driven (handler.js:1179 wires
res.on('close') to reader.cancel()), so no heartbeat is needed for leak-safety.
A server-stamped sequence needed /api/workers to carry it, which means either a
response-shape change or a header the client layer does not currently read. A
streamAuthoritative flag needs neither: the snapshot is a full roster
replacement, so it can supersede an in-flight refresh through the generation
counter that already exists, and the poll remains the fallback path unchanged.
It is a bulk UPDATE that returns numUpdatedRows and never learns which rows it
flipped, and it runs at boot before any SSE client can have subscribed. The
plan asked for ids to be selected purely to feed a fan-out with no listeners.
finishProvisioning, resetForRetry, claimForRecreate and beginRemoval each return
numUpdatedRows because their where carries a scoping predicate that can
legitimately match nothing. Announcing a refused write says the workspace
changed when it did not, and finishProvisioning returning 0 is the signal that
routes provisionWorker into its orphan-teardown branch.
finishBooting and releaseRecreateClaim return void but both carry a
phase-scoped where. finishBooting's own docblock says it exists so it can
never resurrect an error row or race a concurrent phase write, so zero rows is
a meaningful outcome. observeWorkspace calls it on every silent probe, making
an unguarded touch a delta to every open tab six times a minute for a write
that did nothing.

Separately, one observeWorkspace pass touches twice: a phase write, then the
store's upsert. The SSE route now coalesces per microtask so that is one query
and one frame, and a forget clears any touch still pending for that id.
Nine of the ten writers carry a scoping predicate that can match nothing, so
the guarded-vs-unconditional split was a guess; three separate revisions of it
put a writer on the wrong side. Every writer now notifies only when its write
landed, and the meta-test enforces the form rather than the presence of a call
— asserting a touch exists passes over an unguarded one, which is the defect.
Skipping the roster fetch is not enough. refresh() builds into locals and
commits atomically at the end, so a stale local still overwrites what the
stream delivered during the await — the write race the design removes,
reintroduced one layer down. It also has to drop workerHealth from the
per-worker fan-out, which is N of the N+1 requests this PR removes.
Subscribing before the snapshot query starts prevents a signal being missed; it
does not order the two frames. They are independent promise chains, so a
workspace or forget frame can precede the snapshot on any connection.

A delta landing first is discarded by the client, which is what makes
applyDelta's guard load-bearing. A forget landing first is applied and then the
snapshot resurrects the workspace if its query read the roster before the row
was deleted — and no further signal is coming, so the card lingers.
Everything here is measured against the built artifact rather than reasoned:
the 2ms transition, the request accounting, the socket-close teardown that
makes a heartbeat unnecessary, and the frame timing through the custom entry.

The four that would otherwise be rediscovered the hard way: the bus must be a
leaf or it wires the reverse import edge in a graph neither build warns about;
a phase writer must announce only a write that landed, and the meta-test must
check the form rather than the presence; existence comes from the row, because
a store forget also fires when a recreate destroys a container whose row
survives; and the client must read the roster under untrack(), where a tracked
read is an unbounded effect loop that no unit test can observe.
fix(ui): stop a snapshot discarding the sessions a refresh had fetched
All checks were successful
ci / nexus (pull_request) Successful in 4m7s
ci / images (pull_request) Successful in 17m33s
85a8ddd84b
lz added this to the MCP support (#140) milestone 2026-09-22 10:23:48 +02:00
lz added 2 commits 2026-09-22 11:52:46 +02:00
A five-agent review found three critical defects and nine mutations the suite
survived. They share a root cause: the 10s poll was the client's only repair
mechanism and nothing replaced it. The server reconciles its own store every
60s; the client reconciled never, so every divergence became permanent rather
than bounded. #192's own lesson — a watch owes its watch a resync — was applied
to the server and not one layer up.

The client now resyncs its roster every 60s even while the stream is
authoritative. Roster only: health reaches the client through the stream and the
server's reconcile touches every workspace, so a missed health delta repairs
itself; what cannot is existence, because nothing re-announces a row that is
gone. A resync overtaken by a delta discards its own result, since the delta is
newer than what the fetch returned.

A completed removal now announces itself. runTeardown's row delete raised no
signal at all — fleetBus was not even imported there — while the container
destroy fires seconds earlier, resolving to a workspace delta because the row
still exists, and emptying the store entry so reconcileFleet's orphan sweep had
nothing left to name. The card stuck on "Removing…" in every open tab until a
reload, and only when everything went right: with no container there is no
destroy event, and the sweep cleared it correctly.

A failed snapshot query no longer fabricates an empty fleet. That frame was
wire-indistinguishable from a genuinely empty one, so the client set
streamAuthoritative over an empty roster, disarmed its poll, and marked every
scope dead — and the dock's panel prune is debounced into localStorage, so a
reload restored the emptied layout. The connection now ends instead, which
reaches the client's own onerror and resumes the poll. It closes rather than
errors because controller.error() discards queued chunks, which would have lost
the retry hint enqueued immediately before it; and it releases the subscription
by hand, because cancel() runs only when the CONSUMER cancels.

The three SSE event names are declared once in $lib/api/types.ts and imported by
both sides. EventSource silently ignores an event type nobody registered for, so
drift left the snapshot listener firing, the poll disarmed and the roster frozen
with the page looking healthy — invisible to every gate, since connect() is
reachable only from a component and vitest runs environment: node.

FleetSignal loses its kind. Nothing read it, and its `forget` was a false
cognate for the wire's: the bus fires when a CONTAINER is destroyed, whose row
usually survives a recreate. One signal, and the row it makes the reader query
is what decides existence.

The provisioning meta-test is behavioural rather than syntactic. It asked
whether the notify line also carried `> 0`, which any unrelated predicate
satisfied and which rejected a semantically identical guard-clause rewrite; and
its enumeration could not see a writer declared in another form, so an
arrow-function writer with no notify at all passed. Every writer is now called
against a deleted row and must stay silent, and every export must be a known
writer or a named exemption.

Tests added for the surviving mutations: a delta with null health must not erase
held health (it fires on every spawn); a forget must drop that workspace's
sessions; each connection needs its own coalescing buffer; the buffer must drain
between flushes. Every one was proven by re-breaking the code it guards.
fix(fleet): keepalive the stream, and correct what fact 42 claims
All checks were successful
ci / nexus (pull_request) Successful in 6m21s
ci / images (pull_request) Successful in 8m17s
pr-image-cleanup / delete-pr-images (pull_request) Successful in 19s
e0605c5bfa
A half-open socket — NAT or proxy idle timeout, VPN re-key, suspend — is
invisible to the browser when neither side writes, which is the steady state
here by design. onerror never fires, the client stays authoritative, and it
serves the last-known roster indefinitely. A comment frame every 25s is the
write that eventually fails, which is what reaches the client's error path. It
is cleared on the snapshot-failure path too, since close() does not invoke
cancel().

Fact 42 claimed a snapshot bumps generation. The branch's own last commit
removed that deliberately, one commit after the fact was written, and a future
agent trusting the doc over the code would have reintroduced an empty sidebar on
every page load. It also generalised the socket-close measurement into no
heartbeat is needed, which answered a different question than the one that
matters.

Added: EventSource only reconnects on a network failure or an aborted
established stream, so a 500 or a 401 handshake is terminal; controller.error()
discards queued chunks where close() flushes them; FLEET_EVENTS is declared once
because a drifted name freezes the roster invisibly; and the provisioning guard
is behavioural now, not syntactic.
lz changed target branch from feat/fleet-status to main 2026-09-23 10:36:49 +02:00
lz merged commit 592196b9c1 into main 2026-09-23 10:38:35 +02:00
lz deleted branch feat/fleet-sse 2026-09-23 10:38:36 +02:00
Sign in to join this conversation.
No reviewers
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
lz/agent-nexus!193
No description provided.