feat(fleet): push workspace state to the browser over SSE (#177b) #193
No reviewers
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
lz/agent-nexus!193
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/fleet-sse"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes the UI half of #177 (milestone 27). Targets
feat/fleet-status(#192), which targetsfeat/daemon-container-health(#188) — notmain. This is a three-deep stack; merging out of order will not compile. #188 → #192 → this.What this does
Workspace state stops being polled.
Before,
routes/+page.sveltebumped atickevery 10s andworkspaces.refresh()answered it with 1 roster + 1 previews + 2 × N_running requests — sessions and health per running workspace. At five running workspaces that is 12 requests every 10s, in every open tab, forever. A container transition was visible up to 10s late, and later still if the operator happened to be typing a session name (the tick skips itself to avoid disrupting input).#177a made the server hold
live_statusand health in aFleetStore. This PR pushes them.lib/server/events/bus.ts— a leaf fan-out.subscribe/touch/forget, behind aSymbol.forslot.lib/server/events/payload.ts—fleetSnapshot(literallylistWorkers(db, store)plus the store's health map) andworkspaceDelta. No new wire types: the payload isWorkerDTOandWorkerHealthDTO, so none of the repo's declared-on-both-sides parity rules apply.GET /api/events— SSE, cookie-authenticated, snapshot then per-workspace deltas, coalesced per microtask.workspaces.svelte.ts—connect(),applySnapshot/applyDelta/applyForget, and astreamAuthoritativeflag that decides which single path owns the roster.Sessions stay on the tick.
attention_stateis written by the worker's hooks viaPOST /api/agent/events, which touches no workspace, so a session goingneeds-youraises no fleet signal and has nothing to ride. Putting it on the stream needs a server-side session store plus a notify in that callback — unowned work; no open issue in milestone 27 covers it, and #155 is an MCP tool, not an SSE feature.Measured against the built artifact
Everything here was driven through a browser against
node build/server.jswith real Docker containers, not reasoned about.docker stop→ DOM mutation/api/events(stays open), one/api/workerson mount, zero roster/health after/api/workers0,/api/workers/:id/health0; sessions 10 (2 workspaces × 5 ticks), previews 5, quota 5/api/quotacadence under loadprovision_detailadvancing every ~10s onSILENT_PROBE_INTERVAL_MS, no HTTP request accompanying any of itprovision_percent18 → 27 → 35 → 44 → 52 → 60 → 69 → 78 → 86 → 94 → 99 → 100 across 27pullingdeltas, spaced 760–846 mssnapshotthen 28workspace— snapshot first, as designed/api/eventsunauthenticatedThe pull half was driven by spawning against an image absent from the daemon, forcing a real
docker pull. It confirms the claim the spec made by reasoning:PULL_WRITE_INTERVAL_MS(750 ms) is inherited by the SSE emit for free, because it already gates the call intosetProvisionProgress— the measured 760–846 ms spacing is that throttle, not a second one. The run ended inrecordProvisionError("not a Nexus worker image (missing the org.nexus.worker label)"), so the terminal error phase reached the stream too. That exercises three of the ten phase writers end to end —advanceProvisionPhase,setProvisionProgress,recordProvisionError.Two transport facts, measured because a streaming response through a custom Node entry is exactly the kind of thing that works in dev and not in production: frames arrive 2–3 ms after
enqueuethroughhandle→ adapter-node →src/server.ts'screateServer, and teardown is socket-close-driven (build/handler.jswiresres.on('close')toreader.cancel()), so a closed tab cancels its stream while idle and no heartbeat is needed. Idle is the steady state here by design, so a write-triggered teardown would have leaked a subscriber per closed tab and no test that writes would have caught it.Five defects found during implementation, not by review
Each of these was in the gap between the design and what the code actually does. None would have been caught by reading the spec.
A phase writer must announce only a write that LANDED. Nine of the ten
provisioning.tswriters carry a scoping predicate that can match nothing.notBeingRemovedis the one that bites: it exists so a Remove clicked during a pull is not overwritten, and in exactly that scenario the pull loop callssetProvisionProgressevery 750ms with every call refused — each shipping "this workspace changed" to every tab while the card should read "Removing…". An earlier revision split the writers "guarded vs unconditional" and was wrong three times running; there is now one rule and no classification.provisioning-notify.test.tsenforces the form — the line carryingfleetBus().touch(must also carry> 0— because asserting the call merely exists passes over an unguarded one.Existence is decided by the row, never by the signal.
FleetStore.forgetalso fires when a container is destroyed duringstartWorker's recreate (service.ts:851-862), where the row survives. Forwarding that as "drop this workspace" made a card vanish on Stop/Start. Both signal kinds now runworkspaceDeltaand itsnullis the single authority.The snapshot must be written before any delta. Subscribing before the snapshot query starts prevents a signal being missed; it does not order the two frames, which are independent promise chains. A
forgetarriving first was applied and then the snapshot resurrected the workspace, permanently. Signals now buffer until the snapshot has been sent — including on the snapshot's error path, or a failed query parks every later signal for the life of the connection, silently.The client reads the roster under
untrack(). A tracked read insiderefresh(), which is called from an$effect, is not merely an extra fetch per delta — the authoritative branch readssessionsByWorkerwith no preceding await and the commit then writes it, which iseffect_update_depth_exceeded.$effectis inert under vitest's node environment (Svelte resolves to its SSR build), so no unit test can observe this; it was confirmed in a browser across 16 transitions in 4.0s with a page-level error collector, since Svelte throws that error rather than logging it.A snapshot must not discard an in-flight refresh's sessions. Bumping
generationinapplySnapshotsuperseded the whole commit, includingsessionsByWorkerandpreviews— state the stream does not carry — leaving the sidebar's session list empty for a full 10s on every page load. Time to a visible session name: 10409 / 10389 ms before, 97 / 92 / 80 ms after. ThestreamAuthoritativeguard already protects the roster; the bump protected nothing and cost a tick of session state. Restoring the bump kills the new test and leavesa refresh that started before a delta does not clobber itgreen — which is the proof the bump was never what protected the roster.The mechanism differs between environments, same outcome: in the unit-test timing the bump lands before
listWorkersresolves, so the session fan-out never fires at all; in a browser the fan-out has already been issued and only its commit is thrown away. The mount race above is unchanged by the fix — what changed is that the mount refresh's results now survive.Known limits, stated rather than buried
refresh()can still issue 0–2/api/workers/:id/healthrequests when the snapshot lands mid-fan-out. Bounded to page load, self-correcting, and replacing N every 10s forever. Accepted rather than closed with machinery.sessionsByWorkerentry pruned at the commit; self-corrects on the next tick, and a just-arrived workspace has no sessions to lose.Verification
Gates, each run as its own command:
pnpm lintpnpm typecheckpnpm vitest runpnpm buildpnpm buildmatters here specifically: it is the only gate that validates SvelteKit route-module exports (fact #32), and this PR adds a route.Fourteen mutations were run against the guards this PR adds, each killed, each file restored from a
cpbackup anddiffed byte-identical afterwards. That is not the same as "every guard was proven to fail", and an earlier revision of this body said so incorrectly — a review subsequently found nine mutations the suite SURVIVES (listed below). Two are worth naming because they nearly recorded the wrong result: one mutation duplicated a block instead of moving it, passing as a survival until thediffshowed46a47,56; and the original ordering test asserted.sort()-ed membership, which cannot distinguish sequence from set, so it would have survived an inverted-order mutation. Both were caught and the test was converted to assert the sequence explicitly.AGENTS.md facts #42 and #26 amended.
Review findings — all fixed
A five-agent review (correctness, silent failures, tests, comments, type design) found three critical defects and nine mutations the suite survived. All are fixed on this branch; every fix was proven by re-breaking the code it guards.
The root cause was architectural. The 10s poll was the client's only repair mechanism and nothing replaced it. The server reconciles its own store every 60s; the client reconciled never — so every divergence became permanent instead of ≤10s stale. #192's own lesson, that a watch owes its watch a resync, had been applied to the server and not one layer up.
The client now resyncs its roster every 60s even while the stream is authoritative. Roster only: health arrives via the stream and the server's reconcile keeps pushing it, so a missed health delta self-heals; what cannot is existence, because nothing re-announces a row that is gone. A resync overtaken by a delta discards its own result. Cost is roughly one request per minute; the 2 ms transition latency and the removal of the per-workspace health fetch are untouched.
The three criticals
runTeardown's row delete raised no signal —fleetBuswas not imported inservice.tsat all — while the containerdestroyfires seconds earlier and resolves to aworkspacedelta because the row still exists, and empties the store entry soreconcileFleet's orphan sweep had nothing left to name. The card stuck on "Removing…" in every open tab until a reload, and only when everything went right: with no container there is no destroy event and the sweep cleared it correctly. Measured across a fullremoveWorker: one signal,beginRemoval's claim.streamAuthoritative, disarmed its poll, and marked every scope dead — and the dock's panel prune is debounced intolocalStorage, so a reload restored the emptied layout. The connection now ends instead, reaching the client's ownonerror. It closes rather than errors becausecontroller.error()discards queued chunks, which would have lost theretry:hint enqueued immediately before it; and it releases its subscription and keepalive by hand, becausecancel()runs only when the consumer cancels.EventSourcesilently ignores an event type nobody registered for — no throw, noonerror— so a drifted name left the snapshot listener firing, the poll disarmed and the roster frozen with the page looking healthy and nothing logged at either end. Invisible to every gate. They are now one exported constant in$lib/api/types.ts, imported by both sides.Also fixed
FleetSignal.kindis gone. Nothing read it, and itsforgetwas a false cognate for the wire's — the bus fires when a container is destroyed, whose row usually survives a recreate.> 0: any unrelated predicate satisfied it (message.length > 0passed), it rejected a semantically identical guard-clause rewrite, and its enumeration could not see a writer declared in another form — an arrow-function writer with no notify at all passed every case. Every writer is now called against a deleted row and must stay silent, and every export must be a known writer or a named exemption.streamLost()on unmount (close()fires noonerror);connect()'s docblock corrected — only a network failure or an aborted established stream reconnects, so a 500 or 401 handshake is terminal; the$effectimplicit-return-as-cleanup trap documented.Correction to this PR's own claims
An earlier revision of this body said "every guard was re-broken and proven to fail." That was true of the fourteen guards tested and false as a general claim. AGENTS.md fact #42 also stated a snapshot bumps
generation— the branch's own last commit had removed that deliberately, one commit after the fact was written. Both corrected.Verification
pnpm lintpnpm typecheckpnpm vitest runpnpm build