#177c: observe claude liveness server-side, instead of announcing it from the launch line #195

Merged
lz merged 7 commits from feat/session-exit-reporter into main 2026-09-23 10:39:29 +02:00
Owner

Fourth of the #177 stack. Stacked on feat/fleet-sse (#193) — merge #188, #192, #193 first or this will not compile.

What this fixes

checkSessionHealth has exactly one caller, GET /api/workers/:id/sessions/:sid/health, and only a browser polls it. So with no browser open, Nexus never learned that a session's claude had died. #177 closed that hole for workspace state; it was still open for sessions, and it is the same hole MCP exists to be on the other side of.

The plan changed, and why

#177c was specified as nexus-report-exit: a reporter on the launch line posting to a new /api/agent/session-exit. Two measurements against the installed claude 2.1.278 (not the docs) took its justification away, and a third showed it was the wrong instrument:

  1. SessionEnd is not graceful-exit-only. shutdown(e = 0, n = "other", r) awaits executeSessionEndHooks before forceExit, and SIGTERM (143), SIGHUP (129) and uncaughtException (1) all route into it. Driven end to end: a real claude --print SIGTERM'd mid-run exits 143 and emits {"hook_event_name":"SessionEnd","reason":"other"}, which the plugin's existing matcher already accepts. SIGKILL is the negative control — nothing fires. The uncovered set is SIGKILL/SIGSEGV/OOM, not "a crash".
  2. The "~30 fewer execs" argument is void. Those are waitForClaude's poll; two of its four call sites wait for claude to be present, which a reporter cannot answer, and the other two go through respawn-window -k, which the spec itself says kills the wrapper. Net saving: zero.
  3. It covers the case convergence ignores. The reporter fires when claude exits and the wrapper survives — which is when the tmux WINDOW survives too, because of ; exec bash. Convergence is keyed on the window. Measured, the report also never fires for restartSession (both actions), closeWorkspaceSession, openWorkspaceSession onto an existing window, deleteSession, container stop, or tmux server death.

The wrapper is still a defensible latency optimisation for the SIGKILL set. It is filed as #194 with every measurement attached, including the three things it must get right.

What shipped instead

Liveness as an observation, taken in the exec pass convergeSessions already makes:

  • probeLiveClaudeCwds is probeClaudeCwd's walk with the cwd filter removed and the directory echoed — one exec answers every session of a workspace instead of one each. Session cwds are unique by construction, so the set maps straight onto rows. The self-match immunity is unchanged and never depended on the filter that went: the probe's own cmdline begins /bin/sh, which the claude| test rejects.
  • sessions.claude_alive (migration 0023, unclaimed on every branch in the stack) is NULL until observed. A failed exec writes nothing — uncertainty must not assert absence, the rule the worktree listing already follows.
  • It is forgotten on the two paths where it stops describing anything: a session just relaunched, and a workspace no longer running. Both mirror what attention already does, for the same reason.
  • sessionMark folds the two sources in liveness(), probe first: the probe is fetched on demand and is seconds old, the observation can be a resync interval behind. The observation is consulted when there is no probe, and when the probe says 'unknown' — that it could not look.
  • mcpSession carries it deliberately. mcpSession is a field-by-field allowlist, so a new column reaches MCP only on purpose — and a caller with no browser open is exactly the caller for whom nothing else answers this. status is intent; attention_state is a report a crashed agent cannot send.

No worker script, no new route, no AGENT_CALLBACKS entry, no worker image release — so it works on every existing workspace the day it merges, which the wrapper by construction could not. It also covers a hung claude and the container-stop case. The cost is latency: bounded by the resync interval rather than immediate.

Docs corrected in place

The false premise was written down in four places and would have been re-derived. Fact #26 now records what SessionEnd actually covers, with the signal-handler evidence; fact #15 loses its claim that a dying claude "closes a tmux window" (it does not — ; exec bash is there so the pane survives, and that sentence is what made a reporter look like the missing edge); fact #5 gains the second /proc walk and why both keep the claude| test; the fleet-status spec's Worker changes section is marked superseded rather than deleted, because the goal was right and the instrument was not.

A note for whoever edits the parity suite

attention.test.ts's generated suite now has the observation as a dimension. The byte-for-byte promise holds only while BOTH inputs are unobserved, and a second generated block pins the probe-first precedence in both directions — the null-only suite could not see that, because it only ever passed null.

Not done

Sessions are still not on the SSE stream. That is an explicit non-goal of #193, one commit below this, and needs a server-side session store plus a notify in the attention writer. This changes whether the answer exists with no browser open, not how fast it arrives; the SSE spec now says so where a reader would otherwise assume this closed it.

Fourth of the #177 stack. **Stacked on `feat/fleet-sse` (#193) — merge #188, #192, #193 first or this will not compile.** ## What this fixes `checkSessionHealth` has exactly one caller, `GET /api/workers/:id/sessions/:sid/health`, and only a browser polls it. So **with no browser open, Nexus never learned that a session's claude had died.** #177 closed that hole for workspace state; it was still open for sessions, and it is the same hole MCP exists to be on the other side of. ## The plan changed, and why #177c was specified as `nexus-report-exit`: a reporter on the launch line posting to a new `/api/agent/session-exit`. Two measurements against the installed claude **2.1.278** (not the docs) took its justification away, and a third showed it was the wrong instrument: 1. **`SessionEnd` is not graceful-exit-only.** `shutdown(e = 0, n = "other", r)` awaits `executeSessionEndHooks` before `forceExit`, and SIGTERM (143), SIGHUP (129) and `uncaughtException` (1) all route into it. Driven end to end: a real `claude --print` SIGTERM'd mid-run exits 143 **and** emits `{"hook_event_name":"SessionEnd","reason":"other"}`, which the plugin's existing matcher already accepts. SIGKILL is the negative control — nothing fires. The uncovered set is SIGKILL/SIGSEGV/OOM, not "a crash". 2. **The "~30 fewer execs" argument is void.** Those are `waitForClaude`'s poll; two of its four call sites wait for claude to be *present*, which a reporter cannot answer, and the other two go through `respawn-window -k`, which the spec itself says kills the wrapper. Net saving: zero. 3. **It covers the case convergence ignores.** The reporter fires when claude exits and the wrapper survives — which is when the tmux WINDOW survives too, because of `; exec bash`. Convergence is keyed on the window. Measured, the report also never fires for `restartSession` (both actions), `closeWorkspaceSession`, `openWorkspaceSession` onto an existing window, `deleteSession`, container stop, or tmux server death. The wrapper is still a defensible *latency* optimisation for the SIGKILL set. It is filed as **#194** with every measurement attached, including the three things it must get right. ## What shipped instead Liveness as an **observation**, taken in the exec pass `convergeSessions` already makes: - `probeLiveClaudeCwds` is `probeClaudeCwd`'s walk with the cwd filter removed and the directory echoed — **one** exec answers every session of a workspace instead of one each. Session cwds are unique by construction, so the set maps straight onto rows. The self-match immunity is unchanged and never depended on the filter that went: the probe's own cmdline begins `/bin/sh`, which the `claude|` test rejects. - `sessions.claude_alive` (**migration 0023**, unclaimed on every branch in the stack) is NULL until observed. A failed exec writes **nothing** — uncertainty must not assert absence, the rule the worktree listing already follows. - It is forgotten on the two paths where it stops describing anything: a session just relaunched, and a workspace no longer running. Both mirror what attention already does, for the same reason. - `sessionMark` folds the two sources in `liveness()`, **probe first**: the probe is fetched on demand and is seconds old, the observation can be a resync interval behind. The observation is consulted when there is no probe, and when the probe says `'unknown'` — that it could not look. - `mcpSession` carries it deliberately. `mcpSession` is a field-by-field allowlist, so a new column reaches MCP only on purpose — and a caller with no browser open is exactly the caller for whom nothing else answers this. `status` is intent; `attention_state` is a report a crashed agent cannot send. No worker script, no new route, no `AGENT_CALLBACKS` entry, **no worker image release** — so it works on every existing workspace the day it merges, which the wrapper by construction could not. It also covers a hung claude and the container-stop case. The cost is latency: bounded by the resync interval rather than immediate. ## Docs corrected in place The false premise was written down in four places and would have been re-derived. Fact **#26** now records what `SessionEnd` actually covers, with the signal-handler evidence; fact **#15** loses its claim that a dying claude "closes a tmux window" (it does not — `; exec bash` is there so the pane survives, and that sentence is what made a reporter look like the missing edge); fact **#5** gains the second `/proc` walk and why both keep the `claude|` test; the fleet-status spec's Worker changes section is marked **superseded** rather than deleted, because the goal was right and the instrument was not. ## A note for whoever edits the parity suite `attention.test.ts`'s generated suite now has the observation as a dimension. The byte-for-byte promise holds only while BOTH inputs are unobserved, and a second generated block pins the probe-first precedence in both directions — the null-only suite could not see that, because it only ever passed null. ## Not done Sessions are still not on the SSE stream. That is an explicit non-goal of #193, one commit below this, and needs a server-side session store plus a notify in the attention writer. This changes whether the answer **exists** with no browser open, not how fast it arrives; the SSE spec now says so where a reader would otherwise assume this closed it.
lz added this to the MCP support (#140) milestone 2026-09-22 13:40:38 +02:00
lz added 2 commits 2026-09-22 13:40:38 +02:00
Nothing server-side knew whether a session's agent was alive. `checkSessionHealth`
has one caller — the per-session health route — and only a browser polls it, so
with no browser open a crashed claude read as `running` indefinitely. #177 closed
that hole for workspaces; this closes it for sessions.

`convergeSessions` already execs into every running container, so the walk rides
that pass: `probeLiveClaudeCwds` is `probeClaudeCwd`'s walk with the cwd filter
removed and the directory echoed, answering every session of a workspace in one
exec rather than one each. Session cwds are unique by construction, so the set
maps straight onto rows. The self-match immunity is unchanged and never depended
on the filter that went — the probe's own cmdline begins `/bin/sh`.

`sessions.claude_alive` (migration 0023) is NULL until observed, and a failed
exec writes nothing: uncertainty must not assert absence, the rule the worktree
listing already follows. It is forgotten on the two paths where it stops
describing anything — a session just relaunched, and a workspace that is no
longer running — mirroring what attention already does and for the same reason.

`sessionMark` folds the two sources in `liveness()`, probe first: the probe is
fetched on demand and is seconds old, the observation can be a resync interval
behind, so it must never override a fresh answer. It is consulted when there is
no probe, and when the probe says 'unknown' — that it could not look.

The generated parity suite gains the observation as a dimension: the
byte-for-byte promise is restated as holding when BOTH inputs are unobserved,
and a second generated block pins the precedence in both directions.

`mcpSession` carries it deliberately. A caller with no browser open is exactly
the caller for whom nothing else answers "is this agent alive" — `status` is
intent, and `attention_state` is a report a crashed agent cannot send.
docs: correct what SessionEnd covers, and why the exit reporter was not built
Some checks failed
ci / nexus (pull_request) Successful in 5m59s
ci / images (pull_request) Has been cancelled
c1c6b75152
Fact #26 and the fleet-status spec both said `SessionEnd` fires on a graceful
exit only. Measured against the installed 2.1.278, that is false: `shutdown(e=0,
n="other", r)` awaits `executeSessionEndHooks` before `forceExit`, and SIGTERM
(143), SIGHUP (129) and uncaughtException (1) all route into it. A real
`claude --print` SIGTERM'd mid-run exits 143 AND emits
`{"hook_event_name":"SessionEnd","reason":"other"}`, which the plugin's existing
matcher already accepts. SIGKILL is the negative control: nothing fires. So the
uncovered set is SIGKILL/SIGSEGV/OOM, not "a crash" — and the full hook event
set in that build has no abnormal-exit event at all.

That premise is why #177c was specified as a launch-line exit reporter, so the
spec section is marked superseded rather than deleted: the goal was right and the
instrument was not. The reporter fires exactly when claude exits and the wrapper
survives, which is when the tmux window survives too (`; exec bash`) — and
convergence is keyed on the window, so it covers the case convergence ignores.
Measured, it also never fires for restartSession, closeWorkspaceSession,
openWorkspaceSession onto an existing window, deleteSession, container stop or
tmux server death.

The spec's second justification is void too: the "~30 fewer execs" are
`waitForClaude`'s poll, two of whose four call sites wait for claude to be
PRESENT and the other two kill the wrapper. The reporter removes zero.

Fact #15 loses its claim that a dying claude "closes a tmux window" — it does
not, and that sentence is what made an exit reporter look like the missing edge.
Fact #5 gains the second `/proc` walk and why both keep the `claude|` test.
fix(sessions): forget liveness when the workspace shell is respawned too
Some checks failed
ci / images (pull_request) Has been cancelled
ci / nexus (pull_request) Has been cancelled
edd8c78e72
`openWorkspaceSession` returns `alreadyOpen: false` when it respawned the pane,
so the workspace shell has a relaunch path of its own — and the observation
taken moments earlier in the same loop is exactly as stale there as it is on the
restore branch. Left standing, a workspace shell Nexus has just restarted reads
as a dead agent until the next pass.

The asymmetry is easy to miss because both relaunches sit in one loop and only
one of them looks like a relaunch: the other is a call whose result was
discarded.

The converge fake also had no `respawn-window` branch, so that exec threw and the
caller's catch made a respawn indistinguishable from "already open" — the exact
state this keys on. A fake that cannot express the case cannot fail for it.
style: cut the comment density on the liveness change
All checks were successful
ci / nexus (pull_request) Successful in 7m10s
ci / images (pull_request) Successful in 26m32s
280f047b5d
Explanation left only where the code cannot carry it: why dropping the cwd
filter keeps the self-match immunity, why a failed look writes nothing, and the
probe-before-observation precedence in `liveness()`. Everything restating the
line below it, or narrating how the change came about, is gone.
fix(sessions): close five liveness defects the review found
Some checks failed
ci / nexus (pull_request) Successful in 5m42s
ci / images (pull_request) Has been cancelled
9c2c213c97
**The liveness write could abandon the whole convergence pass.** It was the only
unguarded write in the loop and it ran first, so a rejected update — SQLITE_BUSY,
a locked table — escaped the loop and the `try`, cancelling every restore and
every orphan reap behind it. The hot-path caller catches at `debug` while
production logs at `info`, so nothing would have said so. It is the least
important write in that loop; it now catches like its neighbours.

**A walk that never ran was indistinguishable from an empty one.** An unreadable
`/proc` leaves the glob unexpanded, `head` fails on the literal, `|| continue`
swallows it and the script still exits 0 — writing "no claude" for every session
in the workspace. The script now echoes a completion sentinel and `ok` requires
it. `exit 0` also made the old `exitCode !== 0` arm nearly unreachable, so the
test drove a path production cannot produce; the rejected-exec arm is covered now.

**The workspace shell's clear was derived from the wrong thing.**
`openWorkspaceSession` returns `alreadyOpen: true` from its insert-race path and
nothing at all if it throws after respawning — both of which relaunch. The clear
now keys on what the walk observed, before the call, so a throw anywhere inside
cannot leave the row asserting a dead agent.

**`setSessionAgent` is the fourth relaunch path.** It confirms liveness with
`waitForClaude` and discarded the answer, so a restarted agent kept a stale `0`
until the next resync — up to 60s, since `FleetStore.silent()` excludes healthy
workspaces from the fast timer — and MCP reported a stopped agent as alive. Both
arms now record what they confirmed, and NULL when they confirmed nothing.

**`unhealthyReason` and the mark read one authority again.** `liveness()` is
exported and both use it; gating the fault line on `health.health` alone left an
observed-dead session painting red with nothing beside it, which on a phone —
no hover, no title — says only that something is wrong.

Also: `claude_alive` is typed `0 | 1 | null` at the schema so the migration's
prose is a type; `toDTO` reads an unexpected value as "not observed" rather than
"dead", the asymmetric-cost direction; `liveness()` returns the domain union
with an explicit 'unknown' so `sessionMark`'s third case is an arm with an
exhaustive check rather than a fall-through; and one writer replaces two.

Four mutations the earlier nine could not see, all of which SURVIVED before this
and are killed now: the shell's cwd computed as a worktree path, the per-row
write scoped to `worker_id`, `toDTO` collapsing NULL into `false`, and the
liveness write left unguarded. All 26 converge tests seeded exactly one session,
which is why the first two were invisible.
fix(sessions): the second review round — precedence, fail-open, and one walk
All checks were successful
ci / nexus (pull_request) Successful in 6m42s
ci / images (pull_request) Successful in 9m5s
c5259025d8
**A stale observation could overrule a deliberate exit.** `sessionMark` treated
an observed-live claude as proof that an `ended` report was stale, on the
reasoning that applies to the probe. It does not apply here: the observation can
be a resync interval old and for `ended` usually predates the report outright,
because the exit is what produced it. Only a probe may overrule it now. The test
that asserted otherwise was pinning the bug.

**`liveness()` failed open.** `observed === null` let an `undefined` fall through
to `observed ? … : 'unhealthy'` — "never looked" read as "observed dead", the
inversion fact #26 exists to prevent. Unreachable today; one character.

**A dead tmux server left liveness standing forever.** That early return
precedes the walk, so the observation was never refreshed again while the branch
four lines above cleared for exactly the same reason.

**`agentRunning` was a second answer to the same question**, probe-only, so the
mark could read `errored` while the ⋯ menu still offered "Stop agent". It and
`unhealthyReason` now call the exported `liveness()`, which is what makes the
"one authority" claim beside the mark true again.

**The /proc walk is one declaration.** Three probes carried their own copy of
the matcher that provides the self-match immunity, and two of them now feed one
decision — a matcher hardened in one copy would make a session's mark depend on
which probe answered.

Tests: the `creating` skip is pinned (an MCP caller polling during a spawn would
otherwise read a provisioning session as dead); the worker-scoped clear is
asserted over two sessions, since one cannot tell it from an id-scoped one; the
precedence block iterates `status`, which `sessionMark` branches on first,
instead of the probe fields it never reads; and two cases pin that a live
observation cannot outrank a stopped workspace or a stopped row.

The shell fixture now flips claude to running when it sees the respawn, which is
the real sequence — it had depicted a respawn whose claude never comes up, and
cost 3s of real wall clock starving `waitForClaude`. The single-cwd probe answers
for the path it was asked about rather than one global flag.

Docs: the MCP `describe` said "null = never looked" when null is also written on
purpose for a stopped workspace or a just-relaunched session, and repeated the
"a crash cannot send it" claim that #177c itself corrected. Fact #5 said two
walks. `observe.ts`'s docstring said claude liveness is not checked there, which
the converge call it sits above now does.
fix(fleet): clear stale session state where a stopped workspace is noticed
All checks were successful
ci / nexus (pull_request) Successful in 4m42s
ci / images (pull_request) Successful in 7m45s
pr-image-cleanup / delete-pr-images (pull_request) Successful in 31s
553f0013ff
`convergeSessions` cleared attention and liveness for a workspace that is not
running, and could not reach that branch: `observeWorkspace` is its only
unprompted caller and returns for an absent or non-running container before ever
calling it. So a stopped workspace kept its last attention report and its last
liveness observation indefinitely. The UI masks both — it knows the workspace is
down — but `mcpSession` does not, so a model asking what is running was told a
workspace stopped days ago had a live agent mid-turn.

The attention half has been dead since the fleet layer replaced
`checkWorkerHealth`; this PR put a sibling beside it without noticing. Both are
fixed together because it is one call site and because two adjacent fields with
different reachability is the state that produced this.

`forgetWorkerSessionState` is now called from the three places `observeWorkspace`
decides a workspace is not running — an absent container, a non-running one, and
a 404 from inspect. That is where every route to "stopped" arrives: the
operator's stop, a container that exited on its own, one stopped outside Nexus,
one the OOM killer took. `stopWorker` would have caught only the first.

`convergeSessions` keeps its own branch as belt-and-braces — the container can
stop between the two checks — but its test no longer stands in for a path
production cannot take; `observe.test.ts` drives the real one.

Both clears are scoped `is not null`. That is an optimisation, not a correctness
property: unscoped they rewrite NULL over NULL for every stopped workspace on
every resync, forever. It has no test and dropping it is not a regression — a
mutation confirms the suite stays green without it.
lz changed target branch from feat/fleet-sse to main 2026-09-23 10:38:36 +02:00
lz merged commit 8ce44727e9 into main 2026-09-23 10:39:29 +02:00
lz deleted branch feat/session-exit-reporter 2026-09-23 10:39:30 +02:00
Sign in to join this conversation.
No reviewers
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
lz/agent-nexus!195
No description provided.