feat(workers): show real startup progress, and let a failed provision retry #99

Merged

lz merged 5 commits from feat/startup-visibility into main 2026-09-04 17:24:15 +02:00

Conversation 0 Commits 5 Files changed 19 +1142 -67

lz commented 2026-09-02 16:29:51 +02:00

Owner

Copy link

Stacked on #98 — targets fix/mobile-sidebar-scroll, not main. Merge #98 first. Second of the four PRs in the issue #96 spec (design D2); independent of the sidebar work.

The gap

finishProvisioning cleared provision_phase the instant container.start() returned. The entrypoint then spent another 30–90s on git identity, clone, npm install -g @anthropic-ai/claude-code@latest and finally tmux new-session — during which the card read running · healthy and every session action failed with no explanation on screen.

What's in

A booting phase that survives finishProvisioning and is cleared by checkWorkerHealth on the first successful tmux has-session probe — tmux answering is the definition of usable. While booting, the card shows the last ==> marker the entrypoint printed. entrypoint.sh already emits those, so no worker-side change was needed.

Pull progress. pullWorkerImage already yielded per-layer {id, current, total}; provisionWorker discarded all of it except as a boolean "has anything happened yet" flag. PullProgress folds those into a percentage and a layer/byte line, throttled to one DB write per 750ms (docker emits per layer per chunk) with a forced final write so the bar cannot freeze mid-percentage. Percent is null — an indeterminate bar — whenever any layer lacks a known total, rather than a number that races upward and jumps back as sizes arrive.

Retry on a failed card. It cannot restart anything: provisionWorker tears the container down on every failure path and container_id was never written. So retryWorker re-enters the pipeline from queued on the same row, keeping the frozen mounts_json snapshot it was created with (the operator retried this workspace, not a new one). It needs the master key to rebuild the sealed secret block, so it is an operator route and never something convergeSessions could do. The provider is resolved before the row is claimed, so a retry that fails early leaves the row in error and still retryable.

Two traps worth calling out in review

failStuckProvisioning must SPARE booting. Nothing in Nexus drives that phase — the container is started and its entrypoint runs to completion whether or not Nexus is up. Sweeping it would errored-out every workspace that happened to be booting across a Nexus restart, when the next health poll would have cleared it correctly. Verified live: on a harness restart the two pulling rows failed and the booting row was untouched.

Session controls are gated on the phase, not just on running. Sessions, the shell agent and the view panels all need tmux; leaving them live during boot just moves the failure one click later. isReady = isRunning && provision_phase !== 'booting'.

Verification

Browser-driven against a seeded instance, one card per state:

state	rendered
pulling, all sizes known	determinate bar at 62%, 4 of 7 layers · 218.0 MB / 351.0 MB
pulling, a size unknown	indeterminate bar, no percentage
booting	Starting up…, entrypoint marker, no "start agent", "Sessions appear once the workspace finishes starting."
error	reason + Retry · Logs · Remove
steady	unchanged

Also exercised the route directly: POST /api/workers/w3/retry on a booting row → 400 only a failed workspace can be retried; a retry that failed at provider resolution left the row's original image pull stalled… message intact.

pnpm test 109 files / 1028 tests pass (+15; the 5 pre-existing provisioning.test.ts cases are preserved, 2 amended where the behaviour genuinely changed) · pnpm run typecheck 4859 files, 0 errors · pnpm run lint clean.

Adding 'booting' to ProvisionPhase made provision-label.ts's exhaustive switch and two DTO fixtures fail to compile until updated — the guard working as intended.

Migration 0017_provision_detail.ts adds two nullable display-only columns. No backfill: a null pair is exactly the "nothing reported" state the card already renders.

---

Added after the spec audit

A five-agent pass over the shipped code against the spec found two D2 gaps:

• The five-segment step strip (queued / pulling / creating+starting / booting / ready) is in every provisioning frame of the mock and was absent. creating and starting share a segment on purpose — both are sub-second docker calls, and a segment that lights for 40ms reads as a glitch. On error the strip renders nothing rather than guessing which step failed; the row does not record it.
• retry.test.ts, named in the spec's testing plan, did not exist. Its cases now sit where each rule lives: the vault gate and the 400-mapping beside the route, the failed-state guard and the frozen-mounts reuse beside the service. The mounts case seeds a repo default that differs from the row's snapshot and asserts which one reached docker.createContainer, so it fails with the real regression — verified by pointing retryWorker at resolveSpawnMounts and watching it go red naming /mnt/newer-default.

**Stacked on #98** — targets `fix/mobile-sidebar-scroll`, not `main`. Merge #98 first. Second of the four PRs in the issue #96 spec (design D2); independent of the sidebar work. ## The gap `finishProvisioning` cleared `provision_phase` the instant `container.start()` returned. The entrypoint then spent another 30–90s on git identity, clone, `npm install -g @anthropic-ai/claude-code@latest` and finally `tmux new-session` — during which the card read **running · healthy** and every session action failed with no explanation on screen. ## What's in **A `booting` phase** that survives `finishProvisioning` and is cleared by `checkWorkerHealth` on the first successful `tmux has-session` probe — tmux answering *is* the definition of usable. While booting, the card shows the last `==> ` marker the entrypoint printed. `entrypoint.sh` already emits those, so **no worker-side change was needed**. **Pull progress.** `pullWorkerImage` already yielded per-layer `{id, current, total}`; `provisionWorker` discarded all of it except as a boolean "has anything happened yet" flag. `PullProgress` folds those into a percentage and a layer/byte line, throttled to one DB write per 750ms (docker emits per layer per chunk) with a forced final write so the bar cannot freeze mid-percentage. **Percent is `null` — an indeterminate bar — whenever any layer lacks a known total**, rather than a number that races upward and jumps back as sizes arrive. **Retry** on a failed card. It cannot restart anything: `provisionWorker` tears the container down on every failure path and `container_id` was never written. So `retryWorker` re-enters the pipeline from `queued` on the same row, keeping the frozen `mounts_json` snapshot it was created with (the operator retried *this* workspace, not a new one). It needs the master key to rebuild the sealed secret block, so it is an operator route and never something `convergeSessions` could do. The provider is resolved **before** the row is claimed, so a retry that fails early leaves the row in `error` and still retryable. ## Two traps worth calling out in review **`failStuckProvisioning` must SPARE `booting`.** Nothing in Nexus drives that phase — the container is started and its entrypoint runs to completion whether or not Nexus is up. Sweeping it would errored-out every workspace that happened to be booting across a Nexus restart, when the next health poll would have cleared it correctly. Verified live: on a harness restart the two `pulling` rows failed and the `booting` row was untouched. **Session controls are gated on the phase, not just on `running`.** Sessions, the shell agent and the view panels all need tmux; leaving them live during boot just moves the failure one click later. `isReady = isRunning && provision_phase !== 'booting'`. ## Verification Browser-driven against a seeded instance, one card per state: | state | rendered | |---|---| | `pulling`, all sizes known | determinate bar at 62%, `4 of 7 layers · 218.0 MB / 351.0 MB` | | `pulling`, a size unknown | indeterminate bar, no percentage | | `booting` | `Starting up…`, entrypoint marker, **no** "start agent", "Sessions appear once the workspace finishes starting." | | `error` | reason + **Retry** · Logs · Remove | | steady | unchanged | Also exercised the route directly: `POST /api/workers/w3/retry` on a `booting` row → `400 only a failed workspace can be retried`; a retry that failed at provider resolution left the row's original `image pull stalled…` message intact. `pnpm test` 109 files / **1028 tests** pass (+15; the 5 pre-existing `provisioning.test.ts` cases are preserved, 2 amended where the behaviour genuinely changed) · `pnpm run typecheck` 4859 files, 0 errors · `pnpm run lint` clean. Adding `'booting'` to `ProvisionPhase` made `provision-label.ts`'s exhaustive switch and two DTO fixtures fail to compile until updated — the guard working as intended. Migration `0017_provision_detail.ts` adds two nullable display-only columns. No backfill: a null pair is exactly the "nothing reported" state the card already renders. --- ### Added after the spec audit A five-agent pass over the shipped code against the spec found two D2 gaps: - **The five-segment step strip** (queued / pulling / creating+starting / booting / ready) is in every provisioning frame of the mock and was absent. `creating` and `starting` share a segment on purpose — both are sub-second docker calls, and a segment that lights for 40ms reads as a glitch. On `error` the strip renders nothing rather than guessing which step failed; the row does not record it. - **`retry.test.ts`**, named in the spec's testing plan, did not exist. Its cases now sit where each rule lives: the vault gate and the 400-mapping beside the route, the failed-state guard and the frozen-mounts reuse beside the service. The mounts case seeds a repo default that *differs* from the row's snapshot and asserts which one reached `docker.createContainer`, so it fails with the real regression — verified by pointing `retryWorker` at `resolveSpawnMounts` and watching it go red naming `/mnt/newer-default`.

lz added 1 commit 2026-09-02 16:29:52 +02:00

feat(workers): show real startup progress, and let a failed provision retry ... 78fc7bc8de

The card used to claim "running · healthy" the instant container.start()
returned, while the entrypoint spent another 30-90s cloning the repo and
running `npm install -g @anthropic-ai/claude-code@latest`. Every session action
failed in that window with no explanation on screen.

Adds a `booting` phase that survives finishProvisioning and is cleared by
checkWorkerHealth on the first successful `tmux has-session` probe -- tmux
answering IS the definition of usable. While booting, the card shows the last
`==> ` marker the entrypoint printed; entrypoint.sh already emits those, so no
worker-side change was needed.

Adds pull progress. pullWorkerImage already yielded per-layer
{id, current, total} and provisionWorker discarded all of it except as a
boolean "has anything happened yet" flag. PullProgress folds those into a
percentage and a layer/byte line, throttled to one DB write per 750ms (docker
emits per layer per chunk) with a forced final write so the bar cannot freeze
mid-percentage. Percent is null -- an indeterminate bar -- whenever any layer
lacks a known total, rather than showing a number that races upward and jumps
back as sizes arrive.

Adds Retry on a failed card. It cannot restart anything: provisionWorker tears
the container down on every failure path and container_id was never written, so
retryWorker re-enters the pipeline from `queued` on the same row, keeping the
frozen mounts_json snapshot it was created with. It needs the master key to
rebuild the sealed secret block, so it is an operator route and never something
convergeSessions could do. The provider is resolved BEFORE the row is claimed,
so a retry that fails early leaves the row in `error` and retryable.

Two traps handled:

- failStuckProvisioning must SPARE `booting`. Nothing in Nexus drives that
  phase -- the container is started and its entrypoint runs to completion
  whether or not Nexus is up -- so sweeping it would errored-out every
  workspace booting across a Nexus restart. Verified live: on a harness restart
  the two `pulling` rows failed and the `booting` row was untouched.
- Session controls are gated on the phase, not just on `running`. Sessions, the
  shell agent and the view panels all need tmux, so leaving them live during
  boot just moves the failure one click later.

Verified in a browser against a seeded instance: determinate bar at 62% with
"4 of 7 layers · 218.0 MB / 351.0 MB", indeterminate bar when a layer size is
unknown, the booting card showing its entrypoint marker with no "start agent"
button, POST /retry rejecting a non-errored row with 400, and a failed retry
leaving the row's original error message intact.

lz added 1 commit 2026-09-02 17:17:55 +02:00

fix(workers): correct pull layer counting and stale phase detail ... 37e9968951

Both found by running the real worker image through a real dind daemon, not by
the suite.

1. The tag was counted as a layer. Docker's stream-level "Pulling from <repo>"
   event carries an `id` -- the TAG, not a layer -- so observe() registered it
   alongside the real layers. Measured against docker 29.6.2: a 5-layer image
   reported 6. Stream-level statuses are now filtered out.

2. Completion could not be inferred from bytes. A cached layer reports
   "Already exists" and transfers nothing, so it never reaches a total and was
   counted as incomplete forever. Layer completion now comes from the terminal
   statuses ("Pull complete" / "Already exists"), which is what docker actually
   tells us. A fully cached pull previously read "0 of 6 layers · 0.0 MB"; it
   now reads "5 of 5 layers", and the byte clause is dropped entirely when
   nothing was transferred rather than shouting "0.0 MB".

3. Phase detail outlived its phase. advanceProvisionPhase left provision_detail
   and provision_percent in place, so the card showed "Creating container… 100%"
   with the pull's byte count under it. Observed live at the pulling→creating
   transition. Advancing now clears both; the new phase writes its own.

Live trace after the fixes, against x.lck.sh/lz/agent-nexus-worker:latest
(318 MB, 20 layers) pulled from an empty local store:

  queued
  pulling   pct=null  1 of 20 layers · 5.3 MB
  pulling   pct=38    15 of 20 layers · 122.5 MB / 318.1 MB
  pulling   pct=100   19 of 20 layers · 318.1 MB / 318.1 MB
  creating  pct=null  (no detail)
  booting   Cloning https://x.lck.sh/lz/agent-nexus.git
  booting   Registering forgejo-mcp
  booting   Updating @anthropic-ai/claude-code to latest
  null      (ready -- healthy, tmux up, repo cloned on main)

Percent is null only for the first ~10s, while docker is still announcing layer
sizes; once every total is known it rises monotonically to 100. The earlier
worry that the determinate bar would rarely appear did not hold up.

lz added 1 commit 2026-09-02 17:23:15 +02:00

refactor(workers): tail the container log once per health poll ... 67135b61e2

A `booting` worker tailed its container log TWICE in the same poll: once to
read the entrypoint's `==> ` marker, and again to fill the report's
recentLogs. tailLogs resolves the container itself (findContainerByWorkerId
lists containers), so that was two extra docker round-trips every 10s for every
second of the boot window.

One tail now feeds both, and the `booting` test is hoisted into a local instead
of being re-derived at three call sites.

Also merges a duplicated `.bar i` CSS rule.

lz added 1 commit 2026-09-02 17:42:58 +02:00

refactor(workers): apply /simplify review — plus two real gaps it found ... cb0263ccef

Four review passes over the diff (reuse / simplification / efficiency /
altitude). Two of their findings were behaviour bugs, not tidiness:

1. `startWorker` never set `booting`. Restarting re-runs the whole entrypoint
   (claude-code update, tmux start), so Stop -> Start went straight back to a
   green card whose every session action failed -- the exact defect this branch
   exists to remove, surviving on the path nobody tested. One field in an UPDATE
   that already existed. Verified live: Start now shows "Starting up..." with a
   live marker until tmux answers.

2. `booting` was the only phase with no failure exit. Every other in-flight
   phase is bounded (`pulling` by PROVISION_PULL_STALL_MS, the rest by
   failStuckProvisioning), but excluding `booting` from that sweep removed its
   only bound without adding another -- so an entrypoint stalled on, say, its
   wait-for-secrets loop would show a cheerful "Starting up..." forever. Bounded
   by BOOT_GRACE_MS against `last_seen_at`, which was already stamped. Past it
   the phase is DROPPED rather than errored: the container really is running, so
   plain `unhealthy` is the honest label.

3. `WorkspaceRail` painted a booting workspace green and opened a terminal
   against a container with no tmux. `isRunning`/`isReady` are now exported once
   from the store and used by both the rail and the card, instead of a fifth
   inline derivation.

Quality fixes:

- pull-progress.test.ts was a BINARY file. A test for docker's 8-byte frame
  header embedded literal NUL bytes, so git classified the whole file as binary
  and `git diff`/`git blame` were useless on it. The defence was dead anyway:
  the only caller passes tailLogs output, and tailLogs already runs demuxLogs.
  lastBootMarker now anchors with startsWith -- which also stops a "==> "
  appearing mid-line in unrelated output from counting as a stage.
- The throttle moved out of PullProgress into the loop that owns the clock,
  deleting a `force` parameter that had no caller and whose doc comment
  described a mechanism that was never wired.
- snapshot() folded four passes over the layer Map into one, and two returns
  repeating the same template into one.
- The indeterminate bar animated `margin-left`, forcing layout+paint on the main
  thread every frame for the whole boot window, on a page also running xterm
  terminals. Now `transform: translateX`, which is composited.
- `provision_detail: null, provision_percent: null` was written five times; now
  one CLEAR_PROGRESS spread.
- The booting transition is extracted as `settleBootingPhase`, so the decision
  is testable against the in-memory DB rather than only reachable inside a
  dockerode-bound function.
- recentLogs is no longer shipped while booting -- nothing reads it, and this
  branch turned a rare fault path into the routine path for every spawn.

Deliberately NOT taken: structuring provision_detail into columns
(provision_error is already free text in the same table), a server-side `ready`
DTO field (a third representation of what tmuxRunning already carries), and a
shared formatBytes (no drop-in exists; the nearest is private to a component and
uses a different base).

lz referenced this pull request 2026-09-02 18:47:48 +02:00

feat(ui): declutter the sidebar onto a shared target menu #100

lz added this to the Sidebar declutter, startup visibility & the target menu (#96) milestone 2026-09-02 23:50:22 +02:00

lz added 1 commit 2026-09-03 00:01:13 +02:00

feat(workers): the step strip D2.5 specifies, and the retry tests ... 1772a55ca2

Two gaps an audit of the shipped code against the spec turned up.

The five-segment step strip (queued / pulling / creating+starting / booting /
ready) is in every provisioning frame of the mock and was absent from the card.
`creating` and `starting` share a segment deliberately: both are sub-second
docker calls, and a segment that lights for 40ms reads as a glitch. `Ready` is
never the active step — a workspace reaches it by `provision_phase` going null,
at which point the strip is gone — and a test pins that rather than trusting the
switch to stay honest.

On `error` the strip renders nothing. Pointing at a failed step would claim to
know which one failed, and the row does not record it; the card already shows
the reason plus Retry / Logs / Remove.

`retry.test.ts` is named in the spec's testing plan and did not exist. Its three
cases now sit at the layer that actually owns each rule: the vault gate and the
error mapping are the route's, so they live beside it; the failed-state guard
and the frozen-mounts reuse are the service's.

The mounts case is the one worth having. It seeds a repo default that DIFFERS
from the row's snapshot and asserts which one reached `docker.createContainer`,
so it fails with the actual regression rather than on a mock call count —
verified by pointing retryWorker at `resolveSpawnMounts` and watching it go red
naming `/mnt/newer-default`.

lz referenced this pull request from a commit 2026-09-04 00:07:02 +02:00

chore(db): land 0017 ahead of the issue-96 release candidate

lz referenced this pull request 2026-09-04 00:07:32 +02:00

chore(db): land migration 0017 ahead of the #96 release candidate #104

lz referenced this pull request from a commit 2026-09-04 00:20:35 +02:00

Merge PR #104: land migration 0017 ahead of the #96 release candidate

lz changed target branch from fix/mobile-sidebar-scroll to main 2026-09-04 17:24:10 +02:00

lz merged commit 3fe3f54718 into main 2026-09-04 17:24:15 +02:00

lz referenced this pull request from a commit 2026-09-04 17:24:28 +02:00

Merge PR #99: show real startup progress

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

wontfix

This won't be fixed

No labels

bug

duplicate

enhancement

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Sidebar declutter, startup visibility & the target menu (#96)

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lz/agent-nexus!99

No description provided.