Composable worker images: one base + per-feature tarballs, composed locally by Nexus #183

Open
opened 2026-09-19 20:03:06 +02:00 by lz · 0 comments
Owner

Summary

Stop shipping a curated matrix of pre-built worker images. Ship one base image plus one additive tarball per feature, let CI build those in parallel, and have Nexus compose the image it needs locally from base + the selected feature tarballs. Surface the selection as feature checkboxes in the spawn UI instead of an image dropdown, and ship the compose tooling in-repo.

This supersedes the "Why baking (not runtime-install or mounts)" section of docs/superpowers/specs/2026-07-10-dind-worker-image-design.md — not because that analysis was wrong, but because it evaluated three options and there is a fourth.

What the 2026-07-10 doc got right, and must stay

--privileged, mounts, ShmSize, Init are all create-time HostConfig and immutable for the container's life.

Correct, and it constrains this design. No composition strategy can add privilege or mounts to a running container, so the runtime axis stays in capabilities.ts exactly as it is. The 2^N problem is only ever about baked filesystem contents. Everything below is about contents only.

What the doc did not consider is that dev-container Features are not runtime-installed either — they are composed into a build, lazily, per requested combination. Its objection ("runtime-install = 2–20 min cold starts") applies per container; composing a cached image applies per combination, once.

Why now — measured

From CI run 949 (images job, all five targets):

critical path   base  npm install -g      189.5s
                  └→  playwright addon    395.4s
                       └→ dind-playwright addon  468.1s
                                          ───────
                                          1053s  ≈ 83% of the 1266s bake
  • dind-playwright is FROM playwright is FROM base, so that chain is strictly serial. Measured parallelism across the whole bake is only 2.39x (3024s of work in 1266s wall) — the build is critical-path-bound, not CPU-bound, so throwing cores at it changes nothing.
  • The dind addon installs twice — 178s in dind and 468.1s in dind-playwright — because Docker stages cannot have two parents.
  • The combo image is 2.09 GB, and roughly 780s of that run was image/cache transfer ([playwright] exporting to image 344.3s, [webapp] exporting to image 227.2s, cache export 211.7s).
  • Adding a third addon takes the curated set from 4 images to 8.

The shape

CI builds three things, in parallel

base                     (the only place apt runs)
playwright.tar           (additive, ~400 MB)
dind.tar                 (additive)

playwright and dind are each built FROM base, concurrently. dind-playwright stops existing as a build target, which removes both the duplicated dind install and the serialization:

before   base 189s → playwright 395s → dind-playwright 468s   = 1053s
after    base 189s → { playwright 395s ‖ dind 178s }          =  584s

Playwright is already ~97% download-and-extract

Measured inside agent-nexus-worker:latest-dind-playwright0.1.19:

path size nature
/ms-playwright 392 MB a zip from Playwright's CDN; location is set by PLAYWRIGHT_BROWSERS_PATH, i.e. already relocatable by design
/usr/local/lib/node_modules/@playwright/cli 7.7 MB plain node_modules tree
/opt/nexus/plugins/playwright 72 KB copied files
Chromium system libs (install-deps) ~11 MB (core set) the only apt part

dind becomes download-and-extract too

Docker publishes static binaries (docker-<ver>.tgz: dockerd, containerd, runc, ctr, docker-proxy, docker-init), and buildx/compose are single-file CLI plugins. That replaces the apt repo + GPG key dance in worker/addons/dind/install.sh entirely. Only iptables stays a real package — and it belongs in base anyway, along with the update-alternatives legacy switch dockerd needs.

Nexus composes locally — and needs no new privilege

base layers + playwright.tar + dind.tar → assemble a manifest with merged config → docker load.

The socket proxy already allows IMAGES: 1 + POST: 1, which covers POST /images/load. It does not enable BUILD. So this design needs no change to the proxy allowlist, whereas a Dockerfile-based build-on-demand would have required POST /build. That is a significant part of why this shape is preferable.

Composed images are tagged locally and cached, so a given combination is assembled once.

The apt problem, and why it disappears

Naively, two feature layers each built FROM base both carry a modified /var/lib/dpkg/status:

base   : status = S₀
pw     : status = S₀ + P
dind   : status = S₀ + D
composed (dind last) = S₀ + D     ← P's packages on disk, unknown to dpkg

/var/lib/dpkg/info/<pkg>.list is one file per package so it merges fine — only the single-file state collides. Consequence: apt install X may reinstall a library Chromium depends on, and dpkg -l lies to an agent debugging.

The fix is to make it structurally impossible: all apt lives in base; features are pure file drops.

  • base apt-installs the Chromium system libs and iptables, with correct dpkg state.
  • playwright.tar = /ms-playwright + npm global + skill dir + cli.config.json. No apt.
  • dind.tar = static binaries + /etc/nexus/entrypoint.d/50-dockerd.sh. No apt.

A bonus falls out: with apt gone, neither feature deletes anything (addons/dind/install.sh's rm -rf /var/lib/apt/lists/* goes away with it). Both features become purely additive tarballs, so there are no .wh. whiteouts to handle and a plain uncompressed tar is a valid OCI layer. CI does not even have to produce them as Docker layer diffs.

If a future feature genuinely must use apt, the escape hatch is to ship its status stanzas as a side file (/opt/nexus/features/<name>/dpkg-status.delta) and append them from an entrypoint.d/ drop-in — status is an RFC822-style record file, so appending is safe.

Feature selection in the UI

Today the operator picks an image (worker-images.ts offers a curated tag list) and the image's labels imply its capabilities. After this, the operator picks features.

  • Spawn dialog: the image dropdown becomes a checkbox group (playwright, dind, …), sourced from a feature registry rather than a tag list.
  • workers gains a features_json column recording what was selected — the same "freeze what was applied" treatment mounts_json already gets, since composition is create-time.
  • Image resolution becomes: resolve the requested feature set → reuse the cached composed image, or compose it.
  • The dind checkbox keeps its existing privilege warning; selecting it remains the consent, exactly as the 2026-07-10 doc specifies.
  • worker-images.ts's offered-tag list and the playwright-version-parity.test.ts assertions about it are replaced by the feature registry.

Scripts shipped in-repo

  • worker/features/<name>/build.sh — CI-side: produce <name>.tar plus a small feature.json declaring the env vars, OCI labels and entrypoint.d/ files the feature contributes. Replaces worker/addons/<name>/install.sh as the unit of authorship, keeping the self-contained-feature-dir model the 2026-07-10 doc introduced.
  • nexus/src/lib/server/workers/compose.ts — assemble and docker load a composed image; idempotent, content-addressed tag.
  • An operator-facing script to pre-compose a set offline, so a first spawn never pays the assembly cost.

Open question that needs a decision first

Who is trusted to assert org.nexus.dind?

Env vars and labels live in image config, not in a tarball, so Nexus merges them at compose time. Today the label is evidence baked by CI, and capabilities.ts describes the label registry as "the security boundary for IMAGE-DERIVED HostConfig deltas". After this change Nexus writes the label it then reads.

That is arguably fine — Nexus already decides what to spawn, and the operator's feature selection is the same consent the image selection was. But it is a real change to the trust story, the comment in capabilities.ts would need rewording, and it should be decided deliberately rather than absorbed as a side effect of a build optimisation.

Explicitly out of scope

Runtime HostConfig composition. Privilege, mounts, ShmSize and Init remain create-time and stay in capabilities.ts. This issue changes only what is baked into the filesystem.

Context came out of a CI investigation (PR #182, closed unmerged). Two findings from it are relevant here:

  • The images job spends most of its time on the serial chain above, not on parallelisable work.
  • Pushing five overlapping images concurrently reliably triggers a Forgejo bug: two concurrent uploads of the same blob digest race, one violates UQE_package_blob_sha256, and Forgejo nil-pointer-panics in services/packages/container/blob.go:98, surfacing to BuildKit as error writing layer blob: unknown. Fewer and smaller published artifacts — which this design gives — make that race materially less likely, though it should be reported upstream separately (Forgejo 16.0.5).

🤖 Generated with Claude Code

## Summary Stop shipping a curated matrix of pre-built worker images. Ship **one `base` image plus one additive tarball per feature**, let CI build those in parallel, and have **Nexus compose the image it needs locally** from `base` + the selected feature tarballs. Surface the selection as **feature checkboxes in the spawn UI** instead of an image dropdown, and ship the compose tooling in-repo. This supersedes the "Why baking (not runtime-install or mounts)" section of [`docs/superpowers/specs/2026-07-10-dind-worker-image-design.md`](docs/superpowers/specs/2026-07-10-dind-worker-image-design.md) — not because that analysis was wrong, but because it evaluated three options and there is a fourth. ## What the 2026-07-10 doc got right, and must stay > `--privileged`, mounts, `ShmSize`, `Init` are all create-time `HostConfig` and **immutable for the container's life**. Correct, and it constrains this design. No composition strategy can add privilege or mounts to a *running* container, so the runtime axis stays in `capabilities.ts` exactly as it is. **The 2^N problem is only ever about baked filesystem contents.** Everything below is about contents only. What the doc did not consider is that dev-container Features are **not** runtime-installed either — they are composed into a build, lazily, per requested combination. Its objection ("runtime-install = 2–20 min cold starts") applies per *container*; composing a cached image applies per *combination*, once. ## Why now — measured From CI run 949 (`images` job, all five targets): ``` critical path base npm install -g 189.5s └→ playwright addon 395.4s └→ dind-playwright addon 468.1s ─────── 1053s ≈ 83% of the 1266s bake ``` - `dind-playwright` is `FROM playwright` is `FROM base`, so that chain is **strictly serial**. Measured parallelism across the whole bake is only **2.39x** (3024s of work in 1266s wall) — the build is critical-path-bound, not CPU-bound, so throwing cores at it changes nothing. - The **dind addon installs twice** — 178s in `dind` and 468.1s in `dind-playwright` — because Docker stages cannot have two parents. - The combo image is **2.09 GB**, and roughly **780s** of that run was image/cache transfer (`[playwright] exporting to image` 344.3s, `[webapp] exporting to image` 227.2s, cache export 211.7s). - Adding a third addon takes the curated set from 4 images to 8. ## The shape ### CI builds three things, in parallel ``` base (the only place apt runs) playwright.tar (additive, ~400 MB) dind.tar (additive) ``` `playwright` and `dind` are each built `FROM base`, concurrently. **`dind-playwright` stops existing as a build target**, which removes both the duplicated dind install and the serialization: ``` before base 189s → playwright 395s → dind-playwright 468s = 1053s after base 189s → { playwright 395s ‖ dind 178s } = 584s ``` ### Playwright is already ~97% download-and-extract Measured inside `agent-nexus-worker:latest-dind-playwright0.1.19`: | path | size | nature | |---|---|---| | `/ms-playwright` | **392 MB** | a zip from Playwright's CDN; location is set by `PLAYWRIGHT_BROWSERS_PATH`, i.e. already relocatable by design | | `/usr/local/lib/node_modules/@playwright/cli` | 7.7 MB | plain node_modules tree | | `/opt/nexus/plugins/playwright` | 72 KB | copied files | | Chromium system libs (`install-deps`) | ~11 MB (core set) | **the only apt part** | ### dind becomes download-and-extract too Docker publishes static binaries (`docker-<ver>.tgz`: `dockerd`, `containerd`, `runc`, `ctr`, `docker-proxy`, `docker-init`), and buildx/compose are single-file CLI plugins. That replaces the apt repo + GPG key dance in `worker/addons/dind/install.sh` entirely. Only `iptables` stays a real package — and it belongs in `base` anyway, along with the `update-alternatives` legacy switch `dockerd` needs. ### Nexus composes locally — and needs no new privilege `base layers + playwright.tar + dind.tar` → assemble a manifest with merged config → **`docker load`**. The socket proxy already allows `IMAGES: 1` + `POST: 1`, which covers `POST /images/load`. It does **not** enable `BUILD`. So this design needs **no change to the proxy allowlist**, whereas a Dockerfile-based build-on-demand would have required `POST /build`. That is a significant part of why this shape is preferable. Composed images are tagged locally and cached, so a given combination is assembled once. ## The apt problem, and why it disappears Naively, two feature layers each built `FROM base` both carry a modified `/var/lib/dpkg/status`: ``` base : status = S₀ pw : status = S₀ + P dind : status = S₀ + D composed (dind last) = S₀ + D ← P's packages on disk, unknown to dpkg ``` `/var/lib/dpkg/info/<pkg>.list` is one file per package so it merges fine — only the single-file state collides. Consequence: `apt install X` may reinstall a library Chromium depends on, and `dpkg -l` lies to an agent debugging. **The fix is to make it structurally impossible: all apt lives in `base`; features are pure file drops.** - `base` apt-installs the Chromium system libs and `iptables`, with correct dpkg state. - `playwright.tar` = `/ms-playwright` + npm global + skill dir + `cli.config.json`. No apt. - `dind.tar` = static binaries + `/etc/nexus/entrypoint.d/50-dockerd.sh`. No apt. A bonus falls out: with apt gone, **neither feature deletes anything** (`addons/dind/install.sh`'s `rm -rf /var/lib/apt/lists/*` goes away with it). Both features become purely *additive* tarballs, so there are no `.wh.` whiteouts to handle and a plain uncompressed tar is a valid OCI layer. CI does not even have to produce them as Docker layer diffs. If a future feature genuinely must use apt, the escape hatch is to ship its `status` stanzas as a side file (`/opt/nexus/features/<name>/dpkg-status.delta`) and append them from an `entrypoint.d/` drop-in — `status` is an RFC822-style record file, so appending is safe. ## Feature selection in the UI Today the operator picks an **image** (`worker-images.ts` offers a curated tag list) and the image's labels imply its capabilities. After this, the operator picks **features**. - Spawn dialog: the image dropdown becomes a checkbox group (`playwright`, `dind`, …), sourced from a feature registry rather than a tag list. - `workers` gains a `features_json` column recording what was selected — the same "freeze what was applied" treatment `mounts_json` already gets, since composition is create-time. - Image resolution becomes: resolve the requested feature set → reuse the cached composed image, or compose it. - The dind checkbox keeps its existing privilege warning; selecting it remains the consent, exactly as the 2026-07-10 doc specifies. - `worker-images.ts`'s offered-tag list and the `playwright-version-parity.test.ts` assertions about it are replaced by the feature registry. ## Scripts shipped in-repo - `worker/features/<name>/build.sh` — CI-side: produce `<name>.tar` plus a small `feature.json` declaring the env vars, OCI labels and `entrypoint.d/` files the feature contributes. Replaces `worker/addons/<name>/install.sh` as the unit of authorship, keeping the self-contained-feature-dir model the 2026-07-10 doc introduced. - `nexus/src/lib/server/workers/compose.ts` — assemble and `docker load` a composed image; idempotent, content-addressed tag. - An operator-facing script to pre-compose a set offline, so a first spawn never pays the assembly cost. ## Open question that needs a decision first **Who is trusted to assert `org.nexus.dind`?** Env vars and labels live in image config, not in a tarball, so Nexus merges them at compose time. Today the label is *evidence baked by CI*, and `capabilities.ts` describes the label registry as "the security boundary for IMAGE-DERIVED HostConfig deltas". After this change **Nexus writes the label it then reads**. That is arguably fine — Nexus already decides what to spawn, and the operator's feature selection is the same consent the image selection was. But it is a real change to the trust story, the comment in `capabilities.ts` would need rewording, and it should be decided deliberately rather than absorbed as a side effect of a build optimisation. ## Explicitly out of scope Runtime `HostConfig` composition. Privilege, mounts, `ShmSize` and `Init` remain create-time and stay in `capabilities.ts`. This issue changes only what is baked into the filesystem. ## Related Context came out of a CI investigation (PR #182, closed unmerged). Two findings from it are relevant here: - The `images` job spends most of its time on the serial chain above, not on parallelisable work. - Pushing five overlapping images concurrently reliably triggers a **Forgejo bug**: two concurrent uploads of the same blob digest race, one violates `UQE_package_blob_sha256`, and Forgejo nil-pointer-panics in `services/packages/container/blob.go:98`, surfacing to BuildKit as `error writing layer blob: unknown`. Fewer and smaller published artifacts — which this design gives — make that race materially less likely, though it should be reported upstream separately (Forgejo 16.0.5). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
lz/agent-nexus#183
No description provided.