Composable worker images: one base + per-feature tarballs, composed locally by Nexus #183
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
lz/agent-nexus#183
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Stop shipping a curated matrix of pre-built worker images. Ship one
baseimage plus one additive tarball per feature, let CI build those in parallel, and have Nexus compose the image it needs locally frombase+ the selected feature tarballs. Surface the selection as feature checkboxes in the spawn UI instead of an image dropdown, and ship the compose tooling in-repo.This supersedes the "Why baking (not runtime-install or mounts)" section of
docs/superpowers/specs/2026-07-10-dind-worker-image-design.md— not because that analysis was wrong, but because it evaluated three options and there is a fourth.What the 2026-07-10 doc got right, and must stay
Correct, and it constrains this design. No composition strategy can add privilege or mounts to a running container, so the runtime axis stays in
capabilities.tsexactly as it is. The 2^N problem is only ever about baked filesystem contents. Everything below is about contents only.What the doc did not consider is that dev-container Features are not runtime-installed either — they are composed into a build, lazily, per requested combination. Its objection ("runtime-install = 2–20 min cold starts") applies per container; composing a cached image applies per combination, once.
Why now — measured
From CI run 949 (
imagesjob, all five targets):dind-playwrightisFROM playwrightisFROM base, so that chain is strictly serial. Measured parallelism across the whole bake is only 2.39x (3024s of work in 1266s wall) — the build is critical-path-bound, not CPU-bound, so throwing cores at it changes nothing.dindand 468.1s indind-playwright— because Docker stages cannot have two parents.[playwright] exporting to image344.3s,[webapp] exporting to image227.2s, cache export 211.7s).The shape
CI builds three things, in parallel
playwrightanddindare each builtFROM base, concurrently.dind-playwrightstops existing as a build target, which removes both the duplicated dind install and the serialization:Playwright is already ~97% download-and-extract
Measured inside
agent-nexus-worker:latest-dind-playwright0.1.19:/ms-playwrightPLAYWRIGHT_BROWSERS_PATH, i.e. already relocatable by design/usr/local/lib/node_modules/@playwright/cli/opt/nexus/plugins/playwrightinstall-deps)dind becomes download-and-extract too
Docker publishes static binaries (
docker-<ver>.tgz:dockerd,containerd,runc,ctr,docker-proxy,docker-init), and buildx/compose are single-file CLI plugins. That replaces the apt repo + GPG key dance inworker/addons/dind/install.shentirely. Onlyiptablesstays a real package — and it belongs inbaseanyway, along with theupdate-alternativeslegacy switchdockerdneeds.Nexus composes locally — and needs no new privilege
base layers + playwright.tar + dind.tar→ assemble a manifest with merged config →docker load.The socket proxy already allows
IMAGES: 1+POST: 1, which coversPOST /images/load. It does not enableBUILD. So this design needs no change to the proxy allowlist, whereas a Dockerfile-based build-on-demand would have requiredPOST /build. That is a significant part of why this shape is preferable.Composed images are tagged locally and cached, so a given combination is assembled once.
The apt problem, and why it disappears
Naively, two feature layers each built
FROM baseboth carry a modified/var/lib/dpkg/status:/var/lib/dpkg/info/<pkg>.listis one file per package so it merges fine — only the single-file state collides. Consequence:apt install Xmay reinstall a library Chromium depends on, anddpkg -llies to an agent debugging.The fix is to make it structurally impossible: all apt lives in
base; features are pure file drops.baseapt-installs the Chromium system libs andiptables, with correct dpkg state.playwright.tar=/ms-playwright+ npm global + skill dir +cli.config.json. No apt.dind.tar= static binaries +/etc/nexus/entrypoint.d/50-dockerd.sh. No apt.A bonus falls out: with apt gone, neither feature deletes anything (
addons/dind/install.sh'srm -rf /var/lib/apt/lists/*goes away with it). Both features become purely additive tarballs, so there are no.wh.whiteouts to handle and a plain uncompressed tar is a valid OCI layer. CI does not even have to produce them as Docker layer diffs.If a future feature genuinely must use apt, the escape hatch is to ship its
statusstanzas as a side file (/opt/nexus/features/<name>/dpkg-status.delta) and append them from anentrypoint.d/drop-in —statusis an RFC822-style record file, so appending is safe.Feature selection in the UI
Today the operator picks an image (
worker-images.tsoffers a curated tag list) and the image's labels imply its capabilities. After this, the operator picks features.playwright,dind, …), sourced from a feature registry rather than a tag list.workersgains afeatures_jsoncolumn recording what was selected — the same "freeze what was applied" treatmentmounts_jsonalready gets, since composition is create-time.worker-images.ts's offered-tag list and theplaywright-version-parity.test.tsassertions about it are replaced by the feature registry.Scripts shipped in-repo
worker/features/<name>/build.sh— CI-side: produce<name>.tarplus a smallfeature.jsondeclaring the env vars, OCI labels andentrypoint.d/files the feature contributes. Replacesworker/addons/<name>/install.shas the unit of authorship, keeping the self-contained-feature-dir model the 2026-07-10 doc introduced.nexus/src/lib/server/workers/compose.ts— assemble anddocker loada composed image; idempotent, content-addressed tag.Open question that needs a decision first
Who is trusted to assert
org.nexus.dind?Env vars and labels live in image config, not in a tarball, so Nexus merges them at compose time. Today the label is evidence baked by CI, and
capabilities.tsdescribes the label registry as "the security boundary for IMAGE-DERIVED HostConfig deltas". After this change Nexus writes the label it then reads.That is arguably fine — Nexus already decides what to spawn, and the operator's feature selection is the same consent the image selection was. But it is a real change to the trust story, the comment in
capabilities.tswould need rewording, and it should be decided deliberately rather than absorbed as a side effect of a build optimisation.Explicitly out of scope
Runtime
HostConfigcomposition. Privilege, mounts,ShmSizeandInitremain create-time and stay incapabilities.ts. This issue changes only what is baked into the filesystem.Related
Context came out of a CI investigation (PR #182, closed unmerged). Two findings from it are relevant here:
imagesjob spends most of its time on the serial chain above, not on parallelisable work.UQE_package_blob_sha256, and Forgejo nil-pointer-panics inservices/packages/container/blob.go:98, surfacing to BuildKit aserror writing layer blob: unknown. Fewer and smaller published artifacts — which this design gives — make that race materially less likely, though it should be reported upstream separately (Forgejo 16.0.5).🤖 Generated with Claude Code