Add opt-in Docker-in-Docker worker image so agents can spin up disposable stacks #32

New issue

Open

opened 2026-05-28 08:02:19 +02:00 by lz · 0 comments

lz commented

2026-05-28 08:02:19 +02:00

Owner

Motivation

Agents working on infrastructure or full-stack features routinely need to bring up disposable Docker stacks — to smoke-test a docker compose change, to validate that a new image builds and boots, or (the immediate driver) to end-to-end-smoke a UI feature of Nexus itself against a fresh Nexus instance without touching the operator's live one.

The current worker image has no Docker access. Mounting the host /var/run/docker.sock is off the table — it makes worker→host container escape trivial, and the existing tecnativa/docker-socket-proxy allowlist (Nexus's own gateway to Docker) is intentionally narrow and host-Nexus-only.

This issue proposes an opt-in Docker-in-Docker (dind) worker image that runs its own dockerd inside a sidecar pattern, isolated from the host's Docker. The agent gets a fully-functional docker CLI and docker compose, but the resulting containers live in the worker's own daemon — no host escape.

Companion to #2 (Playwright MCP). The two together close the loop on agent-driven end-to-end testing: spin up a stack with dind, drive its UI with Playwright. They don't depend on each other.

Scope

Opt-in per workspace at creation time, same shape as the proposed Playwright image variant in #2.
New worker image tag nexus-worker-dind:latest, derived from nexus-worker:latest.
Dind daemon runs rootful inside the worker (the worker container itself is already root and isolated; rootless dind is a complexity tax for no security gain in our model).
Agent gets docker CLI on $PATH and docker compose (v2 plugin).
Workspace volume layout: a fresh /var/lib/docker tmpfs or volume per worker, ephemeral by default. Persistence is out of scope.
Storage driver vfs for simplicity (no overlay-on-overlay woes); upgrade to fuse-overlayfs if vfs perf becomes a blocker.

Threat model

Single-operator design carries over. Privileged container is a non-starter only if we cared about worker→host escape, which we already don't gate against in practice (the worker is root and has bind mounts to the worktree).
Network isolation: dind's daemon manages its own bridge (default docker0 inside the worker). Containers it spawns are reachable from the worker but not from the host's nexus-workers bridge — they live behind dind's NAT inside the worker's network namespace. This is the intended isolation.
Disk usage: dind images and layers accumulate fast. Default to ephemeral storage (tmpfs or anonymous volume); document the docker system prune habit. A hard quota is out of scope for v1.
Operator surfacing: dind-spawned services that the agent wants the operator to see (e.g., a Nexus instance the agent stood up) already work via notify-preview — that CLI runs in the worker namespace, so when the agent maps dind:3001 → worker:3001 via docker run -p, notify-preview 3001 surfaces it to the operator unchanged.

Verified facts (worth knowing before implementing)

--privileged is required. Dind without privileged fails at iptables setup (no CAP_NET_ADMIN). The dockerless sysbox alternative is a custom runtime — out of scope for a stock Docker host.
/var/lib/docker MUST NOT be on overlayfs. Overlay-on-overlay is broken on most kernels. Either use a named volume (ext4-backed), a tmpfs (smaller stacks fit fine), or set --storage-driver vfs explicitly.
Dind needs its own cgroupfs mount. Recent Docker handles this automatically via --cgroupns=host or the official docker:dind image's entrypoint; if we run dockerd ourselves we need to set --exec-opt native.cgroupdriver=cgroupfs and ensure systemd isn't fighting it.
Boot ordering matters. Worker boot must start dockerd BEFORE tmux idles, because the agent will expect docker to work the moment it opens a session. The existing worker/entrypoint.sh already has a pre-tmux setup block (where forgejo-mcp registration lives) — dockerd boot fits there.
docker compose is a plugin (not a separate binary) on Debian-derived images. Install via docker-ce-cli + docker-compose-plugin apt packages, NOT the standalone docker-compose Python script.
pnpm exec playwright install-style binary downloads can blow tmpfs /var/lib/docker. Plan storage size for the realistic agent workload (image builds with multi-hundred-MB layers).

Suggested implementation

1. New worker image variant

worker/Dockerfile.dind, extending the base image:

FROM nexus-worker:latest

# Install Docker CE CLI + dockerd + compose plugin.
RUN apt-get update && apt-get install -y --no-install-recommends \
      ca-certificates curl gnupg \
    && install -m 0755 -d /etc/apt/keyrings \
    && curl -fsSL https://download.docker.com/linux/debian/gpg \
       | gpg --dearmor -o /etc/apt/keyrings/docker.gpg \
    && chmod a+r /etc/apt/keyrings/docker.gpg \
    && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
       https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
       > /etc/apt/sources.list.d/docker.list \
    && apt-get update && apt-get install -y --no-install-recommends \
       docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin \
       iptables \
    && rm -rf /var/lib/apt/lists/*

# vfs avoids overlay-on-overlay pain. Override via --storage-driver if needed.
ENV DOCKERD_FLAGS="--storage-driver=vfs --host=unix:///var/run/docker.sock"

COPY entrypoint-dind.sh /entrypoint-dind.sh
RUN chmod +x /entrypoint-dind.sh
ENTRYPOINT ["/entrypoint-dind.sh"]

CI: add to .forgejo/workflows/ci.yml build matrix alongside the proposed Playwright variant from #2.

2. Entrypoint augmentation

worker/entrypoint-dind.sh runs the existing boot sequence, then before tmux setup:

echo "==> Starting dockerd"
mkdir -p /var/log
nohup dockerd $DOCKERD_FLAGS > /var/log/dockerd.log 2>&1 &
DOCKERD_PID=$!

# Wait for socket up to ~10s; continue regardless (agent will see clear errors if it fails)
for _ in $(seq 1 20); do
  if docker version >/dev/null 2>&1; then
    echo "==> dockerd ready (pid $DOCKERD_PID)"
    break
  fi
  sleep 0.5
done

Then delegate to the existing entrypoint's tmux setup. The dockerd process becomes a child of pid 1; the existing entrypoint already exits after tmux idles, so dockerd inherits the standard process tree.

3. Spawn-time HostConfig

Nexus container spawn at nexus/src/lib/server/workers/service.ts:

HostConfig.Privileged: true for nexus-worker-dind:latest (and ONLY this image — never for default workers).
HostConfig.SecurityOpt: ['seccomp=unconfined'] (some kernels need this for the inner dockerd to set up netns).
Volume: anonymous Docker-managed volume mounted at /var/lib/docker, NOT a bind mount. Ephemeral with the container.
Image allowlist: extend the existing worker_image validation to include nexus-worker-dind:latest.

4. Image picker UI

Extend the workspace-create form (already proposed for the Playwright variant in #2) with a third radio option: "With Docker (dind)". If #2 ships first, this is purely additive — same widget, one more value. If this issue ships first, the widget is born.

5. CLAUDE.md guidance

Add a section to the agent-facing default-CLAUDE.md:

### Spinning up disposable Docker stacks

If this worker was created with the "With Docker" image variant, you have a
local `dockerd` running inside the container. `docker ps`, `docker compose
up`, `docker build` all work. Containers you start live inside this worker —
they are NOT reachable from outside, and they are wiped when the worker is
removed.

To surface a dind-spawned service to the operator:

  docker run -p 3001:3001 nexus-worker:latest ...
  notify-preview 3001 "fresh nexus stack"

The `-p` publishes the inner container's port onto the worker's loopback;
notify-preview then registers the worker port with the operator as usual.

Out of scope

Rootless dind. Adds complexity (userns mapping, cgroup v2 quirks) for no security benefit in our trust model.
Persistent /var/lib/docker. Smoke-test workloads are ephemeral by nature; if persistence becomes a feature ask, add it then.
sysbox / kata / gvisor. Alternative runtimes that avoid --privileged. Stock Docker host doesn't have them; revisit if security posture changes.
Cross-worker container sharing. Each worker's dind is fully isolated; no inter-worker container DNS or routing.
Disk quota enforcement. Document the docker system prune habit; add quotas later if abuse is observed.

Verification

docker build -f worker/Dockerfile.dind -t nexus-worker-dind:latest worker/ succeeds in CI.
Spawn a workspace with the dind image. Open a session. Run docker version — both client and server should report 24+ (or whatever ships). Run docker run --rm hello-world — exits clean.
Build the agent-nexus image inside dind: git clone <agent-nexus repo> → cd nexus && docker build .. Should succeed (vfs is slow but correct).
docker compose up -d on the project's own compose file. Open notify-preview 3001 "inner nexus". Operator should see the pending preview, approve it, and reach the inner Nexus through their browser.
Tear down: docker compose down -v. Container's /var/lib/docker shrinks (or doesn't, with vfs; that's fine — the worker volume is anonymous and will be GC'd with the worker).
Default (non-dind) workers still work unchanged — no privileged flag, no docker CLI, no dockerd in their image.

Open questions

Image size budget. Dind variant likely adds ~300–400 MB on top of base. Acceptable for opt-in; would be excessive as default.
Storage driver choice. vfs is the safe default but very slow for large image pulls. Worth measuring before optimizing — fuse-overlayfs is the usual upgrade if needed.
Should the dind variant also include the Playwright runtime? They compose naturally for the smoke-test use case. Could either be a third variant (nexus-worker-dind-playwright:latest) or — if we want fewer combinations — accept that the agent installs Playwright into a fresh container inside dind on demand. I'd start with separate variants and combine only if usage warrants.

## Motivation Agents working on infrastructure or full-stack features routinely need to bring up disposable Docker stacks — to smoke-test a `docker compose` change, to validate that a new image builds and boots, or (the immediate driver) to end-to-end-smoke a UI feature of Nexus itself against a fresh Nexus instance without touching the operator's live one. The current worker image has no Docker access. Mounting the host `/var/run/docker.sock` is off the table — it makes worker→host container escape trivial, and the existing `tecnativa/docker-socket-proxy` allowlist (Nexus's own gateway to Docker) is intentionally narrow and host-Nexus-only. This issue proposes an **opt-in Docker-in-Docker (dind) worker image** that runs its own `dockerd` inside a sidecar pattern, isolated from the host's Docker. The agent gets a fully-functional `docker` CLI and `docker compose`, but the resulting containers live in the worker's own daemon — no host escape. Companion to #2 (Playwright MCP). The two together close the loop on agent-driven end-to-end testing: spin up a stack with dind, drive its UI with Playwright. They don't depend on each other. ## Scope - Opt-in per workspace at creation time, same shape as the proposed Playwright image variant in #2. - New worker image tag `nexus-worker-dind:latest`, derived from `nexus-worker:latest`. - Dind daemon runs **rootful inside the worker** (the worker container itself is already root and isolated; rootless dind is a complexity tax for no security gain in our model). - Agent gets `docker` CLI on `$PATH` and `docker compose` (v2 plugin). - Workspace volume layout: a fresh `/var/lib/docker` tmpfs or volume per worker, ephemeral by default. Persistence is out of scope. - Storage driver `vfs` for simplicity (no overlay-on-overlay woes); upgrade to `fuse-overlayfs` if vfs perf becomes a blocker. ## Threat model - **Single-operator design** carries over. Privileged container is a non-starter only if we cared about worker→host escape, which we already don't gate against in practice (the worker is root and has bind mounts to the worktree). - **Network isolation**: dind's daemon manages its own bridge (default `docker0` inside the worker). Containers it spawns are reachable from the worker but **not** from the host's `nexus-workers` bridge — they live behind dind's NAT inside the worker's network namespace. This is the intended isolation. - **Disk usage**: dind images and layers accumulate fast. Default to ephemeral storage (tmpfs or anonymous volume); document the `docker system prune` habit. A hard quota is out of scope for v1. - **Operator surfacing**: dind-spawned services that the agent wants the operator to see (e.g., a Nexus instance the agent stood up) already work via `notify-preview` — that CLI runs in the worker namespace, so when the agent maps `dind:3001 → worker:3001` via `docker run -p`, `notify-preview 3001` surfaces it to the operator unchanged. ## Verified facts (worth knowing before implementing) - **`--privileged` is required.** Dind without privileged fails at `iptables` setup (no `CAP_NET_ADMIN`). The dockerless `sysbox` alternative is a custom runtime — out of scope for a stock Docker host. - **`/var/lib/docker` MUST NOT be on overlayfs.** Overlay-on-overlay is broken on most kernels. Either use a named volume (ext4-backed), a tmpfs (smaller stacks fit fine), or set `--storage-driver vfs` explicitly. - **Dind needs its own `cgroupfs` mount.** Recent Docker handles this automatically via `--cgroupns=host` or the official `docker:dind` image's entrypoint; if we run dockerd ourselves we need to set `--exec-opt native.cgroupdriver=cgroupfs` and ensure systemd isn't fighting it. - **Boot ordering matters.** Worker boot must start dockerd BEFORE tmux idles, because the agent will expect `docker` to work the moment it opens a session. The existing `worker/entrypoint.sh` already has a pre-tmux setup block (where forgejo-mcp registration lives) — dockerd boot fits there. - **`docker compose` is a plugin** (not a separate binary) on Debian-derived images. Install via `docker-ce-cli` + `docker-compose-plugin` apt packages, NOT the standalone `docker-compose` Python script. - **`pnpm exec playwright install`-style binary downloads** can blow tmpfs `/var/lib/docker`. Plan storage size for the realistic agent workload (image builds with multi-hundred-MB layers). ## Suggested implementation ### 1. New worker image variant `worker/Dockerfile.dind`, extending the base image: ```dockerfile FROM nexus-worker:latest # Install Docker CE CLI + dockerd + compose plugin. RUN apt-get update && apt-get install -y --no-install-recommends \ ca-certificates curl gnupg \ && install -m 0755 -d /etc/apt/keyrings \ && curl -fsSL https://download.docker.com/linux/debian/gpg \ | gpg --dearmor -o /etc/apt/keyrings/docker.gpg \ && chmod a+r /etc/apt/keyrings/docker.gpg \ && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \ https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \ > /etc/apt/sources.list.d/docker.list \ && apt-get update && apt-get install -y --no-install-recommends \ docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin \ iptables \ && rm -rf /var/lib/apt/lists/* # vfs avoids overlay-on-overlay pain. Override via --storage-driver if needed. ENV DOCKERD_FLAGS="--storage-driver=vfs --host=unix:///var/run/docker.sock" COPY entrypoint-dind.sh /entrypoint-dind.sh RUN chmod +x /entrypoint-dind.sh ENTRYPOINT ["/entrypoint-dind.sh"] ``` CI: add to `.forgejo/workflows/ci.yml` build matrix alongside the proposed Playwright variant from #2. ### 2. Entrypoint augmentation `worker/entrypoint-dind.sh` runs the existing boot sequence, then before tmux setup: ```bash echo "==> Starting dockerd" mkdir -p /var/log nohup dockerd $DOCKERD_FLAGS > /var/log/dockerd.log 2>&1 & DOCKERD_PID=$! # Wait for socket up to ~10s; continue regardless (agent will see clear errors if it fails) for _ in $(seq 1 20); do if docker version >/dev/null 2>&1; then echo "==> dockerd ready (pid $DOCKERD_PID)" break fi sleep 0.5 done ``` Then delegate to the existing entrypoint's tmux setup. The dockerd process becomes a child of pid 1; the existing entrypoint already exits after tmux idles, so dockerd inherits the standard process tree. ### 3. Spawn-time HostConfig Nexus container spawn at [nexus/src/lib/server/workers/service.ts](nexus/src/lib/server/workers/service.ts): - **`HostConfig.Privileged: true`** for `nexus-worker-dind:latest` (and ONLY this image — never for default workers). - **`HostConfig.SecurityOpt: ['seccomp=unconfined']`** (some kernels need this for the inner dockerd to set up netns). - **Volume**: anonymous Docker-managed volume mounted at `/var/lib/docker`, NOT a bind mount. Ephemeral with the container. - Image allowlist: extend the existing `worker_image` validation to include `nexus-worker-dind:latest`. ### 4. Image picker UI Extend the workspace-create form (already proposed for the Playwright variant in #2) with a third radio option: **"With Docker (dind)"**. If #2 ships first, this is purely additive — same widget, one more value. If this issue ships first, the widget is born. ### 5. CLAUDE.md guidance Add a section to the agent-facing `default-CLAUDE.md`: ```markdown ### Spinning up disposable Docker stacks If this worker was created with the "With Docker" image variant, you have a local `dockerd` running inside the container. `docker ps`, `docker compose up`, `docker build` all work. Containers you start live inside this worker — they are NOT reachable from outside, and they are wiped when the worker is removed. To surface a dind-spawned service to the operator: docker run -p 3001:3001 nexus-worker:latest ... notify-preview 3001 "fresh nexus stack" The `-p` publishes the inner container's port onto the worker's loopback; notify-preview then registers the worker port with the operator as usual. ``` ## Out of scope - **Rootless dind.** Adds complexity (userns mapping, cgroup v2 quirks) for no security benefit in our trust model. - **Persistent `/var/lib/docker`.** Smoke-test workloads are ephemeral by nature; if persistence becomes a feature ask, add it then. - **sysbox / kata / gvisor.** Alternative runtimes that avoid `--privileged`. Stock Docker host doesn't have them; revisit if security posture changes. - **Cross-worker container sharing.** Each worker's dind is fully isolated; no inter-worker container DNS or routing. - **Disk quota enforcement.** Document the `docker system prune` habit; add quotas later if abuse is observed. ## Verification - `docker build -f worker/Dockerfile.dind -t nexus-worker-dind:latest worker/` succeeds in CI. - Spawn a workspace with the dind image. Open a session. Run `docker version` — both client and server should report 24+ (or whatever ships). Run `docker run --rm hello-world` — exits clean. - Build the agent-nexus image inside dind: `git clone <agent-nexus repo>` → `cd nexus && docker build .`. Should succeed (vfs is slow but correct). - `docker compose up -d` on the project's own compose file. Open `notify-preview 3001 "inner nexus"`. Operator should see the pending preview, approve it, and reach the inner Nexus through their browser. - Tear down: `docker compose down -v`. Container's `/var/lib/docker` shrinks (or doesn't, with vfs; that's fine — the worker volume is anonymous and will be GC'd with the worker). - Default (non-dind) workers still work unchanged — no privileged flag, no docker CLI, no dockerd in their image. ## Open questions - **Image size budget.** Dind variant likely adds ~300–400 MB on top of base. Acceptable for opt-in; would be excessive as default. - **Storage driver choice.** vfs is the safe default but very slow for large image pulls. Worth measuring before optimizing — fuse-overlayfs is the usual upgrade if needed. - **Should the dind variant also include the Playwright runtime?** They compose naturally for the smoke-test use case. Could either be a third variant (`nexus-worker-dind-playwright:latest`) or — if we want fewer combinations — accept that the agent installs Playwright into a fresh container inside dind on demand. I'd start with separate variants and combine only if usage warrants.