Nested/side-by-side Nexus hangs on spawn: WORKERS_BRIDGE_SUBNET defaults collide #59

New issue

Open

opened 2026-07-12 12:28:29 +02:00 by lz · 0 comments

lz commented

2026-07-12 12:28:29 +02:00

Owner

Symptom

Spawning a worker from a second Nexus instance that runs inside a Nexus worker (Docker-in-Docker) hangs indefinitely at "creating the container". The UI shows no error. Worse, the hang is accompanied by a loss of network connectivity from the worker container to the host Nexus — notify-preview / notify-artifact start timing out.

Cause

WORKERS_BRIDGE_SUBNET defaults to 172.30.0.0/16 (nexus/src/lib/server/config.ts), for every instance.

A Nexus worker container is itself attached to the host Nexus's nexus-workers bridge, so its eth0 is on 172.30.0.0/16 and its default gateway is 172.30.0.1 — which is also how it reaches the host Nexus (NEXUS_URL=http://172.30.0.1:3001).

When a Nexus running inside that worker calls ensureWorkersBridge, it creates a bridge with the same default 172.30.0.0/16. It is creating a network on top of its own gateway. The route to 172.30.0.1 is hijacked by the new bridge, the callback path dies, and the spawn wedges.

Reproduced on the feat/dockview-pr-a branch while standing up an isolated test instance in a DinD worker. Confirmed by:

docker network inspect nexus-workers inside DinD → 172.30.0.0/16
/proc/net/route in the worker → default gw 172.30.0.1 via eth0, network 172.30.0.0/16
curl http://172.30.0.1:3001/api/state → timeout while the bridge existed; 200 immediately after docker network rm nexus-workers

Workaround: set WORKERS_BRIDGE_SUBNET (and ideally WORKERS_BRIDGE_NAME) to something non-colliding, e.g. 172.28.0.0/16.

Why it's worth fixing

Two things are wrong independently:

It fails silently, as a hang. AGENTS.md already warns "pick a /16 that doesn't collide with existing docker networks" — but nothing enforces it, and the failure mode is an indefinite hang rather than an error. That's the expensive part: there is no signal pointing at the subnet.
The default is not self-aware. Nexus can see its own container's networks. It could detect that the subnet it is about to create overlaps a subnet it is currently reachable on.

Suggested fix (smallest first)

Minimum: in ensureWorkersBridge (nexus/src/lib/server/workers/workers-bridge.ts), before creating the bridge, check whether the requested subnet overlaps any network already visible to this Docker daemon or any address on the Nexus process's own interfaces. If it overlaps, throw a clear error naming the collision and pointing at WORKERS_BRIDGE_SUBNET. A loud failure beats a hang.

Better: on collision, auto-pick the next free /16 from a candidate list (172.30, 172.28, 172.27, …) and log which one was chosen. WORKERS_BRIDGE_SUBNET stays as the explicit override.

Note the subnet is only consulted on first create — an existing network with the right name is reused as-is — so the check only needs to run on the create path.

## Symptom Spawning a worker from a second Nexus instance that runs *inside* a Nexus worker (Docker-in-Docker) hangs indefinitely at "creating the container". The UI shows no error. Worse, the hang is accompanied by a **loss of network connectivity from the worker container to the host Nexus** — `notify-preview` / `notify-artifact` start timing out. ## Cause `WORKERS_BRIDGE_SUBNET` defaults to `172.30.0.0/16` (`nexus/src/lib/server/config.ts`), for *every* instance. A Nexus worker container is itself attached to the host Nexus's `nexus-workers` bridge, so its `eth0` is on `172.30.0.0/16` and its default gateway is `172.30.0.1` — which is also how it reaches the host Nexus (`NEXUS_URL=http://172.30.0.1:3001`). When a Nexus running *inside* that worker calls `ensureWorkersBridge`, it creates a bridge with the same default `172.30.0.0/16`. It is creating a network on top of its own gateway. The route to `172.30.0.1` is hijacked by the new bridge, the callback path dies, and the spawn wedges. Reproduced on the `feat/dockview-pr-a` branch while standing up an isolated test instance in a DinD worker. Confirmed by: - `docker network inspect nexus-workers` inside DinD → `172.30.0.0/16` - `/proc/net/route` in the worker → default gw `172.30.0.1` via `eth0`, network `172.30.0.0/16` - `curl http://172.30.0.1:3001/api/state` → timeout while the bridge existed; **200 immediately after `docker network rm nexus-workers`** Workaround: set `WORKERS_BRIDGE_SUBNET` (and ideally `WORKERS_BRIDGE_NAME`) to something non-colliding, e.g. `172.28.0.0/16`. ## Why it's worth fixing Two things are wrong independently: 1. **It fails silently, as a hang.** `AGENTS.md` already warns "pick a /16 that doesn't collide with existing docker networks" — but nothing enforces it, and the failure mode is an indefinite hang rather than an error. That's the expensive part: there is no signal pointing at the subnet. 2. **The default is not self-aware.** Nexus can see its own container's networks. It could detect that the subnet it is about to create overlaps a subnet it is *currently reachable on*. ## Suggested fix (smallest first) **Minimum:** in `ensureWorkersBridge` (`nexus/src/lib/server/workers/workers-bridge.ts`), before creating the bridge, check whether the requested subnet overlaps any network already visible to this Docker daemon **or** any address on the Nexus process's own interfaces. If it overlaps, **throw a clear error** naming the collision and pointing at `WORKERS_BRIDGE_SUBNET`. A loud failure beats a hang. **Better:** on collision, auto-pick the next free /16 from a candidate list (`172.30`, `172.28`, `172.27`, …) and log which one was chosen. `WORKERS_BRIDGE_SUBNET` stays as the explicit override. Note the subnet is only consulted on *first create* — an existing network with the right name is reused as-is — so the check only needs to run on the create path.

lz referenced this issue from a commit

2026-07-17 18:10:40 +02:00

docs(quota): guard Task 12 against the DinD bridge subnet collision