Hardcoded workers-bridge subnet (172.30.0.0/16) collides and kills egress when Nexus bootstraps inside a DinD worker #74

New issue

Closed

opened 2026-07-17 07:27:52 +02:00 by lz · 0 comments

lz commented

2026-07-17 07:27:52 +02:00

Owner

Impact: Total loss of external network connectivity in nexus-agent-nexus (dev container). Any Claude Code / API / package-install activity inside the container fails. The failure is silent and misleading: DNS keeps resolving, so it does not look like a network-layer problem.

Root cause

nexus/src/lib/server/config.ts:

WORKERS_BRIDGE_NAME:   z.string().default('nexus-workers'),      // L14
WORKERS_BRIDGE_SUBNET: z.string().default('172.30.0.0/16'),     // L17

When Nexus bootstraps inside a DinD worker that is itself attached to the outer nexus-workers network (172.30.0.0/16, gw 172.30.0.1), the inner daemon creates a bridge with the identical name and subnet. The inner network is created explicitly with Subnet: 172.30.0.0/16 (label nexus-managed=true) — this is not IPAM auto-allocation, so Docker's own pool-collision avoidance never engages.

Result inside the container: 172.30.0.1 (the outer default gateway) is assigned to the inner bridge and becomes a /32 host LOCAL route. Two competing 172.30.0.0/16 routes exist (eth0 and br-*), and all gateway-bound traffic is delivered locally. Egress is dead.

Observed on nexus-agent-nexus (inner net created 2026-07-16T19:39:12Z):

$ hostname -I
172.30.0.3 172.17.0.1 172.30.0.1     # <-- owns the outer gateway IP

$ cat /proc/net/route
eth0             00000000 01001EAC   # default via 172.30.0.1
eth0             00001EAC 00000000   # 172.30.0.0/16
br-f6aae6a745af  00001EAC 00000000   # 172.30.0.0/16  <-- COLLISION

$ getent hosts api.anthropic.com      # resolves fine (local 127.0.0.11)
$ curl https://1.1.1.1                # http=000 (dead)

The in-code comment ("Pick a /16 that doesn't collide with existing docker networks. Only consulted on first create.") assumes a human picks a safe value. That assumption breaks under nesting, where the colliding network is the one the bootstrapping container is currently reachable on.

Collateral: an npm install -g @anthropic-ai/claude-code that ran during the outage completed "successfully" but silently skipped its platform-native optional dependency, leaving a claude wrapper with no binary ("native binary not installed"). Anything installing packages during such an outage is at risk of half-installed state.

Mitigation applied

docker exec nexus-agent-nexus docker network rm nexus-workers
docker exec nexus-agent-nexus npm install -g @anthropic-ai/claude-code@2.1.212

Connectivity and Claude Code restored. This will recur on the next nested bootstrap.

Suggested fixes (roughly in order of value)

Detect the collision before creating: enumerate the container's own routes / interface subnets and refuse to create a bridge overlapping any of them. Fail loudly with an actionable error rather than severing the network.
Don't hardcode a default subnet: let Docker's IPAM allocate, and only pin a subnet when explicitly configured. IPAM already avoids collisions with networks the daemon knows about.
Nesting-awareness: when Nexus detects it is running inside a Nexus-managed worker, derive a distinct bridge name/subnet (or reuse the existing network via WORKER_NETWORK) instead of recreating the outer topology.
Guardrail: make the create path refuse any subnet containing the container's current default gateway. That single check would have prevented this outage.

**Impact:** Total loss of external network connectivity in `nexus-agent-nexus` (dev container). Any Claude Code / API / package-install activity inside the container fails. The failure is silent and misleading: DNS keeps resolving, so it does not look like a network-layer problem. ## Root cause `nexus/src/lib/server/config.ts`: ```ts WORKERS_BRIDGE_NAME: z.string().default('nexus-workers'), // L14 WORKERS_BRIDGE_SUBNET: z.string().default('172.30.0.0/16'), // L17 ``` When Nexus bootstraps *inside* a DinD worker that is itself attached to the outer `nexus-workers` network (`172.30.0.0/16`, gw `172.30.0.1`), the inner daemon creates a bridge with the identical name **and** subnet. The inner network is created explicitly with `Subnet: 172.30.0.0/16` (label `nexus-managed=true`) — this is not IPAM auto-allocation, so Docker's own pool-collision avoidance never engages. Result inside the container: `172.30.0.1` (the **outer** default gateway) is assigned to the inner bridge and becomes a `/32 host LOCAL` route. Two competing `172.30.0.0/16` routes exist (`eth0` and `br-*`), and all gateway-bound traffic is delivered locally. Egress is dead. Observed on `nexus-agent-nexus` (inner net created `2026-07-16T19:39:12Z`): ```console $ hostname -I 172.30.0.3 172.17.0.1 172.30.0.1 # <-- owns the outer gateway IP $ cat /proc/net/route eth0 00000000 01001EAC # default via 172.30.0.1 eth0 00001EAC 00000000 # 172.30.0.0/16 br-f6aae6a745af 00001EAC 00000000 # 172.30.0.0/16 <-- COLLISION $ getent hosts api.anthropic.com # resolves fine (local 127.0.0.11) $ curl https://1.1.1.1 # http=000 (dead) ``` The in-code comment ("Pick a /16 that doesn't collide with existing docker networks. Only consulted on first create.") assumes a human picks a safe value. That assumption breaks under nesting, where the colliding network is the one the bootstrapping container is currently reachable on. **Collateral:** an `npm install -g @anthropic-ai/claude-code` that ran during the outage completed "successfully" but silently skipped its platform-native optional dependency, leaving a `claude` wrapper with no binary ("native binary not installed"). Anything installing packages during such an outage is at risk of half-installed state. ## Mitigation applied ```sh docker exec nexus-agent-nexus docker network rm nexus-workers docker exec nexus-agent-nexus npm install -g @anthropic-ai/claude-code@2.1.212 ``` Connectivity and Claude Code restored. This will recur on the next nested bootstrap. ## Suggested fixes (roughly in order of value) 1. **Detect the collision before creating:** enumerate the container's own routes / interface subnets and refuse to create a bridge overlapping any of them. Fail loudly with an actionable error rather than severing the network. 2. **Don't hardcode a default subnet:** let Docker's IPAM allocate, and only pin a subnet when explicitly configured. IPAM already avoids collisions with networks the daemon knows about. 3. **Nesting-awareness:** when Nexus detects it is running inside a Nexus-managed worker, derive a distinct bridge name/subnet (or reuse the existing network via `WORKER_NETWORK`) instead of recreating the outer topology. 4. **Guardrail:** make the create path refuse any subnet containing the container's current default gateway. That single check would have prevented this outage.

lz added the

bug

label

2026-07-17 07:27:52 +02:00

lz referenced this issue from a commit

2026-07-17 17:42:01 +02:00

fix(workers-bridge): refuse a subnet that swallows the default gateway (#74)

lz referenced this issue from a pull request that will close it,

2026-07-17 17:42:15 +02:00

fix(workers-bridge): refuse a subnet that swallows the default gateway (#74) #79

lz closed this issue

2026-07-17 17:43:36 +02:00