lz/agent-nexus
lz/agent-nexus
1
0
Fork 0
Code Issues 10 Pull requests 4 Projects Releases Packages 2 Wiki Activity Actions

Add in-container Playwright MCP so agents can navigate and screenshot apps they build #2

New issue
Closed
opened 2026-05-12 18:32:00 +02:00 by lz · 1 comment
lz commented 2026-05-12 18:32:00 +02:00
Owner
Copy link

Motivation

Agents that build web frontends should be able to verify their own work — that pages load, layouts aren't broken, console is clean, screenshots match expectations — without asking the operator to look. Today they can't. This issue adds a headless Chromium + @playwright/mcp server inside opt-in worker containers, exposed to Claude Code over HTTP MCP so the agent gets navigation, snapshot, click-by-locator, and screenshot tools.

Companion to #3 (operator port forwarding). The two features are independent: agents browse their own apps internally, while the operator gets a separate forward-then-open path. There's no overlap.

Scope

  • Opt-in per workspace. Chromium + Playwright runtime adds ~500–650 MB to the image, so we don't bake it into the default worker.
  • Headless Chromium only. Firefox/WebKit excluded.
  • Single Playwright MCP per container (not per session). Defer agent-teams isolation until we hit a conflict in practice.
  • Screenshots and accessibility-tree snapshots both available out of the box. Coordinate-based mouse tools also enabled (--caps vision) so agents can pick whichever interaction style fits the task.

Verified facts (worth knowing before implementing)

The original proposal doc got several things wrong; the implementation must use the corrected values:

  • Package is @playwright/mcp (Microsoft official). Latest at time of writing is 0.0.75. Fast-moving 0.x — pin a concrete version.
  • HTTP transport is enabled by --port N. Server exposes both /mcp (Streamable HTTP, modern) and /sse (legacy). Use /mcp for Claude Code 2.1.x.
  • Screenshots are in default core caps, NOT behind --caps vision. --caps vision enables coordinate-based mouse tools (browser_mouse_click_xy etc.) — useful for development but not required for screenshots.
  • Claude Code .mcp.json HTTP schema is {"type": "http", "url": "..."} — field name is type, not transport.
  • Headless Chromium in Docker requires --no-sandbox when running as root (we do), --init to reap zombie processes, and --shm-size=1g (or --ipc=host) to keep Chromium from crashing on /dev/shm exhaustion.
  • HTTP-transport MCPs were buggy in earlier Claude Code releases (anthropics/claude-code#39039) — verify against the pinned claude version before merging.

Suggested implementation

1. New worker image tag

Create worker/Dockerfile.playwright, extending the existing base image:

FROM nexus-worker:latest

RUN apt-get update && apt-get install -y --no-install-recommends \
      libnss3 libnspr4 \
      libatk1.0-0 libatk-bridge2.0-0 \
      libcups2 libdrm2 libgbm1 \
      libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
      libasound2 libpango-1.0-0 libcairo2 \
      fonts-liberation \
    && rm -rf /var/lib/apt/lists/*

RUN npm install -g @playwright/mcp@0.0.75
RUN npx playwright install chromium

COPY entrypoint-playwright.sh /entrypoint-playwright.sh
RUN chmod +x /entrypoint-playwright.sh
ENTRYPOINT ["/entrypoint-playwright.sh"]

Add to the .forgejo/workflows/ci.yml build matrix so it ships alongside nexus-worker:latest.

2. Entrypoint augmentation

worker/entrypoint-playwright.sh runs the same boot sequence as the default entrypoint, then before tmux new-session:

echo "==> Starting Playwright MCP"
nohup npx @playwright/mcp@0.0.75 \
  --port 8931 --host 127.0.0.1 \
  --headless --browser chromium --no-sandbox \
  --caps vision \
  > /var/log/playwright-mcp.log 2>&1 &

# Best-effort readiness wait; continue regardless (forgejo-mcp pattern)
for _ in 1 2 3 4 5 6 7 8 9 10; do
  curl -sf -o /dev/null http://127.0.0.1:8931/mcp && break
  sleep 0.5
done

echo "==> Registering playwright MCP with claude"
claude mcp add-json playwright \
  '{"type":"http","url":"http://127.0.0.1:8931/mcp"}' 2>/dev/null || true

Matches the forgejo-mcp registration shape at worker/entrypoint.sh#L91-L98.

3. Image selection at worker creation

  • Schema: new migration adds worker_image TEXT NOT NULL DEFAULT 'nexus-worker:latest' to the workers table.
  • Backend: in nexus/src/workers/service.ts#L137, use row.worker_image instead of config.WORKER_IMAGE when calling docker.createContainer. Validate against an allowlist (nexus-worker:latest, nexus-worker-playwright:latest) on the way in from the form so the value can't escape.
  • HostConfig at nexus/src/workers/service.ts#L147-L157: when the chosen image is the Playwright one, add Init: true and ShmSize: 1073741824 (1 GiB). Don't apply the shm bump to the default image — it's a real memory cost.
  • UI: add an image-picker (radio or select) to the workspace-create form. Labels: "Default" / "With browser (Playwright)". Default selection: "Default".

4. Convention for screenshots

Document in worker/CLAUDE.md (or similar) that agents should save screenshots to .agent/screenshots/<timestamp>-<short-desc>.png inside the worktree so the operator can find them.

Out of scope

  • Operator-side preview / port forwarding — tracked in #3.
  • Per-worktree (agent-teams) MCP isolation — defer until needed; --isolated is available later if it becomes a problem.
  • Firefox / WebKit support — Chromium-only keeps image size and complexity down.
  • Visual regression baselines — separate problem, not addressed here.

Verification

  • docker build -f worker/Dockerfile.playwright -t nexus-worker-playwright:latest worker/ succeeds in CI.
  • Spawn a workspace with the Playwright image. From a session, ask Claude list mcp servers and confirm playwright is listed.
  • From a session: start a tiny static server (python3 -m http.server 5173), have Claude navigate to http://localhost:5173 via Playwright, take a screenshot, confirm the file lands at .agent/screenshots/....
  • Confirm /dev/shm is sized correctly inside the container: df -h /dev/shm shows ~1 GiB.
  • Default workers still work (no shm bump, no Playwright in their image).

Open questions

  • Single Playwright MCP per container vs. per session — start with per container. Revisit if multiple agent-teams workspaces in one container start fighting over a single browser context.
  • Should we let users add their own image tags later? Allowlist is restrictive but safe; can be relaxed when there's a real need.
## Motivation Agents that build web frontends should be able to verify their own work — that pages load, layouts aren't broken, console is clean, screenshots match expectations — without asking the operator to look. Today they can't. This issue adds a headless Chromium + [`@playwright/mcp`](https://github.com/microsoft/playwright-mcp) server inside opt-in worker containers, exposed to Claude Code over HTTP MCP so the agent gets navigation, snapshot, click-by-locator, and screenshot tools. Companion to #3 (operator port forwarding). The two features are independent: agents browse their own apps internally, while the operator gets a separate forward-then-open path. There's no overlap. ## Scope - Opt-in per workspace. Chromium + Playwright runtime adds ~500–650 MB to the image, so we **don't** bake it into the default worker. - Headless Chromium only. Firefox/WebKit excluded. - Single Playwright MCP per container (not per session). Defer agent-teams isolation until we hit a conflict in practice. - Screenshots and accessibility-tree snapshots both available out of the box. Coordinate-based mouse tools also enabled (`--caps vision`) so agents can pick whichever interaction style fits the task. ## Verified facts (worth knowing before implementing) The original proposal doc got several things wrong; the implementation must use the corrected values: - Package is **`@playwright/mcp`** (Microsoft official). Latest at time of writing is `0.0.75`. Fast-moving 0.x — pin a concrete version. - HTTP transport is enabled by `--port N`. Server exposes **both** `/mcp` (Streamable HTTP, modern) and `/sse` (legacy). Use `/mcp` for Claude Code 2.1.x. - **Screenshots are in default `core` caps, NOT behind `--caps vision`.** `--caps vision` enables coordinate-based mouse tools (`browser_mouse_click_xy` etc.) — useful for development but not required for screenshots. - Claude Code `.mcp.json` HTTP schema is `{"type": "http", "url": "..."}` — field name is `type`, not `transport`. - Headless Chromium in Docker requires `--no-sandbox` when running as root (we do), `--init` to reap zombie processes, and `--shm-size=1g` (or `--ipc=host`) to keep Chromium from crashing on `/dev/shm` exhaustion. - HTTP-transport MCPs were buggy in earlier Claude Code releases ([anthropics/claude-code#39039](https://github.com/anthropics/claude-code/issues/39039)) — verify against the pinned `claude` version before merging. ## Suggested implementation ### 1. New worker image tag Create `worker/Dockerfile.playwright`, extending the existing base image: ```dockerfile FROM nexus-worker:latest RUN apt-get update && apt-get install -y --no-install-recommends \ libnss3 libnspr4 \ libatk1.0-0 libatk-bridge2.0-0 \ libcups2 libdrm2 libgbm1 \ libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \ libasound2 libpango-1.0-0 libcairo2 \ fonts-liberation \ && rm -rf /var/lib/apt/lists/* RUN npm install -g @playwright/mcp@0.0.75 RUN npx playwright install chromium COPY entrypoint-playwright.sh /entrypoint-playwright.sh RUN chmod +x /entrypoint-playwright.sh ENTRYPOINT ["/entrypoint-playwright.sh"] ``` Add to the `.forgejo/workflows/ci.yml` build matrix so it ships alongside `nexus-worker:latest`. ### 2. Entrypoint augmentation `worker/entrypoint-playwright.sh` runs the same boot sequence as the default entrypoint, then before `tmux new-session`: ```bash echo "==> Starting Playwright MCP" nohup npx @playwright/mcp@0.0.75 \ --port 8931 --host 127.0.0.1 \ --headless --browser chromium --no-sandbox \ --caps vision \ > /var/log/playwright-mcp.log 2>&1 & # Best-effort readiness wait; continue regardless (forgejo-mcp pattern) for _ in 1 2 3 4 5 6 7 8 9 10; do curl -sf -o /dev/null http://127.0.0.1:8931/mcp && break sleep 0.5 done echo "==> Registering playwright MCP with claude" claude mcp add-json playwright \ '{"type":"http","url":"http://127.0.0.1:8931/mcp"}' 2>/dev/null || true ``` Matches the forgejo-mcp registration shape at [worker/entrypoint.sh#L91-L98](worker/entrypoint.sh#L91-L98). ### 3. Image selection at worker creation - **Schema**: new migration adds `worker_image TEXT NOT NULL DEFAULT 'nexus-worker:latest'` to the `workers` table. - **Backend**: in [nexus/src/workers/service.ts#L137](nexus/src/workers/service.ts#L137), use `row.worker_image` instead of `config.WORKER_IMAGE` when calling `docker.createContainer`. Validate against an allowlist (`nexus-worker:latest`, `nexus-worker-playwright:latest`) on the way in from the form so the value can't escape. - **HostConfig** at [nexus/src/workers/service.ts#L147-L157](nexus/src/workers/service.ts#L147-L157): when the chosen image is the Playwright one, add `Init: true` and `ShmSize: 1073741824` (1 GiB). Don't apply the shm bump to the default image — it's a real memory cost. - **UI**: add an image-picker (radio or select) to the workspace-create form. Labels: "Default" / "With browser (Playwright)". Default selection: "Default". ### 4. Convention for screenshots Document in `worker/CLAUDE.md` (or similar) that agents should save screenshots to `.agent/screenshots/<timestamp>-<short-desc>.png` inside the worktree so the operator can find them. ## Out of scope - Operator-side preview / port forwarding — tracked in #3. - Per-worktree (agent-teams) MCP isolation — defer until needed; `--isolated` is available later if it becomes a problem. - Firefox / WebKit support — Chromium-only keeps image size and complexity down. - Visual regression baselines — separate problem, not addressed here. ## Verification - `docker build -f worker/Dockerfile.playwright -t nexus-worker-playwright:latest worker/` succeeds in CI. - Spawn a workspace with the Playwright image. From a session, ask Claude `list mcp servers` and confirm `playwright` is listed. - From a session: start a tiny static server (`python3 -m http.server 5173`), have Claude navigate to `http://localhost:5173` via Playwright, take a screenshot, confirm the file lands at `.agent/screenshots/...`. - Confirm `/dev/shm` is sized correctly inside the container: `df -h /dev/shm` shows ~1 GiB. - Default workers still work (no shm bump, no Playwright in their image). ## Open questions - Single Playwright MCP per container vs. per session — start with per container. Revisit if multiple agent-teams workspaces in one container start fighting over a single browser context. - Should we let users add their own image tags later? Allowlist is restrictive but safe; can be relaxed when there's a real need.
lz commented 2026-05-28 08:01:09 +02:00
Author
Owner
Copy link

Adding a related use case discovered while smoke-testing PR #31 (artifacts):

When an agent works on a UI feature of Nexus itself (operator-facing screens), the natural end-of-feature step is to drive the 10–15-step manual smoke autonomously instead of asking the operator to click through it. With this issue's Playwright MCP baked in, that becomes possible for any UI the agent runs locally (pnpm dev, python3 -m http.server, etc.) — but the Nexus-testing-Nexus case still needs a Nexus instance the agent can reach.

Two paths to make that work:

  1. Spin up a fresh Nexus stack via Docker-in-Docker inside the worker → drive its UI with this Playwright MCP. Self-contained, no risk to the operator's live instance. Tracked separately in the companion dind issue (filed alongside this comment).
  2. Surface the operator's live Nexus into the agent's network namespace so Playwright MCP can drive it. Inverted shape of #3's port-forwarding: instead of the operator approving outbound (worker→host), the agent gets a controlled inbound channel (host→worker). Useful only for read-only flows; destructive smoke would risk the operator's actual data. Not pursuing this without a clearer use case.

Practical implication for this issue: the proposed scope is correct as-is — single Playwright MCP per container, --caps vision enabled, screenshots in .agent/screenshots/. The dind companion is purely additive: it lets the MCP point at a localhost URL that the agent itself stood up, which is already the dominant use case described here.

The two issues together would let agents close the smoke-test loop for any feature they ship — Nexus-internal or otherwise. Worth implementing them in either order; they don't depend on each other.

Adding a related use case discovered while smoke-testing PR #31 (artifacts): When an agent works on a **UI feature of Nexus itself** (operator-facing screens), the natural end-of-feature step is to drive the 10–15-step manual smoke autonomously instead of asking the operator to click through it. With this issue's Playwright MCP baked in, that becomes possible for *any* UI the agent runs locally (`pnpm dev`, `python3 -m http.server`, etc.) — but the Nexus-testing-Nexus case still needs a Nexus instance the agent can reach. Two paths to make that work: 1. **Spin up a fresh Nexus stack via Docker-in-Docker inside the worker** → drive its UI with this Playwright MCP. Self-contained, no risk to the operator's live instance. Tracked separately in the companion dind issue (filed alongside this comment). 2. **Surface the operator's live Nexus into the agent's network namespace** so Playwright MCP can drive it. Inverted shape of #3's port-forwarding: instead of the operator approving outbound (worker→host), the agent gets a controlled inbound channel (host→worker). Useful only for read-only flows; destructive smoke would risk the operator's actual data. Not pursuing this without a clearer use case. **Practical implication for this issue:** the proposed scope is correct as-is — single Playwright MCP per container, `--caps vision` enabled, screenshots in `.agent/screenshots/`. The dind companion is purely additive: it lets the MCP point at a localhost URL that the agent itself stood up, which is already the dominant use case described here. The two issues together would let agents close the smoke-test loop for any feature they ship — Nexus-internal or otherwise. Worth implementing them in either order; they don't depend on each other.
lz closed this issue 2026-05-30 22:52:43 +02:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
fix/workspace-artifacts
feat/codegraph
fix/comment-collapse
feat/api-proxy
No results found.
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lz/agent-nexus#2
No description provided.