ci: fix build cache, stop rebuilding what a PR did not touch #182

Closed
lz wants to merge 5 commits from ci/cache-and-target-selection into main
Owner

What

The images job spent roughly 60% of its wall time moving cache rather than building. Measured on run 945 (build step, 535s):

cache export   base 176.8s + playwright 146.5s + dind 115.6s
               + dind-playwright 65.7s + webapp 50.5s
cache import   170.9s stalled on a step buildx had already marked CACHED

Root cause

type=gha stores each cache scope as one opaque tarball. All four worker targets are FROM base, but nothing can be shared between scopes, so the runner's cache store held three near-identical copies of the same base layers:

453188176 bytes
453188080 bytes
453178159 bytes

Same content, three copies, differing only in gzip framing. Every run re-uploaded all three. And because a scope is all-or-nothing on the way down, a cache hit still cost a full download — which is why the log showed a few-kilobyte COPY correctly marked CACHED yet taking 170.9s:

#70 [webapp build 8/13] COPY scripts ./scripts
#70 CACHED
#70 DONE 170.9s

The cache was functioning. It just cost more than it saved.

Changes

  • Registry-backed layer cache instead of type=gha. Layers are content-addressed, so the base every worker variant shares is stored and fetched once. image-manifest=true,oci-mediatypes=true is required — Forgejo rejects the default OCI-index cache form, the same reason provenance = false was already set.
  • PRs read the cache, never write it. A PR pays no export cost and cannot poison the shared cache from a branch that may not merge. CACHE_WRITE is set on main only.
  • Target selection by changed paths. A PR touching only nexus/ no longer bakes the four worker images, and vice versa. main always bakes everything, so :latest and the shared cache are never built from a partial graph. Anything outside nexus/ and worker/ falls back to baking everything.
  • Workflow-level concurrency. The runner is a single LXC with capacity: 1, so three pushes to a PR queued three full runs and the two obsolete ones blocked the one that mattered.
  • nexus/Dockerfile installed with --frozen-lockfile=false while the CI job used --frozen-lockfile, so the shipped image could carry dependency versions CI never linted, typechecked or tested.

pnpm run build deliberately stays in the nexus job even though the webapp image build runs the identical command: per AGENTS.md fact #32 it is the only gate that validates SvelteKit route-module exports, and running it there fails the run before the bake starts rather than after it.

Verification

Measured against the live Forgejo registry, on a fresh builder with no local cache — exactly what CI gets each run:

before after
webapp, warm cache ~220s of cache movement 24.2s, all layers CACHED, zero blob re-download
all 4 workers, warm cache — 59.4s, 70 CACHED steps

Also confirmed: Forgejo accepts the cache manifests; the shared base layer digest sha256:470d9dd2… is written by both [base] and [playwright] into a single stored blob; and the target-selection logic was unit-tested over 9 cases including the fail-safe ones (empty diff and a nexus-extra/ decoy prefix both fall back to default).

All five buildcache-* slots have been pre-warmed in the registry, so this run should land warm rather than cold.

The playwright-version parity invariants (nexus/src/lib/playwright-version-parity.test.ts) were re-checked by replicating each of its assertions against the edited files — all pass.

Note

This PR is itself the first end-to-end test on the runner: every component was verified locally against the real registry, but the workflow has not yet executed in CI. It touches docker-bake.hcl and the workflow, so it will bake all five targets.

🤖 Generated with Claude Code

## What The `images` job spent roughly **60% of its wall time moving cache rather than building**. Measured on run 945 (build step, 535s): ``` cache export base 176.8s + playwright 146.5s + dind 115.6s + dind-playwright 65.7s + webapp 50.5s cache import 170.9s stalled on a step buildx had already marked CACHED ``` ## Root cause `type=gha` stores each cache scope as **one opaque tarball**. All four worker targets are `FROM base`, but nothing can be shared between scopes, so the runner's cache store held three near-identical copies of the same base layers: ``` 453188176 bytes 453188080 bytes 453178159 bytes ``` Same content, three copies, differing only in gzip framing. Every run re-uploaded all three. And because a scope is all-or-nothing on the way down, a *cache hit* still cost a full download — which is why the log showed a few-kilobyte `COPY` correctly marked CACHED yet taking 170.9s: ``` #70 [webapp build 8/13] COPY scripts ./scripts #70 CACHED #70 DONE 170.9s ``` The cache was functioning. It just cost more than it saved. ## Changes - **Registry-backed layer cache instead of `type=gha`.** Layers are content-addressed, so the base every worker variant shares is stored and fetched once. `image-manifest=true,oci-mediatypes=true` is required — Forgejo rejects the default OCI-index cache form, the same reason `provenance = false` was already set. - **PRs read the cache, never write it.** A PR pays no export cost and cannot poison the shared cache from a branch that may not merge. `CACHE_WRITE` is set on main only. - **Target selection by changed paths.** A PR touching only `nexus/` no longer bakes the four worker images, and vice versa. main always bakes everything, so `:latest` and the shared cache are never built from a partial graph. Anything outside `nexus/` and `worker/` falls back to baking everything. - **Workflow-level `concurrency`.** The runner is a single LXC with `capacity: 1`, so three pushes to a PR queued three full runs and the two obsolete ones blocked the one that mattered. - **`nexus/Dockerfile` installed with `--frozen-lockfile=false`** while the CI job used `--frozen-lockfile`, so the shipped image could carry dependency versions CI never linted, typechecked or tested. `pnpm run build` deliberately **stays** in the `nexus` job even though the webapp image build runs the identical command: per AGENTS.md fact #32 it is the only gate that validates SvelteKit route-module exports, and running it there fails the run before the bake starts rather than after it. ## Verification Measured against the live Forgejo registry, on a **fresh builder** with no local cache — exactly what CI gets each run: | | before | after | |---|---|---| | webapp, warm cache | ~220s of cache movement | **24.2s**, all layers CACHED, zero blob re-download | | all 4 workers, warm cache | — | **59.4s**, 70 CACHED steps | Also confirmed: Forgejo accepts the cache manifests; the shared base layer digest `sha256:470d9dd2…` is written by both `[base]` and `[playwright]` into a single stored blob; and the target-selection logic was unit-tested over 9 cases including the fail-safe ones (empty diff and a `nexus-extra/` decoy prefix both fall back to `default`). All five `buildcache-*` slots have been pre-warmed in the registry, so this run should land warm rather than cold. The playwright-version parity invariants (`nexus/src/lib/playwright-version-parity.test.ts`) were re-checked by replicating each of its assertions against the edited files — all pass. ## Note This PR is itself the first end-to-end test on the runner: every component was verified locally against the real registry, but the workflow has not yet executed in CI. It touches `docker-bake.hcl` and the workflow, so it will bake **all five** targets. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
lz added 2 commits 2026-09-19 17:45:14 +02:00
The `images` job spent roughly 60% of its wall time moving cache around
rather than building. Measured on run 945 (build step, 535s):

  cache export   base 176.8s + playwright 146.5s + dind 115.6s
                 + dind-playwright 65.7s + webapp 50.5s
  cache import   170.9s stalled on a step buildx had already marked CACHED

Root cause: `type=gha` stores each scope as ONE OPAQUE TARBALL. All four
worker targets are `FROM base`, but nothing can be shared between scopes,
so the runner's cache store held three near-identical copies of the same
base layers (453188176 / 453188080 / 453178159 bytes) and every run
re-uploaded all three — and re-downloaded a whole 432 MB blob just to
satisfy a cache HIT.

Switch to registry-backed cache in the Forgejo registry. Layers are
content-addressed there, so the shared base is stored and fetched once
regardless of how many targets reference it. `image-manifest=true,
oci-mediatypes=true` is required — Forgejo rejects the default OCI-index
cache form, the same reason `provenance = false` was already set.

Also:

- PRs now READ the cache and never write it. A PR pays no export cost at
  all, and cannot poison the shared cache from a branch that may not
  merge. `CACHE_WRITE` is set on main only.

- A PR that only touches nexus/ no longer bakes the four worker images,
  and vice versa. main still bakes everything, so :latest and the shared
  cache are never built from a partial graph. Anything outside nexus/ and
  worker/ falls back to baking everything.

- Workflow-level `concurrency`, cancelling superseded PR runs. The runner
  is a single LXC with `capacity: 1`, so three pushes to a PR queued three
  full runs and the two obsolete ones blocked the one that mattered. main
  is never cancelled: its runs publish :latest and write the cache.

- nexus/Dockerfile installed with `--frozen-lockfile=false` while the CI
  job used `--frozen-lockfile`, so the shipped image could carry
  dependency versions CI never linted, typechecked or tested.

`pnpm run build` stays in the `nexus` job despite the webapp image build
running the identical command: per AGENTS.md fact #32 it is the only gate
that validates SvelteKit route-module exports, and running it there fails
the run before the bake starts rather than after it.

Verified against the live Forgejo registry (throwaway package, since
deleted): cache export writes a manifest Forgejo accepts, a warm rerun is
all-CACHED with no blob re-download, and the shared base layer digest is
written by both [base] and [playwright] into a single stored blob.
ci: make cancel-in-progress a literal, not an expression
All checks were successful
ci / nexus (pull_request) Successful in 8m16s
ci / images (pull_request) Successful in 20m12s
c42e13a9a5
`cancel-in-progress: ${{ github.event_name == 'pull_request' }}` was there
to exempt main from cancellation. Forgejo 16.0.5 does implement workflow
concurrency (action_run carries concurrency_group / concurrency_type), but
whether it evaluates an expression in this specific field is unverified
here — and an expression it failed to parse would break the entire
workflow, not just the exemption.

The group is keyed on github.ref, so a run is only ever cancelled by a
newer push to the same branch, which is the run you wanted anyway. For
main that means two pushes inside one build window supersede each other;
rare, and the later one republishes :latest and rewrites the cache
regardless. Cheap insurance against a parse error that would cost far more.
ci: let every worker target read every worker cache ref
All checks were successful
ci / nexus (pull_request) Successful in 7m54s
ci / images (pull_request) Successful in 19m13s
ee1da41377
CI run 949 built the `base` stage TWICE inside a single bake. Two parallel
chains, from the log:

  #19 → #21 → #23 → #29..#45   rebuilt  (npm install -g = 154.2s)
  #46..#62                      all CACHED

`docker buildx bake` runs each target as a SEPARATE solve, not one shared
graph. All four worker targets are `FROM base`, but each could only import its
own `buildcache-<target>` ref — so a solve whose own ref did not match rebuilt
the shared base chain from scratch even though a sibling ref already held it,
while another solve sat on a clean hit.

That is most of the 1055s bake step: 154s of `npm install -g` plus the whole
20-step chain behind it, then the playwright (349.8s) and dind (374.7s) addons
on top of the rebuilt base.

Fix: every worker target now imports all four worker refs. A cache import is a
manifest fetch, so reading four instead of one costs almost nothing, and it
removes the race entirely — any solve can find the shared base chain wherever
it happens to live. `cache-to` stays one ref per target so concurrent solves
never write the same ref.

webapp keeps a single ref: different Dockerfile, different context, no shared
stages with the workers.

Note this did NOT reproduce locally, which is why it shipped: a local fresh
builder still hit cache via its own content store. It only appears on the
runner, whose buildx (v0.30.1, from Alpine's docker-cli-buildx) is older than
the v0.33.0 used locally. If this does not fully close the gap, pinning buildx
in the workflow is the next lever.
ci: TEMPORARY - let this PR populate the registry cache
Some checks failed
ci / images (pull_request) Has been cancelled
ci / nexus (pull_request) Has been cancelled
d56a335364
Reverted by the next commit. Do not merge with this in.

The registry cache currently in the Forgejo registry was written by hand from
a workstation running buildx v0.33.0. CI runs buildx v0.30.1 (Alpine's
docker-cli-buildx). Its cache keys do not line up with what CI computes, so
run 949 and run 950 both rebuilt the shared `base` chain from scratch —
`npm install -g` for 154s and 141s respectively — while a separately imported
copy sat CACHED beside it. Adding more cache-from refs (the previous commit)
did not help, because the problem is keys that do not match, not refs that
were not consulted.

Because PRs are deliberately read-only on cache, this PR can never write a
cache CI could then read, so it cannot demonstrate the change either way. That
makes the 687s (main, gha) vs 1212s (PR, registry) comparison invalid rather
than merely unfavourable.

So: this commit lets the PR write the cache ONCE, using CI's own toolchain.
The revert commit that follows is the actual measurement — read-only, steady
state config, reading a cache CI itself produced. If `images` does not drop
well below 687s there, the registry backend is wrong for this runner and the
honest move is to go back to type=gha.

The worker targets on this branch are byte-identical to main, so the cache
this writes is valid for main too.
ci: give a PR its own cache refs instead of making it read-only
Some checks failed
ci / nexus (pull_request) Successful in 8m2s
ci / images (pull_request) Failing after 23m21s
pr-image-cleanup / delete-pr-images (pull_request) Successful in 22s
4c069c8284
Replaces both the read-only-on-PR rule and the temporary commit that lifted it.

CACHE_SCOPE is "" on main and "pr-<n>-" on a pull request:

  main   writes buildcache-<slot>          reads buildcache-<slot>
  a PR   writes pr-<n>-buildcache-<slot>   reads BOTH its own and main's

The read-only rule avoided a real hazard — the cache tag is mutable, so a PR
writing main's ref replaces main's index with the state of a branch that may
never merge — but it paid for that with two problems. A PR could not benefit
from its own previous run, and, more seriously, a PR could not bootstrap a
cache at all. That is why runs 949 and 950 proved nothing: the only registry
cache in existence had been written by hand from a workstation on buildx
v0.33.0, CI runs v0.30.1, the keys did not line up, and a read-only PR had no
way to produce a cache CI could then read. The 687s (main, gha) vs 1212s (PR,
registry) comparison was invalid rather than merely unfavourable.

Scoping keeps the hazard closed — main's index is never written by a PR — and
lets a PR populate and then reuse its own cache, so the next run on this branch
is the measurement that was previously impossible.

The `pr-<n>-` prefix is chosen so the refs match the owner's existing `pr-.+`
package retention rule; pr-image-cleanup.yml also deletes them on PR close, the
same prompt-cleanup-plus-backstop arrangement already used for PR images.

Worker targets additionally read all four worker refs, since bake runs each
target as a separate solve and all four share the `base` chain.
Author
Owner

Closing unmerged. The registry-backed cache in this branch never demonstrated a win in CI:

run config images job
945 main, type=gha 687s
949 this branch, per-target cache refs 1212s
950 this branch, all-refs cache-from same double-build
952 this branch, PR-scoped refs failed

Two things went wrong, and both are worth recording:

  1. The cache was never usable by CI. It had been pre-warmed by hand from a workstation on buildx v0.33.0; the runner uses v0.30.1 from Alpine's docker-cli-buildx. The keys did not line up, so every run rebuilt the shared base chain (npm install -g ~150s) while a separately imported copy sat CACHED beside it. Local verification had used the same buildx that wrote the cache, so it hit every time and proved nothing.

  2. Run 952 hit a Forgejo bug. Two concurrent uploads of the same blob digest race; one wins, the other violates UQE_package_blob_sha256 and Forgejo dereferences a nil pointer in services/packages/container/blob.go:98, surfacing to BuildKit as error writing layer blob: unknown. bake pushes all targets concurrently and the worker images share most layers, so this is easy to trigger. Worth reporting upstream (Forgejo 16.0.5).

The genuine findings are kept: the measured critical path is base 189s -> playwright 395s -> dind-playwright 468s, the dind addon installs twice, and ~780s of that run was image/cache transfer over a NAT hairpin. Those motivate the composable-worker-image redesign in the issue opened alongside this, which removes the dind-playwright build target entirely.

Branch deleted; build-cache and pr-182-* tags removed from the registry.

Closing unmerged. The registry-backed cache in this branch never demonstrated a win in CI: | run | config | `images` job | |---|---|---| | 945 | main, `type=gha` | **687s** | | 949 | this branch, per-target cache refs | 1212s | | 950 | this branch, all-refs cache-from | same double-build | | 952 | this branch, PR-scoped refs | **failed** | Two things went wrong, and both are worth recording: 1. **The cache was never usable by CI.** It had been pre-warmed by hand from a workstation on buildx v0.33.0; the runner uses v0.30.1 from Alpine's `docker-cli-buildx`. The keys did not line up, so every run rebuilt the shared `base` chain (`npm install -g` ~150s) while a separately imported copy sat CACHED beside it. Local verification had used the same buildx that wrote the cache, so it hit every time and proved nothing. 2. **Run 952 hit a Forgejo bug.** Two concurrent uploads of the same blob digest race; one wins, the other violates `UQE_package_blob_sha256` and Forgejo dereferences a nil pointer in `services/packages/container/blob.go:98`, surfacing to BuildKit as `error writing layer blob: unknown`. `bake` pushes all targets concurrently and the worker images share most layers, so this is easy to trigger. Worth reporting upstream (Forgejo 16.0.5). The genuine findings are kept: the measured critical path is `base 189s -> playwright 395s -> dind-playwright 468s`, the dind addon installs **twice**, and ~780s of that run was image/cache transfer over a NAT hairpin. Those motivate the composable-worker-image redesign in the issue opened alongside this, which removes the `dind-playwright` build target entirely. Branch deleted; build-cache and `pr-182-*` tags removed from the registry.
lz closed this pull request 2026-09-19 20:00:13 +02:00
Some checks failed
ci / nexus (pull_request) Successful in 8m2s
ci / images (pull_request) Failing after 23m21s
pr-image-cleanup / delete-pr-images (pull_request) Successful in 22s

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
lz/agent-nexus!182
No description provided.