ci: fix build cache, stop rebuilding what a PR did not touch #182
Loading…
Reference in a new issue
No description provided.
Delete branch "ci/cache-and-target-selection"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
The
imagesjob spent roughly 60% of its wall time moving cache rather than building. Measured on run 945 (build step, 535s):Root cause
type=ghastores each cache scope as one opaque tarball. All four worker targets areFROM base, but nothing can be shared between scopes, so the runner's cache store held three near-identical copies of the same base layers:Same content, three copies, differing only in gzip framing. Every run re-uploaded all three. And because a scope is all-or-nothing on the way down, a cache hit still cost a full download — which is why the log showed a few-kilobyte
COPYcorrectly marked CACHED yet taking 170.9s:The cache was functioning. It just cost more than it saved.
Changes
type=gha. Layers are content-addressed, so the base every worker variant shares is stored and fetched once.image-manifest=true,oci-mediatypes=trueis required — Forgejo rejects the default OCI-index cache form, the same reasonprovenance = falsewas already set.CACHE_WRITEis set on main only.nexus/no longer bakes the four worker images, and vice versa. main always bakes everything, so:latestand the shared cache are never built from a partial graph. Anything outsidenexus/andworker/falls back to baking everything.concurrency. The runner is a single LXC withcapacity: 1, so three pushes to a PR queued three full runs and the two obsolete ones blocked the one that mattered.nexus/Dockerfileinstalled with--frozen-lockfile=falsewhile the CI job used--frozen-lockfile, so the shipped image could carry dependency versions CI never linted, typechecked or tested.pnpm run builddeliberately stays in thenexusjob even though the webapp image build runs the identical command: per AGENTS.md fact #32 it is the only gate that validates SvelteKit route-module exports, and running it there fails the run before the bake starts rather than after it.Verification
Measured against the live Forgejo registry, on a fresh builder with no local cache — exactly what CI gets each run:
Also confirmed: Forgejo accepts the cache manifests; the shared base layer digest
sha256:470d9dd2…is written by both[base]and[playwright]into a single stored blob; and the target-selection logic was unit-tested over 9 cases including the fail-safe ones (empty diff and anexus-extra/decoy prefix both fall back todefault).All five
buildcache-*slots have been pre-warmed in the registry, so this run should land warm rather than cold.The playwright-version parity invariants (
nexus/src/lib/playwright-version-parity.test.ts) were re-checked by replicating each of its assertions against the edited files — all pass.Note
This PR is itself the first end-to-end test on the runner: every component was verified locally against the real registry, but the workflow has not yet executed in CI. It touches
docker-bake.hcland the workflow, so it will bake all five targets.🤖 Generated with Claude Code
The `images` job spent roughly 60% of its wall time moving cache around rather than building. Measured on run 945 (build step, 535s): cache export base 176.8s + playwright 146.5s + dind 115.6s + dind-playwright 65.7s + webapp 50.5s cache import 170.9s stalled on a step buildx had already marked CACHED Root cause: `type=gha` stores each scope as ONE OPAQUE TARBALL. All four worker targets are `FROM base`, but nothing can be shared between scopes, so the runner's cache store held three near-identical copies of the same base layers (453188176 / 453188080 / 453178159 bytes) and every run re-uploaded all three — and re-downloaded a whole 432 MB blob just to satisfy a cache HIT. Switch to registry-backed cache in the Forgejo registry. Layers are content-addressed there, so the shared base is stored and fetched once regardless of how many targets reference it. `image-manifest=true, oci-mediatypes=true` is required — Forgejo rejects the default OCI-index cache form, the same reason `provenance = false` was already set. Also: - PRs now READ the cache and never write it. A PR pays no export cost at all, and cannot poison the shared cache from a branch that may not merge. `CACHE_WRITE` is set on main only. - A PR that only touches nexus/ no longer bakes the four worker images, and vice versa. main still bakes everything, so :latest and the shared cache are never built from a partial graph. Anything outside nexus/ and worker/ falls back to baking everything. - Workflow-level `concurrency`, cancelling superseded PR runs. The runner is a single LXC with `capacity: 1`, so three pushes to a PR queued three full runs and the two obsolete ones blocked the one that mattered. main is never cancelled: its runs publish :latest and write the cache. - nexus/Dockerfile installed with `--frozen-lockfile=false` while the CI job used `--frozen-lockfile`, so the shipped image could carry dependency versions CI never linted, typechecked or tested. `pnpm run build` stays in the `nexus` job despite the webapp image build running the identical command: per AGENTS.md fact #32 it is the only gate that validates SvelteKit route-module exports, and running it there fails the run before the bake starts rather than after it. Verified against the live Forgejo registry (throwaway package, since deleted): cache export writes a manifest Forgejo accepts, a warm rerun is all-CACHED with no blob re-download, and the shared base layer digest is written by both [base] and [playwright] into a single stored blob.`cancel-in-progress: ${{ github.event_name == 'pull_request' }}` was there to exempt main from cancellation. Forgejo 16.0.5 does implement workflow concurrency (action_run carries concurrency_group / concurrency_type), but whether it evaluates an expression in this specific field is unverified here — and an expression it failed to parse would break the entire workflow, not just the exemption. The group is keyed on github.ref, so a run is only ever cancelled by a newer push to the same branch, which is the run you wanted anyway. For main that means two pushes inside one build window supersede each other; rare, and the later one republishes :latest and rewrites the cache regardless. Cheap insurance against a parse error that would cost far more.Closing unmerged. The registry-backed cache in this branch never demonstrated a win in CI:
imagesjobtype=ghaTwo things went wrong, and both are worth recording:
The cache was never usable by CI. It had been pre-warmed by hand from a workstation on buildx v0.33.0; the runner uses v0.30.1 from Alpine's
docker-cli-buildx. The keys did not line up, so every run rebuilt the sharedbasechain (npm install -g~150s) while a separately imported copy sat CACHED beside it. Local verification had used the same buildx that wrote the cache, so it hit every time and proved nothing.Run 952 hit a Forgejo bug. Two concurrent uploads of the same blob digest race; one wins, the other violates
UQE_package_blob_sha256and Forgejo dereferences a nil pointer inservices/packages/container/blob.go:98, surfacing to BuildKit aserror writing layer blob: unknown.bakepushes all targets concurrently and the worker images share most layers, so this is easy to trigger. Worth reporting upstream (Forgejo 16.0.5).The genuine findings are kept: the measured critical path is
base 189s -> playwright 395s -> dind-playwright 468s, the dind addon installs twice, and ~780s of that run was image/cache transfer over a NAT hairpin. Those motivate the composable-worker-image redesign in the issue opened alongside this, which removes thedind-playwrightbuild target entirely.Branch deleted; build-cache and
pr-182-*tags removed from the registry.Pull request closed