document Chisel's cache and how to configure its location #27

Activity

1. polarathene commented on Nov 13, 2025

   

   polarathene

   on Nov 13, 2025

   Contributor

   More actions

   TL;DR: Questions users may want to know

   • Is the cache able to handle multiple archs (either via --arch, or using a chisel binary from another arch) or does it replace any prior cache when arch differs?
   • How compatible is chisel with concurrent access to the cache? (especially when this occurs in container builds, which prevents some detection mechanisms depending on context)

   ---

   It would also be helpful to document any concerns to be aware of when using the cache in concurrent builds (see Dockerfile + docker-bake.hcl here).

   Notably here:

   # Cache mount must be locked similar to the cache for `apt`,
   # this limits parallelism but should work effectively here by using an id constrained to the target arch:
   RUN --mount=type=cache,target=/root/.cache/chisel,id="chisel-cache-${TARGETARCH}",sharing=locked \
    chisel cut --release ubuntu-24.04 --arch "${TARGETARCH}" --root /root-fs ${SLICES} base-files_chisel

   When using a cache mount, if it is shared between any other active builders/stages running chisel behaves like apt and will start to fail IIRC unless using sharing=locked (which IIRC prevents any other build instances proceeding with a RUN instruction when the same cache id mount is referenced, until the existing one is done to remove the lock).

   I assume the Chisel cache is otherwise shareable across projects, but I'm not sure about if it's cache is multi-arch compatible? Does the arch chosen replace cache for any previous arch used? Or does it differ due to assumptions if the chisel binary is itself compiled for a different arch (emulated builds via QEMU)?

   In my example I've configured the cache ID to differ by what will be used for chisel cut --arch ..., even if Chisel could share a cache across different archs, it doesn't appear capable of concurrent I/O (which for example cargo can do via it's cache lock files to communicate to other cargo processes), so for not blocking chisel cut on concurrent builds it would either require not using cache mounts (wasted build layer cache) or ensuring each permutation of a build has a unique cache ID (and thus using extra storage of 100MB+ each).

   I assume my example cache ID and using sharing=locked is the most appropriate tradeoff, but it would be good to have clarification in docs about concurrent + multi-arch sharing concerns.

   ---

   For reference, here is an example of Cargo which was a bit more problematic due to layout (interleaved cache lock files in it's home directory):

   Collapsed for brevity (click to view)

   # ...
   
   # Mapping ARM64 / AMD64 naming conventions (build target specific):
   FROM builder-base AS builder-amd64
   ENV DOCKERFILE_TARGET_ARCH=x86_64
   FROM builder-base AS builder-arm64
   ENV DOCKERFILE_TARGET_ARCH=aarch64
   
   FROM builder-${TARGETARCH} AS builder
   ENV CARGO_HOME=/build/cache
   ENV CARGO_TARGET_DIR=/build/target
   ENV CARGO_BUILD_TARGET=${DOCKERFILE_TARGET_ARCH}-unknown-linux-musl
   ARG APP_NAME=example
   ARG TARGETARCH
   
   RUN \
     # Use bind mounts to access Cargo config, lock, and sources; without needing to copy them into a build layer
     --mount=type=bind,source=src,target=src \
     --mount=type=bind,source=Cargo.toml,target=Cargo.toml \
     --mount=type=bind,source=Cargo.lock,target=Cargo.lock \
     # Add cache mounts to speed up builds (also avoids bloating the docker build layer cache):
     --mount=type=cache,target="/root/.cache/zig",id="zig-cache" \
     # Cache mounts (dirs for crates cache + build target):
     # https://doc.rust-lang.org/cargo/guide/cargo-home.html#caching-the-cargo-home-in-ci
     # CAUTION: As cargo uses multiple lock files (eg: `${CARGO_HOME}/{.global-cache,.package-cache,.package-cache-mutate}`), do not mount subdirs individually.
     --mount=type=cache,target="${CARGO_HOME}",id="cargo-cache" \
     # This cache mount is specific enough that you may not have any concurrent builds needing to share it, communicate that expectation explicitly:
     --mount=type=cache,target="/build/target/",id="cargo-target-${TARGETARCH}-${APP_NAME}",sharing=locked \
     # These are redundant as they're easily reconstructed from cache above. Use TMPFS mounts to exclude from cache mounts:
     # TMPFS mount are a better choice than `rm -rf` command (which is risky on a cache mount that is shared across concurrent builds).
     --mount=type=tmpfs,target="${CARGO_HOME}/registry/src" \
     --mount=type=tmpfs,target="${CARGO_HOME}/git/checkouts" \
     <<HEREDOC
   
       cargo zigbuild --locked --release --target "${CARGO_BUILD_TARGET}" --bin "${APP_NAME}"
       mv "${CARGO_TARGET_DIR}/${CARGO_BUILD_TARGET}/release/${APP_NAME}" "/build/app-${TARGETARCH}"
   
   HEREDOC
   
   # ...

   You can see that:

   • Cargo has .global-cache, .package-cache and .package-cache-mutate lock files in it's top-level home directory, this prevents requiring sharing=locked there.
   • sharing=locked was added to the /build/target/ cache mount that followed, only to communicate that it should not be accidentally shared concurrently (highly unlikely to occur even if other projects were building at the same time due to the more specific APP_NAME suffix in the cache ID key).
   • tmpfs mounts are optionally added as persisting that to disk via the cache mount is mostly wasteful.
2. asanvaq

   added

   Documentation

   on Dec 3, 2025
3. asanvaq

   removed

   Documentation

   on Sep 8, 2026