Repository navigation
Is the chisel cache working as intended? #254
Description
It seems to be functional when fetching packages, but I'm not sure why indexes are always fetched? Is there any information on how to prevent that?
- The index update seems to be populating the
~/.cache/sha256/directory with the same content redundantly (or at least updating timestamps). Non-issue in aDockerfileif using a cache mount. - The procesing of a release seems to take quite a bit of time, could that be cached as well? (especially given the example just repeats the same command in this case)
Context
When building a variety of images and using chisel sharing a cache makes little difference at minimizing install time (12 images with a single chisel command each built concurrently took in 210s with a shared cache mount forcing them to build sequentially, 234s with cache id for each platform resulting in 3 images built concurrently on a 1 vCPU 2GB build host VPS with NVMe storage).
Ideally the overhead noted below could be minimized?
$ chisel cut --release ubuntu-24.04 --arch "amd64" --root /root-fs base-files_base base-files_chisel
2025/11/09 10:37:25 Consulting release repository...
2025/11/09 10:37:26 Fetching current ubuntu-24.04 release...
2025/11/09 10:37:26 Processing ubuntu-24.04 release...
2025/11/09 10:37:39 Selecting slices...
2025/11/09 10:37:39 Fetching ubuntu 24.04 noble suite details...
2025/11/09 10:37:40 Release date: Thu, 25 Apr 2024 15:10:33 UTC
2025/11/09 10:37:40 Fetching index for ubuntu 24.04 noble main component...
2025/11/09 10:37:42 Fetching index for ubuntu 24.04 noble universe component...
2025/11/09 10:37:44 Fetching ubuntu 24.04 noble-security suite details...
2025/11/09 10:37:44 Release date: Sun, 09 Nov 2025 8:02:04 UTC
2025/11/09 10:37:44 Fetching index for ubuntu 24.04 noble-security main component...
2025/11/09 10:37:45 Fetching index for ubuntu 24.04 noble-security universe component...
2025/11/09 10:37:45 Fetching ubuntu 24.04 noble-updates suite details...
2025/11/09 10:37:45 Release date: Sun, 09 Nov 2025 8:03:32 UTC
2025/11/09 10:37:45 Fetching index for ubuntu 24.04 noble-updates main component...
2025/11/09 10:37:45 Fetching index for ubuntu 24.04 noble-updates universe component...
2025/11/09 10:37:46 Cannot open credentials directory "/etc/apt/auth.conf.d": open /etc/apt/auth.conf.d: no such file or directory
2025/11/09 10:37:46 Archive "ubuntu-esm-infra" ignored: credentials not found
2025/11/09 10:37:46 Cannot open credentials directory "/etc/apt/auth.conf.d": open /etc/apt/auth.conf.d: no such file or directory
2025/11/09 10:37:46 Archive "ubuntu-esm-apps" ignored: credentials not found
2025/11/09 10:37:46 Fetching pool/main/b/base-files/base-files_13ubuntu10.3_amd64.deb...
2025/11/09 10:37:46 Extracting files from package "base-files"...
2025/11/09 10:37:46 Generating manifest at /var/lib/chisel/manifest.wall...$ ls -l /root/.cache/chisel/sha256/
-rw-r--r-- 1 root root 7322803 Nov 9 10:37 02b719e9a7f630e8268ee7a6b36dc284b2b3b08337cdc66557e3a98416ca5616
-rw------- 1 root root 126127 Nov 9 10:37 351cae55448e86a6618fc3a5ed815901cc56ff8b0cdd059010425ea9de5725d7
-rw-r--r-- 1 root root 8828159 Nov 9 10:37 431e48d02016f2877683433bedc215a5fb0c94a5478f88c02122605f492d7b33
-rw-r--r-- 1 root root 73379142 Nov 9 10:37 687ffc969e7700137677b9cfe2f614ee2d638052a91a7c9bf31f3f6a3040a99a
-rw-r--r-- 1 root root 5651691 Nov 9 10:37 6c2f1b4fbdb8c6d0bb9b6af6a6ed4f65e8605bfa28e1a89fe63ebca2eaab4d1b
-rw-r--r-- 1 root root 7165069 Nov 9 10:37 8f6f71ae839c8cba390a7643fcbbdacddb0bc7d12c1583a2dd80a1f8443a30e5
-rw-rw-rw- 1 root root 73218 Nov 9 10:37 9bdab117436877d18211cf43811df481e138a2e9ae8571cd25e7fa46b57b3757
-rw------- 1 root root 255850 Nov 9 10:37 cdb2f31d809f589719a53c6ad15f255b27569c4059542ada282aaa21b8e164b0
-rw------- 1 root root 126125 Nov 9 10:37 e26cfafcd2f5e4cd864dab116d919c3005e03e26e0e3adeb9b587f88b44fe4a2
-rw-r--r-- 1 root root 9454576 Nov 9 10:37 face3cae09e1fff661ca2a574a0c128cb2507ad1700e9b43426fbcc86217a4edRepeating that exact same chisel cut invocation, output is mostly the same:
$ chisel cut --release ubuntu-24.04 --arch "amd64" --root /root-fs base-files_base base-files_chisel
2025/11/09 10:38:14 Consulting release repository...
2025/11/09 10:38:14 Cached ubuntu-24.04 release is still up-to-date.
2025/11/09 10:38:14 Processing ubuntu-24.04 release...
2025/11/09 10:38:28 Selecting slices...
2025/11/09 10:38:28 Fetching ubuntu 24.04 noble suite details...
2025/11/09 10:38:30 Release date: Thu, 25 Apr 2024 15:10:33 UTC
2025/11/09 10:38:30 Fetching index for ubuntu 24.04 noble main component...
2025/11/09 10:38:30 Fetching index for ubuntu 24.04 noble universe component...
2025/11/09 10:38:30 Fetching ubuntu 24.04 noble-security suite details...
2025/11/09 10:38:30 Release date: Sun, 09 Nov 2025 8:02:04 UTC
2025/11/09 10:38:30 Fetching index for ubuntu 24.04 noble-security main component...
2025/11/09 10:38:30 Fetching index for ubuntu 24.04 noble-security universe component...
2025/11/09 10:38:30 Fetching ubuntu 24.04 noble-updates suite details...
2025/11/09 10:38:31 Release date: Sun, 09 Nov 2025 10:07:17 UTC
2025/11/09 10:38:31 Fetching index for ubuntu 24.04 noble-updates main component...
2025/11/09 10:38:31 Fetching index for ubuntu 24.04 noble-updates universe component...
2025/11/09 10:38:31 Cannot open credentials directory "/etc/apt/auth.conf.d": open /etc/apt/auth.conf.d: no such file or directory
2025/11/09 10:38:31 Archive "ubuntu-esm-apps" ignored: credentials not found
2025/11/09 10:38:31 Cannot open credentials directory "/etc/apt/auth.conf.d": open /etc/apt/auth.conf.d: no such file or directory
2025/11/09 10:38:31 Archive "ubuntu-esm-infra" ignored: credentials not found
2025/11/09 10:38:31 Fetching pool/main/b/base-files/base-files_13ubuntu10.3_amd64.deb...
2025/11/09 10:38:31 Extracting files from package "base-files"...
2025/11/09 10:38:31 Generating manifest at /var/lib/chisel/manifest.wall...Note that the timestamp of many cache items was for some reason updated (in docker this adds excess weight if not using a cache mount), perhaps the cache was repopulated redundantly rather than used?:
$ ls -l /root/.cache/chisel/sha256/
-rw-r--r-- 1 root root 7322803 Nov 9 10:38 02b719e9a7f630e8268ee7a6b36dc284b2b3b08337cdc66557e3a98416ca5616
-rw------- 1 root root 126127 Nov 9 10:38 351cae55448e86a6618fc3a5ed815901cc56ff8b0cdd059010425ea9de5725d7
-rw-r--r-- 1 root root 8828159 Nov 9 10:38 431e48d02016f2877683433bedc215a5fb0c94a5478f88c02122605f492d7b33
-rw-r--r-- 1 root root 73379142 Nov 9 10:38 687ffc969e7700137677b9cfe2f614ee2d638052a91a7c9bf31f3f6a3040a99a
-rw-r--r-- 1 root root 5651691 Nov 9 10:38 6c2f1b4fbdb8c6d0bb9b6af6a6ed4f65e8605bfa28e1a89fe63ebca2eaab4d1b
-rw------- 1 root root 126125 Nov 9 10:38 774609b2b3bff3be8ea048247e98d35508daf42a1e9333939a7d4ec16bc779bd
-rw-r--r-- 1 root root 7165069 Nov 9 10:38 8f6f71ae839c8cba390a7643fcbbdacddb0bc7d12c1583a2dd80a1f8443a30e5
-rw-rw-rw- 1 root root 73218 Nov 9 10:38 9bdab117436877d18211cf43811df481e138a2e9ae8571cd25e7fa46b57b3757
-rw------- 1 root root 255850 Nov 9 10:38 cdb2f31d809f589719a53c6ad15f255b27569c4059542ada282aaa21b8e164b0
-rw------- 1 root root 126125 Nov 9 10:37 e26cfafcd2f5e4cd864dab116d919c3005e03e26e0e3adeb9b587f88b44fe4a2
-rw-r--r-- 1 root root 9454576 Nov 9 10:38 face3cae09e1fff661ca2a574a0c128cb2507ad1700e9b43426fbcc86217a4edWith a larger list of slices to install, I've observed similar time where a bulk of the command is spent regardless of the cache being present, a good chunk of time is used here (averaging 15s):
2025/11/09 10:37:26 Processing ubuntu-24.04 release...
2025/11/09 10:37:39 Selecting slices...
Fetching indexes uses up to 7s which presumably shouldn't be necessary on a 2nd run? Is this where caches are being updated? I noticed one of those blobs is approx 80MB, this is similar to apt-get update? Could it be opt-out, given the cache exists and was fairly recent since it's last update/refresh?
Fetching packages then occurs and I've seen that at approx 8s for the equivalent to Google Distroless cc image deps. When doing the cached run this stage of the command is very quick (perhaps the logs should indicate these steps aren't fetching but are already cached, similar to how docker build does with cached layers getting a [CACHED] prefix?).
Example reproduction
Images will build per platform sequentially, ideally after the cache is populated the subsequent images of each target platform arch would not need to fully process the release and fetch indexes again? (+15-20s per image)
# Build all variants for the three platform targets (4x3 = 12 images total):
docker buildx bake chisel --set 'chisel*.platform=linux/amd64,linux/arm64,linux/riscv64'
# Alternatively without `docker-bake.hcl`, just build the `root-fs` stage for a
# container to shell into with chisel ready to go:
docker build --tag localhost/chisel --target root-fs .Dockerfile:
FROM --platform=${BUILDPLATFORM} alpine AS get-chisel
ARG CHISEL_VERSION=1.3.0
ARG BUILDARCH
ARG CHISEL_RELEASE="https://github.com/canonical/chisel/releases/download/v${CHISEL_VERSION}/chisel_v${CHISEL_VERSION}_linux_${BUILDARCH}.tar.gz"
# NOTE: `--no-same-owner` used as `chisel` release has ownership of `1001:128`
RUN wget -qO - "${CHISEL_RELEASE}" | tar -xz --no-same-owner -C /usr/local/bin chisel
FROM get-chisel AS root-fs
WORKDIR /root-fs
ARG TARGETARCH
ARG SLICES="base-files_base bash_bins"
# Cache mount must be locked similar to the cache for `apt`,
# this limits parallelism but should work effectively here by using an id constrained to the target arch:
RUN --mount=type=cache,target=/root/.cache/chisel,id="chisel-cache-${TARGETARCH}",sharing=locked \
chisel cut --release ubuntu-24.04 --arch "${TARGETARCH}" --root /root-fs ${SLICES} base-files_chisel
FROM scratch
COPY --link --from=root-fs /root-fs /This docker-bake.hcl file defines equivalent variants of Google Distroless images, where the matrix config of the chisel target will configure 4 target images to build (multiplied by the platforms set by the CLI):
docker-bake.hcl:
variable "SLICES_STATIC" {
default = "base-files_base base-files_release-info netbase_config tzdata_zoneinfo base-passwd_data libc-bin_nsswitch ca-certificates_data"
}
variable "SLICES_BASE_NOSSL" {
default = "${SLICES_STATIC} libc6_libs libc6_gconv libc6_config"
}
variable "SLICES_BASE" {
default = "${SLICES_BASE_NOSSL} libssl3t64_libs"
}
variable "SLICES_CC" {
default = "${SLICES_BASE} libgomp1_libs libstdc++6_libs libgcc-s1_libs bash_bins"
}
target "chisel" {
context = "."
dockerfile = "Dockerfile"
name = "chisel_${item.variant}"
matrix = {
item = [
{
variant = "static"
slices = "${SLICES_STATIC}"
},
{
variant = "base-nossl"
slices = "${SLICES_BASE_NOSSL}"
},
{
variant = "base"
slices = "${SLICES_BASE}"
},
{
variant = "cc"
slices = "${SLICES_CC}"
}
]
}
args = {
SLICES = item.slices
}
tags = ["localhost/chisel-${item.variant}:latest"]
}I'd have the variants extend each other, but chisel cut still doesn't support using the manifest to minimize it's work (the mentioned cache content is modified, at least by timestamps and the var/lib/chisel/manifest.wall file is overwritten rather than merged/updated), so it seems inefficient when slices get reinstalled.
Related issues regarding optimized writes:
Activity
letFunny commentedon Nov 11, 2025
Hey @polarathene, thank you for the detailed comment. You are totally right on most of the things you mentioned. I will try to address them one by one.
About validating the release taking a long time, this is in fact the most expensive part of every chisel command. In the past we spent some time trying to optimize it to get better performance (see pr) but unfortunately, as the release grows, the complexity will grow as well inevitably as we need to check every path against any other path. There are multiple options here for us, we could try to optimize the speed for parsing the release or we can try to "cache" the validation. The former is very hard given that we need to do some finer performance work and possibly rewrite the algorithm and make the code much more complex. The latter seems simpler as we seemingly only need to create another field in the cache to store the result and somehow identify the chisel release and the archive point in time.
The expectation here is that your cache will not be present in the final image anyways so updating the time to when it was last used should not be a problem. Do you have a use-case in mind when this is not the case?
This feature is coming in the next 6 months. It will take some time to make sure we are not breaking any promises regarding determinism and consistency but it will be available soon (see also the comments on #208).
Please let me know if I missed anything.
CC: @cjdcordeiro @upils
letFunny commentedon Nov 11, 2025
Let's keep the conversation about validation in #253 to centralize the discussion. The issue at hand is to discuss whether we need to do something about the cache.
polarathene commentedon Nov 11, 2025
Not really, I think initially I wasn't aware of the cache due to the multi-stage usage, but when troubleshooting and building the
root-fsstage I have to runchiselwith I had noticed it had some extra weight and I found it had been populating a cache which I thought I might have been able to leverage to speed up subsequent runs with a cache mount, however it didn't reduce much of the overhead.It doesn't seem like the cache is documented in the CLI? (or docs) There was this dismissed PR to add a custom
--cache-dirlocation, although as can be seen in the source the standardXDG_CACHE_HOMEENV is checked and used by default when present (where there only seems to be an issue of mine showing that in use with a minimalscratchbase image that bind image mountschiselin aRUNinstruction).So maybe that's worth having a bit more visibility/discovery for, I'm not sure. I likely discovered it via
divefor an image layer diff inspection, and then looking through source code to see howchiselwas configuring it's cache dir location to see if I could adjust it.My observation of the mtime changing I think was more focused on the fact that I didn't expect cached entries that are used to have their mtime adjusted (that said be careful with relying on mtime for cache purposes, since I know
cargodoes that with source input which makes restoring cache less useful in CI as a git checkout doesn't have mtime preserved).Since the mtime changed, if nothing was modified by
chisel, I assumed what was happening waschiselnaively replaced the cache content, possibly from redundantly sourcing the remote source again so soon after the previous operation.So to clarify, this portion is entirely validation?:
And can you confirm that the fetching of indexes is what's in the cache? Is it effectively like
apt-get update? Is it writing over 100MB each time?I've not tested if part of the overhead for indexes is from network or disk I/O speed, but one would think that refreshing the index again so soon would be redundant? 🤔 (in images I build without
chisel, I know I'd need to refresh the package index should that expectation change due to adding a third-party repo for example)letFunny commentedon Nov 13, 2025
I created an issue and we will add it to the docs, thanks for the suggestion.
This is exactly what happens in the code.
Yes indeed, reading the release and running all the validation with conflicts, essentials, etc.
We only fetch the InRelease file from the archive (example) and read from the cache unless there is a newer version from the package upstream.
It is more about simplifying the model and reducing the amount of failures. There are many questions like, what if the filesystem does not update the mtime? What do we consider to be old enough? It is simpler for Chisel to download the file and verify the signature each time to establish the "trust" in the chain if that makes sense.
polarathene commentedon Nov 13, 2025
While I can understand that reasoning I think it should be possible to opt-out?
How likely is chisel to fail in a scenario like I've provided above via the
docker-bake.hcl? Building for 3 different platforms with 4 image variants (that would ideally be sequentially layered by extending the earlier variants produced, butchiselisn't smart enough yet to avoid redundant writes for already installed slices).That takes 3-4 minutes on a system where the bulk of the time is
chiselfetching indexes and performing validation that ideally once the cache is populated should be viable to leverage by opt-out of those steps?While I do lean more towards agreement with you of playing it safe and ensuring correctness/reliability, I think there are valid scenarios where the current overhead is a bit undesirable, especially when I'm iterating on images/builds where the chisel layer gets invalidated. Before I was aware of how to restrict the parallelism of the Docker image builds, those 12 images on the system did slow it down to 900s (even though QEMU was not in use, just too much I/O)
How is that being used? I see an
Expire()method, but that seems to only be used by a test presently with an input of an hour?Normally package managers have a separate arg or flag to force refreshing a package index, and so long as inputs have been validated and cached recently, that should be something that can be minimized too where appropriate?
I assume the
Expire()method would be good for GC? Docker or BuildKit IIRC has some GC config that might be of interest forchiselas I recall they support pruning by age or size boundaries. Cache can be useful for the frequency of CI runs but obviously I agree there is risk if cache was used more aggressively but held back updating packages (unless there was something akin to a lock file, but IIRCchiseldoesn't plan on supporting pinning of packages for reproducible builds).letFunny commentedon Nov 14, 2025
The majority of the time is spent on release validation not on fetching the latest InRelease file which is why I was suggesting to speed up release validation itself.
About opting out, in general we try to refrain from exposing those options. Our promise is that if you build a Chisel image you will get a secure and small container. If it were possible to bypass validation or getting the most up to date release, then users will rightfully complain when the images they create do not work (because no conflict validation) or they are missing security patches and are vulnerable to CVEs.
However, I do believe that we should speed up release validation and that is going to have the most impact on your use-case.
letFunny commentedon Nov 17, 2025
Closing this issue to centralize the discussion in #253 and canonical/chisel-docs#27.