Repository navigation
Concurrent chisels race on <digest>.tmp #340
Description
two chisel processes that miss the cache on the same blob at the same time write to the same temporary file, and one of them fails. the other may publish a torn file.
this happens since os.Create(c.filePath(digestKind, digest+".tmp")) in cache.go is a per-digest fixed name with no locks. with processes A and B fetching the same deb (or the same Packages index) into a cold cache:
- A creates
<digest>.tmpand starts writing. - B's
os.Createtruncates the same inode to zero. A keeps writing at its own offset, so the file has a hole of zeros at the front until B fills it. - whoever closes first hashes its own stream (correct), renames the inode to
<digest>and opens it. the other process may still be filling the hole, so the reader can see zeros where deb bytes should be;Cache.Opendoes not re-verify. - the other process finishes and its rename of
<digest>.tmpfails withno such file or directory, so the cut dies withcannot fetch from archive: rename ....
a variant: if B's download fails half way, fail removes <digest>.tmp from under A, and A's close fails the same way.
i've come across this when doing chisel-releases testing and cutting every slice of a release on every architecture. in order to make the cuts faster, arch: all debs and the per-suite indexes were shared by every worker and that's where i started getting races.
repro
two cuts at once on a cold shared cache, 20 times:
#!/usr/bin/env bash
set -e
work=$(mktemp -d)
git clone -q --depth 1 -b ubuntu-24.04 https://github.com/canonical/chisel-releases "$work/chisel-releases"
for run in $(seq 20); do
rm -rf "$work/cache" "$work"/root-* && mkdir -p "$work/root-1" "$work/root-2"
for i in 1 2; do
XDG_CACHE_HOME="$work/cache" chisel cut \
--release "$work/chisel-releases" \
--root "$work/root-$i" \
--arch amd64 base-files_base > "$work/out-$i" 2>&1 \
|| echo "run $run job $i: $(grep -m1 '^error:' "$work/out-$i")" &
done
wait # wait for both cuts
for f in "$work"/cache/chisel/sha256/[0-9a-f]*; do
[ "$(sha256sum "$f" | cut -d' ' -f1)" = "$(basename "$f")" ] \
|| echo "run $run: torn blob $(basename "$f")"
done
doneobv a bit random but in the run i've done just now 13 of the 20 runs lost one of the two cuts, every time on a Packages index (these the biggest blobs, so the race window is longest):
run 1 job 1: error: cannot fetch from archive: rename .../8f6f71ae...a30e5.tmp .../8f6f71ae...a30e5: no such file or directory
...
proposal
file, err = os.CreateTemp(c.filePath(digestKind, ""), digest+".tmp.*")or a per-digest fslock which is what fetch does.
Activity
upils commentedon Oct 22, 2026
Chisel works under the assumption a single instance is running at a time, and so the cache is never concurrently managed. Changing this assumption exposes us to other issues I think: for example we are not yet actually expiring entries in the cache, but the day we do that it may break this concurrent use case.
I think I would rather see if this is something we really want/need to properly support and then sit down and determine how to properly do it. I do not want to make a half-backed promise.
For your use case: what about "warming up" the cache first with a call downloading most of the packages you need, and then use the directory as a base layer with something like fuse-overlayfs?
Edit: Considering what I explained above I think what you request would be a feature more than a bug so I am changing labels accordingly.
[-]bug: concurrent chisels race on <digest>.tmp[/-][+]bug: concurrent chisels race on `<digest>.tmp`[/+]lczyk commentedon Oct 23, 2026
i think we should drop that assumption, for chisel-releases ci as much as for the ever-increasing use of chisel in builds (e.g. #254 is people already running cuts in parallel against one cache mount). also, the single instance assumption does not really match with the code. fetch already fslocks releases. its just the blob cache which is missing either locks or atomics or
<insert-your-favourite-syncronisation-primitive>.yep. tracked in #261. i would fix the expiry first.
yeah, thats v close to what i ended up doing.
[-]bug: concurrent chisels race on `<digest>.tmp`[/-][+]Concurrent chisels race on `<digest>.tmp`[/+]feat: parallel cuts with private views of chisel's cache
upils commentedon Oct 28, 2026
What the code does (or doesn't do) is different than what the project promises to do. For now running 2 Chisel instances at the same time is undefined behavior, exactly as targeting the same rootfs twice. Maybe things will go well, maybe not :). Having users doing wild things does not mean we must support these things.
Promising that multiple instances of Chisel can safely run concurrently might be solved "easily" today but it means this is one more promise we must ensure we keep when adding new features in the future.
lczyk commentedon Nov 3, 2026
i'm not suggesting for chisel to promise anything. i think there is a middle ground between 'promise as part of official api' and 'undefined behaviour'. we could make this use case work better for our own internal' sake but just not promise anything as part of the public api. this way if we wanna change things we're not barring ourselves from changing how things work, but have a cleaner implementation nonetheless.
i disagree that this is a 'wild thing'. also, a degree of wildness of a use case has absolutely nothing to do with the validity of the issue at hand -- if someone reports a segfault which prohibits them from using chisel to peal potatoes in space, it's still a segfault.