Docker saved tarballs cannot be used with linuxkit cache #4126
Description
I am trying to bump our linuxkit version from 0.8 to 1.6. This involves bringing the new cache system into our pipeline. This is fine.
Out pipeline currently does
- docker build ... $some_container
- docker save $some_container
- linuxkit cache import $some_container
- linuxkit build $our_vm
I note that the build fails because images cannot be pulled from dockerhub: could not pull image $image:latest: error getting manifest for image docker.io/$image:latest: GET https://index.docker.io/v2/$image/manifests/latest: MANIFEST_UNKNOWN: manifest unknown; unknown tag=latest
This is unsurprising, as we do not push to dockerhub in our pipeline. What is surprising is linuxkit build is reaching out to dockerhub when the image is in cache:
❯ linuxkit --cache=/tmp/nix-shell.IruLEu/bazel_vm_linuxkit_cache.hud1FJ cache ls
image name root manifest hash
...
docker.io/$image:latest sha256:$a_sha
...
I note some trawling through -v 2 in the build nets me:
time="2025-06-24T12:48:13+01:00" level=debug msg="Image docker.io/$image:latest platform linux/amd64 incomplete or invalid in local cache, error invalid image, validating layers: gzip: invalid header, pulling"
After some searching through codebases, my current best guess is this is because docker save outputs blobs uncompressed but linuxkit assumes gzip?:
❯ tar -xf $image_tarball.tar.gz blobs/sha256/$a_sha manifest.json
❯ cat manifest.json | jq .
...
"sha256:$a_sha": {
"mediaType": "application/vnd.oci.image.layer.v1.tar",
"size": 18848768,
"digest": "sha256:$a_sha"
},
...
❯ gunzip -t blobs/sha256/$a_sha
gzip: blobs/sha256/$a_sha: not in gzip format
❯ file blobs/sha256/$a_sha
blobs/sha256/$a_sha: POSIX tar archive
❯ tar -tvf blobs/sha256/$a_sha
drwxr-xr-x 0/0 0 2024-06-13 20:47 bin/
lrwxrwxrwx 0/0 0 2024-06-13 20:47 bin/arch -> /bin/busybox
lrwxrwxrwx 0/0 0 2024-06-13 20:47 bin/ash -> /bin/busybox
...
our containers are created in docker with something equivalent to
cat $context_tarball | docker build --tag $a_tag
docker save --output $a_file $a_tag:latest
and imported+built with:
linuxkit --cache /tmp/some_location/ cache import $a_file
linuxkit --cache /tmp/some_location build --format iso-bios --dir $a_dir --no-sbom --name $a_name $a_yaml
would love some more info.
(edit: had to remove some deets).