Image digests change in Python 3.9, image_test fails #2097
Description
Image digests change in Python 3.9, image_test fails
Affected Rule
container_image
Is this a regression?
No, this is an issue with Python 3.9 compatibility.
Description
In Python 3.9, container_image produces images with different digests due to this CPython change to the tarfile module. The change is described as a bug fix to bring tarfile into better alignment with other tar tools, by not writing devmajor and devminor for non-block file types.
The result is tar info header changes for directories, like this:
-00000140: 0000 0000 0000 0000 0030 3030 3030 3030 .........0000000
-00000150: 0030 3030 3030 3030 0000 0000 0000 0000 .0000000........
+00000140: 0000 0000 0000 0000 0000 0000 0000 0000 ................
+00000150: 0000 0000 0000 0000 0000 0000 0000 0000 ................
This of course generates differences in the diff_id and the image digest.
This breaks image_test which has hard-coded digests. Of course it may also cause some one-time thrashing in image identification and deduplication for anyone who updates their build system to Python 3.9 if they are expecting to have reproducible builds through that upgrade.
🔬 Minimal Reproduction
sudo apt install python3.9 # or however you install Python 3.9 on your system
sudo ln -sf /usr/bin/python3.9 /usr/bin/python # or however you set the default Python on your system
sudo ln -sf /usr/bin/python3.9 /usr/bin/python3 # or however you set the default Python on your system
bazel test tests/container:image_test # note this fails with errors about digests
Root causing that:
$ sudo ln -sf /usr/bin/python3.8 /usr/bin/python3
$ rm -f bazel-out/*/bin/testdata/base_with_entrypoint-layer.tar
$ bazel build testdata/base_with_entrypoint-layer.tar
$ cp bazel-out/*/bin/testdata/base_with_entrypoint-layer.tar /tmp/before.tar
$ sudo ln -sf /usr/bin/python3.9 /usr/bin/python3
$ rm -f bazel-out/*/bin/testdata/base_with_entrypoint-layer.tar
$ bazel build testdata/base_with_entrypoint-layer.tar
$ cp bazel-out/*/bin/testdata/base_with_entrypoint-layer.tar /tmp/after.tar
$ diff -u <(xxd /tmp/before.tar) <(xxd /tmp/after.tar)
--- /dev/fd/63 2022-05-28 15:10:43.602739290 -0400
+++ /dev/fd/62 2022-05-28 15:10:43.602739290 -0400
@@ -7,7 +7,7 @@
00000060: 0000 0000 3030 3030 3735 3500 3030 3030 ....0000755.0000
00000070: 3030 3000 3030 3030 3030 3000 3030 3030 000.0000000.0000
00000080: 3030 3030 3030 3000 3030 3030 3030 3030 0000000.00000000
-00000090: 3030 3000 3030 3733 3432 0020 3500 0000 000.007342. 5...
+00000090: 3030 3000 3030 3631 3032 0020 3500 0000 000.006102. 5...
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
@@ -18,8 +18,8 @@
00000110: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000120: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000130: 0000 0000 0000 0000 0000 0000 0000 0000 ................
-00000140: 0000 0000 0000 0000 0030 3030 3030 3030 .........0000000
-00000150: 0030 3030 3030 3030 0000 0000 0000 0000 .0000000........
+00000140: 0000 0000 0000 0000 0000 0000 0000 0000 ................
+00000150: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000160: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000170: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000180: 0000 0000 0000 0000 0000 0000 0000 0000 ................
...
The change starting at byte 0x096 is just a tar header checksum change and a consequence of the devmajor and devminor changes around byte 0x148.
🔥 Exception or Error
//tests/container:image_test fails because all tar digests have changed.
🌍 Your Environment
Operating System:
$ uname -a
Linux bpcreech9048451 5.16.18-1rodete2-amd64 #1 SMP PREEMPT Debian 5.16.18-1rodete2 (2022-04-07) x86_64 GNU/Linux
Output of bazel version:
$ bazel version
Build label: 4.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jan 21 07:33:24 2021 (1611214404)
Build timestamp: 1611214404
Build timestamp as int: 1611214404
Rules_docker version:
$ git rev-parse HEAD
cc315a12926b3c22099a43def524ad25235d6a76
Anything else relevant?