[rawhide+podman-next] podman push/pull registry roundtrip changes the image #20611
Description
Issue Description
This is about the regression that started in PR #20595 and since then has failed the cockpit-podman rawhide test on every PR (example). I don't see a clean pattern to this yet. #20595 fails all tests (yours and ours) on all OSes, and yet that schema bump seems harmless; other PRs succeed your tests, but still fail ours in the same way, but only on rawhide; and your own tests pass in other PRs (mostly). It is related to the containers-common update from 1-97 to 1-98 (-99 fails as well), as downgrading that package makes it work again.
But something doesn't add up. Perhaps the podman-next COPR gets builds not only from main, but from some PRs, or the PRs do builds without rebasing, or don't build against the latest podman-next, or that containers-common has some indirect effect which I don't understand.
The failing test checks image uploading and downloading to/from a registry. Until yesterday, that ended up as the same image, but now it's a different one.
Steps to reproduce the issue
This is a CLI version of the relevant part of the test:
# update to podman-next:
sudo dnf -y copr enable rhcontainerbot/podman-next >&2; dnf -y update --repo 'copr*'
# run local registry
podman run -d -p 5000:5000 --name registry quay.io/libpod/registry:2.8
# take some container image, note its SHA
podman pull docker.io/busybox
# → docker.io/library/busybox latest a416a98b71e2 3 months ago 4.5 MB
# upload it to the registry
podman tag docker.io/library/busybox:latest localhost:5000/my-busybox
podman push localhost:5000/my-busybox
podman rm localhost:5000/my-busybox
# download it again
podman pull localhost:5000/my-busybox
# compare SHAs
podman images | grep busybox
Describe the results you received
With podman-next:
podman-4.8.0~dev-1.20231106154052317574.main.2390.886f932b0.fc40.x86_64
containers-common-1-99.fc40.noarch
the downloaded image is different from the original:
docker.io/library/busybox latest a416a98b71e2 3 months ago 4.5 MB
localhost:5000/my-busybox latest 5ed23df91f27 3 months ago 4.49 MB
Describe the results you expected
With current rawhide:
podman-4.7.0-1.fc40.x86_64
containers-common-1-97.fc40.noarch
the downloaded image is identical to the original docker.io one:
localhost:5000/my-busybox latest a416a98b71e2 3 months ago 4.5 MB
docker.io/library/busybox latest a416a98b71e2 3 months ago 4.5 MB
podman info output
host:
arch: amd64
buildahVersion: 1.33.0-dev
cgroupControllers:
- cpu
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.7-3.fc39.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.7, commit: '
cpuUtilization:
idlePercent: 97.43
systemPercent: 1.22
userPercent: 1.34
cpus: 1
databaseBackend: boltdb
distribution:
distribution: fedora
variant: cloud
version: "40"
eventLogger: journald
freeLocks: 2047
hostname: fedora-rawhide-127-0-0-2-2201
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 524288
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 524288
size: 65536
kernel: 6.7.0-0.rc0.20231031git5a6a09e97199.2.fc40.x86_64
linkmode: dynamic
logDriver: journald
memFree: 405979136
memTotal: 1135865856
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.8.0-1.20231103152128612668.main.26.g0b97b25.x86_64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.9.0-dev
package: netavark-1.8.0-1.20231103122905869245.main.24.gb7e144d.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.9.0-dev
ociRuntime:
name: crun
package: crun-1.11.1-1.20231106135232645587.main.7.g2e35a99.fc40.x86_64
path: /usr/bin/crun
version: |-
crun version UNKNOWN
commit: 3af84aa5c314ce41d579d2cfa0a0ccc0059ca8aa
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-0^20231004.gf851084-1.fc40.x86_64
version: |
pasta 0^20231004.gf851084-1.fc40.x86_64
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: true
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.2-1.fc40.x86_64
version: |-
slirp4netns version 1.2.2
commit: 0ee2d87523e906518d34a6b423271e4826f71faf
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.3
swapFree: 1131147264
swapTotal: 1135603712
uptime: 1h 19m 59.00s (Approximately 0.04 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
localhost:5000:
Blocked: false
Insecure: true
Location: localhost:5000
MirrorByDigestOnly: false
Mirrors: []
Prefix: localhost:5000
PullFromMirror: ""
localhost:6000:
Blocked: false
Insecure: true
Location: localhost:6000
MirrorByDigestOnly: false
Mirrors: []
Prefix: localhost:6000
PullFromMirror: ""
search:
- localhost:5000
- localhost:6000
store:
configFile: /home/admin/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/admin/.local/share/containers/storage
graphRootAllocated: 12798898176
graphRootUsed: 2198786048
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 5
runRoot: /tmp/containers-user-1000/containers
transientStore: false
volumePath: /home/admin/.local/share/containers/storage/volumes
version:
APIVersion: 4.8.0-dev-886f932b0
Built: 1699285434
BuiltTime: Mon Nov 6 15:43:54 2023
GitCommit: ""
GoVersion: go1.21.3
Os: linux
OsArch: linux/amd64
Version: 4.8.0-dev-886f932b0Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
Standard Fedora rawhide cloud image
Additional information
Always happens. Running sudo dnf downgrade containers-common twice to downgrade to 4:1-97.fc40 goes back to the previous working state.