Skip to content

podman takes around 20 seconds to after the executions of containers completed or interrupted by CTRL + C #10

Open
Open
podman takes around 20 seconds to after the executions of containers completed or interrupted by CTRL + C#10
@EIC-JayGanatra

Description

@EIC-JayGanatra
EIC-JayGanatra
opened on Apr 8, 2024

Issue Description

I encountered a situation that required an upgrade of Podman from version 4.1, which comes with Kirkstone by default, to version 4.4.4 in order to utilize the Quadlet feature. To test this, I created a Python-based "Hello World" image. Initially, when I tried to load and run the image, the container executed successfully without any delays.

However, after upgrading Podman to 4.4.4 to use Quadlet, I encountered an issue where, upon completion of the container's execution, it took an additional 20 seconds, and I received the following error:

Exceeded conmon timeout waiting for container e6ac3bb749659a82023fbe35db7b972be8af7d2cc755a32095df04b2beb58e5f to exit

To resolve this, I upgraded not only the Conmon package but also the runc packages and libseccomp. Despite these updates, the issue persisted.

Steps to reproduce the issue

Steps to reproduce the issue

  1. Load the image with podman load -i "tar file name"
  2. Run the image with podman run -ti "image name"

Describe the results you received

root:# podman run -ti --log-level debug hello_world_py
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman run -ti --log-level debug hello_world_py)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /mnt/internal_storage/podman/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /mnt/internal_storage/podman/lib/containers/storage
DEBU[0000] Using run root /mnt/internal_storage/podman/run/containers/storage
DEBU[0000] Using static dir /mnt/internal_storage/podman/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /mnt/internal_storage/podman/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image hello_world_py (policy: missing)
DEBU[0000] Looking up image "hello_world_py" in local containers storage
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying "localhost/hello_world_py:latest" ...
DEBU[0000] Trying "docker.io/library/hello_world_py:latest" ...
DEBU[0000] Trying "registry.fedoraproject.org/hello_world_py:latest" ...
DEBU[0000] Trying "quay.io/hello_world_py:latest" ...
DEBU[0000] Trying "registry.access.redhat.com/hello_world_py:latest" ...
DEBU[0000] Trying "registry.centos.org/hello_world_py:latest" ...
DEBU[0000] Trying "docker.io/library/hello_world_py:latest" ...
DEBU[0000] Trying "hello_world_py" ...
DEBU[0000] Trying "docker.io/custom/hello_world_py:latest" ...
DEBU[0000] parsed reference into "[overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Found image "hello_world_py" as "docker.io/custom/hello_world_py:latest" in local containers storage
DEBU[0000] Found image "hello_world_py" as "docker.io/custom/hello_world_py:latest" in local containers storage ([overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255)
DEBU[0000] exporting opaque data as blob "sha256:be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Looking up image "docker.io/custom/hello_world_py:latest" in local containers storage
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux [] }
DEBU[0000] Trying "docker.io/custom/hello_world_py:latest" ...
DEBU[0000] parsed reference into "[overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Found image "docker.io/custom/hello_world_py:latest" as "docker.io/custom/hello_world_py:latest" in local containers storage
DEBU[0000] Found image "docker.io/custom/hello_world_py:latest" as "docker.io/custom/hello_world_py:latest" in local containers storage ([overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255)
DEBU[0000] exporting opaque data as blob "sha256:be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Looking up image "hello_world_py" in local containers storage
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux [] }
DEBU[0000] Trying "localhost/hello_world_py:latest" ...
DEBU[0000] Trying "docker.io/library/hello_world_py:latest" ...
DEBU[0000] Trying "registry.fedoraproject.org/hello_world_py:latest" ...
DEBU[0000] Trying "quay.io/hello_world_py:latest" ...
DEBU[0000] Trying "registry.access.redhat.com/hello_world_py:latest" ...
DEBU[0000] Trying "registry.centos.org/hello_world_py:latest" ...
DEBU[0000] Trying "docker.io/library/hello_world_py:latest" ...
DEBU[0000] Trying "hello_world_py" ...
DEBU[0000] Trying "docker.io/custom/hello_world_py:latest" ...
DEBU[0000] parsed reference into "[overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Found image "hello_world_py" as "docker.io/custom/hello_world_py:latest" in local containers storage
DEBU[0000] Found image "hello_world_py" as "docker.io/custom/hello_world_py:latest" in local containers storage ([overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255)
DEBU[0000] exporting opaque data as blob "sha256:be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Inspecting image be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255
DEBU[0000] exporting opaque data as blob "sha256:be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Inspecting image be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255
DEBU[0000] Inspecting image be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255
DEBU[0000] Inspecting image be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255
DEBU[0000] Inspecting image be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading default seccomp profile
DEBU[0000] Allocated lock 1 for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8
DEBU[0000] parsed reference into "[overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] exporting opaque data as blob "sha256:be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support
DEBU[0000] Created container "c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8"
DEBU[0000] Container "c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8" has work directory "/mnt/internal_storage/podman/lib/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata"
DEBU[0000] Container "c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8" has run directory "/mnt/internal_storage/podman/run/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata"
DEBU[0000] Handling terminal attach
INFO[0000] Received shutdown.Stop(), terminating! PID=1629
DEBU[0000] Enabling signal proxying
DEBU[0000] overlay: mount_data=lowerdir=/mnt/internal_storage/podman/lib/containers/storage/overlay/l/Y5NARJP77ENBSCPOQ5A6CFVBZK:/mnt/internal_storage/podman/lib/containers/storage/overlay/l/FZ2WWB4RC7VOOBDHZ2NFUQB3MS:/mnt/internal_storage/podman/lib/containers/storage/overlay/l/YCB5QE52WKM3PKPRTJAL5C7JYO:/mnt/internal_storage/podman/lib/containers/storage/overlay/l/MNTZI3SXWFG44CFMFLL5JLJNO6,upperdir=/mnt/internal_storage/podman/lib/containers/storage/overlay/5348e734723000ba186307d7b17d1df0ca7b0086a9884f1c2fd4e0848cae18ec/diff,workdir=/mnt/internal_storage/podman/lib/containers/storage/overlay/5348e734723000ba186307d7b17d1df0ca7b0086a9884f1c2fd4e0848cae18ec/work,
DEBU[0000] Mounted container "c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8" at "/mnt/internal_storage/podman/lib/containers/storage/overlay/5348e734723000ba186307d7b17d1df0ca7b0086a9884f1c2fd4e0848cae18ec/merged"
DEBU[0000] Created root filesystem for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 at /mnt/internal_storage/podman/lib/containers/storage/overlay/5348e734723000ba186307d7b17d1df0ca7b0086a9884f1c2fd4e0848cae18ec/merged
DEBU[0000] Made network namespace at /run/netns/netns-86e266ba-b56c-3e7e-bf14-663a1285af04 for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8
DEBU[0000] cni result for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 network podman: &{0.4.0 [{Name:cni-podman0 Mac:3e:d8:fc:ae:33:99 Sandbox:} {Name:veth84aaebd9 Mac:86:4b:a6:4b:16:f9 Sandbox:} {Name:eth0 Mac:b6:f7:34:42:10:7c Sandbox:/run/netns/netns-86e266ba-b56c-3e7e-bf14-663a1285af04}] [{Version:4 Interface:0x40003f3278 Address:{IP:10.88.0.7 Mask:ffff0000} Gateway:10.88.0.1}] [{Dst:{IP:0.0.0.0 Mask:00000000} GW:}] {[] [] []}}
DEBU[0000] Adding nameserver(s) from network status of '[]'
INFO[0000] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Setting Cgroups for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 to machine.slice:libpod:c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/mnt/internal_storage/podman/lib/containers/storage/overlay/5348e734723000ba186307d7b17d1df0ca7b0086a9884f1c2fd4e0848cae18ec/merged"
DEBU[0000] Created OCI spec for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 at /mnt/internal_storage/podman/lib/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -c c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 -u c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 -r /usr/bin/runc -b /mnt/internal_storage/podman/lib/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata -p /mnt/internal_storage/podman/run/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata/pidfile -n adoring_jones --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level debug --syslog -t --conmon-pidfile /mnt/internal_storage/podman/run/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /mnt/internal_storage/podman/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /mnt/internal_storage/podman/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /mnt/internal_storage/podman/lib/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8.scope
DEBU[0000] Received: 1714
INFO[0000] Got Conmon PID as 1703
DEBU[0000] Created container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 in OCI runtime
DEBU[0000] Attaching to container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8
DEBU[0000] Received a resize event: {Width:142 Height:36}
DEBU[0000] Starting container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 with command [/bin/sh -c /usr/bin/python3 /usr/local/bin/hello_world.py]
DEBU[0000] Started container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8
DEBU[0000] Notify sent successfully
hello world
DEBU[0020] Exceeded conmon timeout waiting for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 to exit
DEBU[0020] Called run.PersistentPostRunE(podman run -ti --log-level debug hello_world_py)
DEBU[0020] Shutting down engines

Describe the results you expected

it should not take extra time after the execution of the container is completed.

podman info output

podman info --debug
host:
  arch: arm64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: 9a3af8f438d89ffc176f3c76c164088e64bb6b19'
  cpuUtilization:
    idlePercent: 98.95
    systemPercent: 0.78
    userPercent: 0.27
  cpus: 4
  distribution:
    distribution: custom-systemd-wayland-virtualization
    version: 1.0.0
  eventLogger: journald
  hostname: custom-imx8mp-bliss2g
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.52+gc56eb7b65d5a
  linkmode: dynamic
  logDriver: journald
  memFree: 897892352
  memTotal: 2074460160
  networkBackend: cni
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.1.7+dev
      commit: v1.1.7-2-gb6109acd-dirty
      spec: 1.0.2-dev
      go: go1.20.7
      libseccomp: 2.5.4
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.0-beta.0+dev
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 0h 34m 33.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /mnt/internal_storage/podman/lib/containers/storage
  graphRootAllocated: 28011606016
  graphRootUsed: 327458816
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /mnt/internal_storage/podman/temp
  imageStore:
    number: 1
  runRoot: /mnt/internal_storage/podman/run/containers/storage
  transientStore: false
  volumePath: /mnt/internal_storage/podman/lib/containers/storage/volumes
version:
  APIVersion: 4.4.4
  Built: 1679938614
  BuiltTime: Mon Mar 27 17:36:54 2023
  GitCommit: c8223435f49a860c8ac4281a85db39232745a7cb
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/arm64
  Version: 4.4.4

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

No

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions