Skip to content

test(migrations): parallel tests rebuild the same test image tag, so docker run can miss it #5299

Open

Description

One test in hack/migration-54-redis-adopt.bats failed once because the jq-enabled test image was not there when docker run asked for it. This was on a full unit run under bats -j 4. Re-running the same tree passed, and an earlier full run on a neighbouring tree passed too, so it is 1 failure in 3 runs. The file is not touched by the change that was being tested.

not ok 1295 absent CRD stamps 55 without patching anything # in 5080 ms
# (in test file hack/migration-54-redis-adopt.bats, line 149)
#   `[ "$rc" -eq 0 ]' failed
# DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
# Unable to find image 'cozystack-migration54-test:alpine-3-24-sha256-294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6' locally
# docker: Error response from daemon: pull access denied for cozystack-migration54-test, repository does not exist or may require 'docker login'

Every test in the file builds and tags the same image. prep() runs docker build -q -t "$TESTIMG" with a fixed tag derived from the alpine pin, and every test calls prep(). Tests in one file run concurrently under bats -j N unless --no-parallelize-within-files is given, and make bats-unit-tests sets -j to nproc when GNU parallel is installed. So on a cold cache several builds of that tag run at once.

Retagging is not atomic on the daemon side. The legacy builder writes a new image config per build, so two uncached builds give two image IDs. With the containerd image store, an existing name with a different target is replaced by deleting the old image and then creating the new one (createOrReplaceImage in daemon/containerd/image_tag.go, moby tag docker-v29.9.3). A docker run from a sibling test that resolves the tag between those two calls finds nothing locally and falls back to a pull, which is the error above. Once the cache is warm the builds produce the same ID and the retag is a no-op, which fits a failure that only shows up rarely.

The daemon of these runs is docker:29.9.3-dind with the containerd image store (overlayfs snapshotter), checked on a daemon started from the same image and flags. The same pattern, one fixed tag rebuilt in prep() by every test, is in the migration 55, 57, 58 and 59 test files too.

Building the image once per file in setup_file, before any test runs, would remove the concurrent builds. Another option is to run these files with --no-parallelize-within-files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage/needs-triageIndicates an issue needs triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions