Fix ctr image mount failing with no such device by ticpu · Pull Request #12581 · containerd/containerd

ticpu · 2025-11-26T23:41:03Z

Fix for #12549, bind mount missing rbind option.

The bind mount created for temporary activations was missing the Options field, causing mount to fail with "no such device" because the MS_BIND flag wasn't being set.

Test was failing first:

❯ sudo go test -v -run "TestTemporary" ./core/mount/manager/ -test.root
[sudo: authenticate] Password: 
=== RUN   TestTemporaryMountActivation
    log_hook.go:47: time="2025-11-26T18:25:54.833945756-05:00" level=debug msg="activating mount" func="manager.(*mountManager).Activate" file="/mnt/bcachefs/home/jerome/GIT/containerd/core/mount/manager/manager.go:134" mounts="[{bind /tmp/TestTemporaryMountActivation905822259/001/source  [rbind ro]}]" name=temp-mount-test testcase=TestTemporaryMountActivation
    manager_linux_test.go:389: 
        	Error Trace:	/mnt/bcachefs/home/jerome/GIT/containerd/core/mount/manager/manager_linux_test.go:389
        	Error:      	Received unexpected error:
        	           	mount source: "/tmp/TestTemporaryMountActivation905822259/001/m/1/1", target: "/tmp/TestTemporaryMountActivation905822259/001/target", fstype: bind, flags: 0, data: "", err: no such device
        	Test:       	TestTemporaryMountActivation
        	Messages:   	Should be able to mount system mounts to target
--- FAIL: TestTemporaryMountActivation (0.01s)
=== RUN   TestTemporaryOverlayMountActivation
    log_hook.go:47: time="2025-11-26T18:25:54.842703306-05:00" level=debug msg="activating mount" func="manager.(*mountManager).Activate" file="/mnt/bcachefs/home/jerome/GIT/containerd/core/mount/manager/manager.go:134" mounts="[{overlay overlay  [lowerdir=/tmp/TestTemporaryOverlayMountActivation2818473812/001/lower2:/tmp/TestTemporaryOverlayMountActivation2818473812/001/lower1 upperdir=/tmp/TestTemporaryOverlayMountActivation2818473812/001/upper workdir=/tmp/TestTemporaryOverlayMountActivation2818473812/001/work]}]" name=temp-overlay-test testcase=TestTemporaryOverlayMountActivation
    manager_linux_test.go:481: 
        	Error Trace:	/mnt/bcachefs/home/jerome/GIT/containerd/core/mount/manager/manager_linux_test.go:481
        	Error:      	Received unexpected error:
        	           	mount source: "/tmp/TestTemporaryOverlayMountActivation2818473812/001/m/1/1", target: "/tmp/TestTemporaryOverlayMountActivation2818473812/001/target", fstype: bind, flags: 0, data: "", err: no such device
        	Test:       	TestTemporaryOverlayMountActivation
        	Messages:   	Should be able to mount system mounts to target
--- FAIL: TestTemporaryOverlayMountActivation (0.00s)
FAIL
FAIL	github.com/containerd/containerd/v2/core/mount/manager	0.013s
FAIL

Now success:

❯ sudo go test -v -run "TestTemporary" ./core/mount/manager/ -test.root
=== RUN   TestTemporaryMountActivation
    log_hook.go:47: time="2025-11-26T18:40:46.444153608-05:00" level=debug msg="activating mount" func="manager.(*mountManager).Activate" file="/mnt/bcachefs/home/jerome/GIT/containerd/core/mount/manager/manager.go:134" mounts="[{bind /tmp/TestTemporaryMountActivation4144393487/001/source  [rbind ro]}]" name=temp-mount-test testcase=TestTemporaryMountActivation
    helpers.go:100: unmount /tmp/TestTemporaryMountActivation4144393487/001/target
--- PASS: TestTemporaryMountActivation (0.00s)
=== RUN   TestTemporaryOverlayMountActivation
    log_hook.go:47: time="2025-11-26T18:40:46.444820178-05:00" level=debug msg="activating mount" func="manager.(*mountManager).Activate" file="/mnt/bcachefs/home/jerome/GIT/containerd/core/mount/manager/manager.go:134" mounts="[{overlay overlay  [lowerdir=/tmp/TestTemporaryOverlayMountActivation2175850857/001/lower2:/tmp/TestTemporaryOverlayMountActivation2175850857/001/lower1 upperdir=/tmp/TestTemporaryOverlayMountActivation2175850857/001/upper workdir=/tmp/TestTemporaryOverlayMountActivation2175850857/001/work]}]" name=temp-overlay-test testcase=TestTemporaryOverlayMountActivation
    helpers.go:100: unmount /tmp/TestTemporaryOverlayMountActivation2175850857/001/target
--- PASS: TestTemporaryOverlayMountActivation (0.00s)
PASS
ok  	github.com/containerd/containerd/v2/core/mount/manager	0.004s

Add tests for the WithTemporary mount activation used by `ctr images mount`. Covers bind mount and overlay scenarios to catch regressions like containerd#12549. Signed-off-by: Jérôme Poulin <[email protected]>

The bind mount created for temporary activations was missing the Options field, causing mount to fail with "no such device" because the MS_BIND flag wasn't being set. Fixes containerd#12549 Signed-off-by: Jérôme Poulin <[email protected]>

hsiangkao

LGTM, I think it's needed for sub-mounts Cc @dmcgowan @fuweid

AntonOfTheWoods · 2026-01-08T11:56:33Z

@dmcgowan @fuweid , this has been opened and LGTMed by @hsiangkao for a while now. Is there something else holding this up?

AkihiroSuda · 2026-01-28T03:02:13Z

/cherry-pick release/2.2
/cherry-pick release/2.1

k8s-infra-cherrypick-robot · 2026-01-28T03:02:15Z

@AkihiroSuda: once the present PR merges, I will cherry-pick it on top of release/2.1, release/2.2 in new PRs and assign them to you.

Details

In response to this:

/cherry-pick release/2.2
/cherry-pick release/2.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-infra-cherrypick-robot · 2026-01-28T03:35:21Z

@AkihiroSuda: #12581 failed to apply on top of branch "release/2.1":

Applying: core/mount/manager: add tests for WithTemporary option
Using index info to reconstruct a base tree...
A	core/mount/manager/manager_linux_test.go
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): core/mount/manager/manager_linux_test.go deleted in HEAD and modified in core/mount/manager: add tests for WithTemporary option. Version core/mount/manager: add tests for WithTemporary option of core/mount/manager/manager_linux_test.go left in tree.
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 core/mount/manager: add tests for WithTemporary option

Details

In response to this:

/cherry-pick release/2.2
/cherry-pick release/2.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-infra-cherrypick-robot · 2026-01-28T03:35:57Z

@AkihiroSuda: new pull request created: #12831

Details

In response to this:

/cherry-pick release/2.2
/cherry-pick release/2.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

hsiangkao · 2026-01-28T03:47:10Z

@AkihiroSuda containerd 2.1 is not impacted.

The docker-popular-containers pipeline started failing from a regression introduced by containerd v2.2.0. An issue has been lodged on their repo, and the fix has seemingly been merged into the v2.2 branch, we are just waiting on a new release version to pick the fix up. In the meantime, we can disable the faulty plugin mount-manager (which was only introduced in v2.2, so we aren't missing out on anything). Relevant links: - Issue: containerd/containerd#12549 - Fix (main branch): containerd/containerd#12581 - Fix (v2.2 branch): containerd/containerd#12831 Signed-off-by: James Curtis <[email protected]>

github-project-automation Bot added this to Pull Request Review Nov 26, 2025

github-project-automation Bot moved this to Needs Triage in Pull Request Review Nov 26, 2025

k8s-ci-robot added the size/L label Nov 26, 2025

dosubot Bot added area/runtime Runtime kind/bug labels Nov 26, 2025

ticpu added 2 commits November 26, 2025 18:42

core/mount/manager: add tests for WithTemporary option

d2593b6

Add tests for the WithTemporary mount activation used by `ctr images mount`. Covers bind mount and overlay scenarios to catch regressions like containerd#12549. Signed-off-by: Jérôme Poulin <[email protected]>

ticpu force-pushed the fix-12549-ctr-image-mount-fails-no-such-device branch from 9b296c1 to 776e50a Compare November 26, 2025 23:42

hsiangkao approved these changes Nov 27, 2025

View reviewed changes

LucaBernstein mentioned this pull request Dec 18, 2025

Update kindest/node Docker tag to v1.34.3 gardener/gardener#13651

Closed

1 task

majst01 mentioned this pull request Jan 12, 2026

Debian 13 metal-stack/metal-images#370

Open

4 tasks

AkihiroSuda approved these changes Jan 28, 2026

View reviewed changes

github-project-automation Bot moved this from Needs Triage to Review In Progress in Pull Request Review Jan 28, 2026

AkihiroSuda added this pull request to the merge queue Jan 28, 2026

dmcgowan approved these changes Jan 28, 2026

View reviewed changes

AkihiroSuda added cherry-pick/2.1.x Change to be cherry picked to release/2.1 branch cherry-pick/2.2.x Change to be cherry picked to release/2.2 branch labels Jan 28, 2026

Merged via the queue into containerd:main with commit 317286a Jan 28, 2026
90 of 92 checks passed

github-project-automation Bot moved this from Review In Progress to Done in Pull Request Review Jan 28, 2026

k8s-infra-cherrypick-robot mentioned this pull request Jan 28, 2026

[release/2.2] Fix ctr image mount failing with "no such device" #12831

Merged

AkihiroSuda added cherry-picked/2.2.x PR commits are cherry-picked into release/2.2 branch and removed cherry-pick/2.2.x Change to be cherry picked to release/2.2 branch labels Jan 28, 2026

AkihiroSuda removed the cherry-pick/2.1.x Change to be cherry picked to release/2.1 branch label Jan 28, 2026

ticpu deleted the fix-12549-ctr-image-mount-fails-no-such-device branch February 25, 2026 23:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix ctr image mount failing with no such device#12581

Fix ctr image mount failing with no such device#12581
AkihiroSuda merged 2 commits intocontainerd:mainfrom
ticpu:fix-12549-ctr-image-mount-fails-no-such-device

ticpu commented Nov 26, 2025

Uh oh!

hsiangkao left a comment

Uh oh!

AntonOfTheWoods commented Jan 8, 2026

Uh oh!

AkihiroSuda commented Jan 28, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jan 28, 2026

Uh oh!

Uh oh!

k8s-infra-cherrypick-robot commented Jan 28, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jan 28, 2026

Uh oh!

hsiangkao commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

ticpu commented Nov 26, 2025

Uh oh!

hsiangkao left a comment

Choose a reason for hiding this comment

Uh oh!

AntonOfTheWoods commented Jan 8, 2026

Uh oh!

AkihiroSuda commented Jan 28, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jan 28, 2026

Uh oh!

Uh oh!

k8s-infra-cherrypick-robot commented Jan 28, 2026

Uh oh!

k8s-infra-cherrypick-robot commented Jan 28, 2026

Uh oh!

hsiangkao commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants