Overlay snapshotter should return overlayfs mount for parallel… by henry118 · Pull Request #13044 · containerd/containerd

henry118 · 2026-03-16T23:59:10Z

During parallel unpack, "parent" is not passed in for snapshot prepare. The result is that overlay SN will return bind mounts instead of overlay mounts.

This PR introduces a label to indicate that the snapshot is to be "rebased" later. This will allow the overlay SN to return overlay mount. This way the whiteout files can be correctly handled by diff-apply logic.

This bug seems only apply to overlay. EROFS works fine based on my testing.

dmcgowan · 2026-03-17T00:23:13Z

This bug seems only apply to overlay. EROFS works fine based on my testing.

That's because erofs uses its a different applier. If it used the default walking applier it would fail as well. Its probably cleaner to have the default walking applier not have any special logic and have different appliers for these use cases. Such as with #12842. Parallel unpack would only be used together with an unpack configuration that knew how to handle it and what differ to use anyway. Without that special logic, then it would just do serial unpack with the walking differ.

For backporting, this change is probably the smallest and best to get in though. I just don't think it is the cleaner long term solution.

henry118 · 2026-03-17T02:31:59Z

Ah I wasn't aware of #12842. I'll try to add an overlay specific applier on top of that work.

samuelkarp · 2026-03-17T06:05:42Z

We should have an integration test around this too, reproing the reported issue.

henry118 · 2026-03-17T19:15:12Z

We should have an integration test around this too, reproing the reported issue.

Yes I have an integration test, but it requires a testing image. Currently the image is hosted on AWS ECR. It's better to be added to containerd's gchr. Could use some help with that as I don't have permission.

$ cat Dockerfile
FROM alpine

RUN touch /file-to-delete
RUN rm /file-to-delete

RUN mkdir /dir-to-delete && touch /dir-to-delete/foo
RUN rm -rf /dir-to-delete

samuelkarp · 2026-03-17T20:08:57Z

Should we host the image in ghcr or just build it as part of the test? I think either is reasonable.

henry118 · 2026-03-17T22:44:07Z

It would be great if the test image is held in ghcr. That'll greatly simplify the test implementation.

Other benefits include:

No runtime dependency on docker, buildx and buildkit (we have to build multi-arch image for both amd64 and arm)
Test image can be reused by multiple tests
The image has to be in registry to test the image pull workflow.

samuelkarp · 2026-03-17T23:53:50Z

I just ran the workflow to mirror your image: https://github.com/containerd/containerd/pkgs/container/whiteout-test

henry118 · 2026-03-18T16:08:22Z

Thanks @samuelkarp. The test is updated to use the ghcr image.

samuelkarp

Mostly looks good. One question about userns, a nit about the test, and some suggested comments to make it clearer to a future reader.

halaney · 2026-03-19T18:58:27Z

+		// If we only have one layer/no parents then just return a bind mount as overlay
+		// will not work. The exception is parallel unpack initially has parentless layers
+		// that need to be rebased on a parent later. For this case, the "overlay" mount
+		// is returned to signal the optimized fast-path in the walking applier.


I must admit that I don't actually follow on a quick read what the fast path / slow path for the walking applier is (that's just my shortcomings of internals here).

Does this trick work if we are doing a "parallel" unpack of a single layer image?

Yes single layer image will work:

The label is only applied to upper layers during unpack. A single layer of an image won't be labeled and will follow the slow-path (tmp mount);

However, we can execute the fast-path for single layer too, because the fast-path doesn't really need the parents to apply the layer content.

I chose to implement the 1st approach in this PR to align more closely with the existing logic.

fuweid · 2026-03-19T22:26:42Z

 	for i := range s.ParentIDs {
 		parentPaths[i] = o.upperPath(s.ParentIDs[i])
 	}
 	options = append(options, fmt.Sprintf("lowerdir=%s", strings.Join(parentPaths, ":")))


kernel will post error like overlay: Bad value for 'lowerdir' if parentPaths is empty.
I am thinking we can use grpc metadata (just like namespace) wrapper and pass option to differ and then it can pick up whiteout converter.

I don't think this problem would happen per the current logic.
This malformed mount option is only used by the fast-path of the applier logic during unpacking.
Regular mount operations will always have parents due to rebase.

Happy path is always happy. I mean it's trying to poke hole as label can be updated by API.
Just feel like it's short-term fix but poke hole of system. I am not against introducing API-level change on applier

I'm working on #13053 as long term fix.

Any chance this PR get in for v2.2 back-port in the interim?

fuweid · 2026-03-20T01:36:31Z

 		}
 		snapshotLabels[labelSnapshotRef] = chainID

+		if i > 0 && parallel {


or tweak the mount info if's overlay snapshot (it's hack)

fuweid · 2026-03-24T03:52:15Z

The change can work. But I still think we should let snapshot return correct mount information and client side can tweak that to let applier to use whiteout converter. Let other reviewers chime in as well.

henry118 · 2026-03-24T18:57:32Z

The change can work. But I still think we should let snapshot return correct mount information and client side can tweak that to let applier to use whiteout converter. Let other reviewers chime in as well.

@fuweid I implemented the alternative solution in #13115. Does it align what you've been thinking?

Signed-off-by: Henry Wang <[email protected]>

henry118 · 2026-03-24T19:18:16Z

I also rebased this PR and got rid of the original label proposed in this PR.

We can just reuse the label introduced by #13071.

henry118 · 2026-03-25T21:32:16Z

#13115 was merged. closing this.

github-project-automation Bot added this to Pull Request Review Mar 16, 2026

github-project-automation Bot moved this to Needs Triage in Pull Request Review Mar 16, 2026

k8s-ci-robot added the size/M label Mar 16, 2026

dosubot Bot added the kind/bug label Mar 16, 2026

henry118 force-pushed the bug-13030 branch from c8ced64 to e2ace7f Compare March 17, 2026 00:00

henry118 mentioned this pull request Mar 17, 2026

whiteout files not honored by max_concurrent_unpacks > 1 #13030

Closed

henry118 force-pushed the bug-13030 branch from e2ace7f to b17ebb7 Compare March 17, 2026 19:06

k8s-ci-robot added size/L and removed size/M labels Mar 17, 2026

henry118 force-pushed the bug-13030 branch from b17ebb7 to 8bc06e6 Compare March 17, 2026 19:11

henry118 changed the title ~~Overlay snapshotter should return overlayfs mount for during parallel…~~ Overlay snapshotter should return overlayfs mount for parallel… Mar 17, 2026

henry118 force-pushed the bug-13030 branch from 8bc06e6 to ce7b39e Compare March 17, 2026 19:58

samuelkarp added the cherry-pick/2.2.x Change to be cherry picked to release/2.2 branch label Mar 17, 2026

henry118 force-pushed the bug-13030 branch from ce7b39e to 0573620 Compare March 18, 2026 02:01

henry118 mentioned this pull request Mar 18, 2026

Add overlay specific applier #13053

Open

samuelkarp reviewed Mar 18, 2026

View reviewed changes

Comment thread integration/client/container_linux_test.go Outdated

Comment thread plugins/snapshots/overlay/plugin/plugin.go

Comment thread plugins/snapshots/overlay/overlay.go Outdated

Comment thread core/unpack/unpacker.go Outdated

henry118 force-pushed the bug-13030 branch from 0573620 to bd84fdd Compare March 18, 2026 22:42

samuelkarp approved these changes Mar 19, 2026

View reviewed changes

henry118 force-pushed the bug-13030 branch from bd84fdd to d743175 Compare March 19, 2026 08:11

halaney reviewed Mar 19, 2026

View reviewed changes

fuweid reviewed Mar 19, 2026

View reviewed changes

fuweid reviewed Mar 20, 2026

View reviewed changes

k8s-ci-robot added the needs-rebase label Mar 24, 2026

henry118 mentioned this pull request Mar 24, 2026

Tweak mount info for overlayfs in case of parallel unpack #13115

Merged

henry118 added 2 commits March 24, 2026 19:10

Add integration test for issue 13030

ba31d2a

Signed-off-by: Henry Wang <[email protected]>

Overlay snapshotter should return overlayfs mount for parallel unpack

33b5bc3

Signed-off-by: Henry Wang <[email protected]>

henry118 force-pushed the bug-13030 branch from d743175 to 33b5bc3 Compare March 24, 2026 19:12

k8s-ci-robot removed the needs-rebase label Mar 24, 2026

dmcgowan approved these changes Mar 24, 2026

View reviewed changes

github-project-automation Bot moved this from Needs Triage to Review In Progress in Pull Request Review Mar 24, 2026

henry118 closed this Mar 25, 2026

github-project-automation Bot moved this from Review In Progress to Done in Pull Request Review Mar 25, 2026

henry118 deleted the bug-13030 branch March 26, 2026 17:43

Conversation

henry118 commented Mar 16, 2026

Uh oh!

dmcgowan commented Mar 17, 2026

Uh oh!

henry118 commented Mar 17, 2026

Uh oh!

samuelkarp commented Mar 17, 2026

Uh oh!

henry118 commented Mar 17, 2026

Uh oh!

samuelkarp commented Mar 17, 2026

Uh oh!

henry118 commented Mar 17, 2026

Uh oh!

samuelkarp commented Mar 17, 2026

Uh oh!

henry118 commented Mar 18, 2026

Uh oh!

samuelkarp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

halaney Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

henry118 Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

fuweid Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

henry118 Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

fuweid Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

henry118 Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

fuweid Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

fuweid commented Mar 24, 2026

Uh oh!

henry118 commented Mar 24, 2026

Uh oh!

henry118 commented Mar 24, 2026

Uh oh!

henry118 commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants