Add overlay specific applier by henry118 · Pull Request #13053 · containerd/containerd

henry118 · 2026-03-18T21:55:01Z

This is a series of patches that aims for a long term solution for #13030. It will eventually replace the short term workaround implemented by #13044.

The patch implements an overlay specific differ, which is also appended to the default differ list in the diff service, and the walking differ is blacklisted for parallel unpacking in v2.3.

In order to support local pull, I had to add a new parallel field to ApplyRequest proto, so that differs under diff service can reject the request if it does not support parallel mode.

This PR can be rebased on top of #12842 if that lands first.

samuelkarp · 2026-03-19T05:35:30Z

blacklists walking differ for parallel unpacking.

If we want to deny the walking differ, we should deprecate it and add a deprecation warning in 2.3 for any manual configuration of the walking differ. Some of my colleagues have been testing a proxy snapshotter with parallel unpack using the walking differ.

If we don't add this, we should delay the denial of walking differ until 2.7.

samuelkarp · 2026-04-15T22:58:20Z

@henry118 Can you rebase?

Copilot

Pull request overview

Adds an overlay-specific applier/differ path to support correct whiteout handling during parallel unpack, and plumbs a new parallel signal through the diff service API so appliers can opt out when parallel mode is requested.

Changes:

Introduces a Linux-only overlay diff plugin that uses a custom apply function with explicit parallel support.
Adds parallel to the DiffService ApplyRequest proto and wires it through local/proxy diff clients plus diff.WithParallel.
Updates unpack logic to request parallel apply when supported and to avoid using the walking applier for parallel unpack.

Reviewed changes

Copilot reviewed 23 out of 28 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
vendor/modules.txt	Updates vendoring metadata for the local `./api` replacement.
vendor/github.com/containerd/containerd/api/services/diff/v1/diff.proto	Vendored proto update for `parallel` field.
vendor/github.com/containerd/containerd/api/services/diff/v1/diff.pb.go	Vendored generated Go update for `parallel` field.
vendor/github.com/containerd/containerd/api/LICENSE	Removes vendored license file for the api module.
plugins/transfer/plugin.go	Tracks the selected applier ID and adds WarningPlugin dependency.
plugins/services/diff/service_unix.go	Appends `overlay` to the default differ order on Unix.
plugins/services/diff/local.go	Passes `parallel` through diff apply options.
plugins/diff/walking/plugin/plugin.go	Switches walking plugin to the new filesystem applier constructor.
plugins/diff/overlay/plugin/plugin_other.go	Non-Linux stub package for overlay plugin.
plugins/diff/overlay/plugin/plugin_linux.go	Registers `overlay` diff plugin (custom apply func + parallel support).
plugins/diff/overlay/applier_other.go	Non-Linux `Apply` returns `ErrNotImplemented`.
plugins/diff/overlay/applier_linux_test.go	Adds unit tests for overlay option parsing and invalid parameters.
plugins/diff/overlay/applier_linux.go	Implements overlay-specific apply behavior (whiteout conversion on bind/overlay mounts).
integration/client/migration_test.go	Updates migration expectations for old defaults to include `default=['walking']`.
integration/build_local_containerd_helper_test.go	Ensures overlay diff plugin is linked for integration tests.
go.sum	Removes sums for the replaced `containerd/api` module.
go.mod	Adds `replace github.com/containerd/containerd/api => ./api`.
core/unpack/unpacker_test.go	Removes tests for the prior bind→overlay workaround.
core/unpack/unpacker.go	Propagates `parallel` into apply opts; deny-lists walking applier for parallel; removes workaround.
core/diff/proxy/differ.go	Plumbs `parallel` into remote DiffService ApplyRequest.
core/diff/diff.go	Adds `ApplyConfig.Parallel` and `diff.WithParallel`.
core/diff/apply/apply.go	Refactors filesystem applier to support custom apply funcs and parallel capability gating.
contrib/diffservice/service.go	Plumbs `parallel` into the contrib diffservice Apply path.
cmd/containerd/builtins/builtins_linux.go	Registers overlay diff plugin in Linux builtins.
api/services/diff/v1/diff.proto	Adds `parallel` field to ApplyRequest.
api/services/diff/v1/diff.pb.go	Regenerates Go bindings for the updated proto.
api/next.txtpb	Updates API descriptor textproto for the new field.
RELEASES.md	Documents walking differ behavior vs. parallel unpack and the new overlay differ.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-23T19:07:49Z

+	for _, o := range opts {
+		if err := o(&config); err != nil {
+			return nil


NewFileSystemApplier silently returns nil when an option returns an error. Returning a nil diff.Applier will typically lead to a panic later and also drops the original error context, making failures hard to debug. Consider changing the API to return (diff.Applier, error), or (if keeping the current signature) logging/panicking immediately so the root cause is visible.

Suggested change

for _, o := range opts {

if err := o(&config); err != nil {

return nil

for i, o := range opts {

if err := o(&config); err != nil {

panic(fmt.Errorf("NewFileSystemApplier option %d failed: %w", i, err))

An error log message was added to aim the troubleshooting. which should be enough to address this concern.

Might be better to return an error here rather than nil still.

Adding an error to the return values would be a breaking change, which would require a new function. Do we want to do that?

henry118 · 2026-04-23T21:21:57Z

@henry118 Can you rebase?

@samuelkarp Sorry for the delay as I was out for the past few days. I've rebased this PR and resolved the conflicts. PTAL, thanks!

samuelkarp · 2026-04-23T23:10:26Z

+	for _, o := range opts {
+		if err := o(&config); err != nil {
+			return nil


Might be better to return an error here rather than nil still.

samuelkarp · 2026-04-23T23:20:46Z

+	// The ApplierID can be empty if the unpack is performed on the client side,
+	// where the applier is just a proxy. In that case the diff service will try each
+	// applier in a configured order.
+	var denylist = []string{"walking"}


Thinking about this more: now that we have config version 4, it would be nice to add a migration for overlay + explicit walking + parallel to the new overlay applier. That way existing configs will continue to work on upgrade, and the migrate command should show the new correct value.

Yep good point. I've added the migration logic

Signed-off-by: Henry Wang <[email protected]>

Copilot

Pull request overview

Copilot reviewed 25 out of 30 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 25 out of 30 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: Henry Wang <[email protected]>

This reverts commit 3382fb7. Signed-off-by: Henry Wang <[email protected]>

Signed-off-by: Henry Wang <[email protected]>

samuelkarp · 2026-04-27T23:32:13Z

+	SyncFs: false,
+}
+
+func configMigration(ctx context.Context, configVersion int, pluginConfigs map[string]any) error {


This needs a unit test.

samuelkarp · 2026-04-27T23:32:49Z

+}
+
+func configMigration(ctx context.Context, configVersion int, pluginConfigs map[string]any) error {
+	if configVersion >= version.ConfigVersion {


Probably not just version.ConfigVersion, but the specific versions we're looking at for the migration. This migration is for moving up to version 4.

samuelkarp · 2026-04-27T23:34:58Z

+	var new []any
+	for _, v := range old {
+		new = append(new, v)
+		if v == "walking" {


It's currently (i.e., before this PR) valid to configure "walking" with "rebase". So it's not just the order that should be modified, but also the capabilities.

rebase is a build-in feature of a snapshotter, which will automatically advertise the cap when it's initialized. currently there is no tight coupling between a differ and a snapshotter (but maybe it makes sense if there was?).

In terms of configuration, there are two cases:

Image pull with transfer service: where users have the option to configure a combination of "snapshotter", "differ" and "unpack parallelism". In this case, parallel unpack is only enabled if a/ snapshotter supports "rebase"; b/ differ is overlay / erofs; c/ configured concurrent unpack > 1;

Local image pull from client side: where client can only specify "snapshotter" and "unpack parallelism". The differ is invoked in the order configured in the diff service. With the change in this PR, image pull would fail if there is an explicit walking in the list. So i think a migration would help to mitigate this risk.

So i have the following questions:

Could you help me to understand what's the actual migration use case you were referring to?

Also given this PR is not part of 2.3, can we make the migration as a separate patch based on the possibility of a new config version number?

Let me know. thanks!

samuelkarp · 2026-04-27T23:35:51Z

I'm removing this from the 2.3 milestone since we're currently working on getting the release candidate out: #13302. I think at this point we should aim for inclusion in 2.4.

k8s-ci-robot · 2026-04-29T17:48:45Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

github-project-automation Bot added this to Pull Request Review Mar 18, 2026

github-project-automation Bot moved this to Needs Triage in Pull Request Review Mar 18, 2026

k8s-ci-robot added do-not-merge/work-in-progress size/L labels Mar 18, 2026

henry118 force-pushed the overlay branch 2 times, most recently from 1569370 to 3bcc5ad Compare March 19, 2026 05:19

henry118 force-pushed the overlay branch 2 times, most recently from 34e7e4e to ef1537c Compare March 19, 2026 20:47

k8s-ci-robot added size/XXL and removed size/L labels Mar 19, 2026

henry118 force-pushed the overlay branch from ef1537c to 68ea431 Compare March 19, 2026 21:41

henry118 marked this pull request as ready for review March 20, 2026 00:28

k8s-ci-robot removed the do-not-merge/work-in-progress label Mar 20, 2026

dosubot Bot added area/runtime Runtime kind/feature labels Mar 20, 2026

henry118 mentioned this pull request Mar 20, 2026

Overlay snapshotter should return overlayfs mount for parallel… #13044

Closed

k8s-ci-robot added the needs-rebase label Mar 23, 2026

dmcgowan added this to the 2.3 milestone Apr 9, 2026

dmcgowan moved this from Needs Triage to Needs Update in Pull Request Review Apr 10, 2026

dmcgowan moved this from Needs Update to Needs Reviewers in Pull Request Review Apr 10, 2026

samuelkarp reviewed Apr 15, 2026

View reviewed changes

Comment thread core/unpack/unpacker.go Outdated

Comment thread pkg/deprecation/deprecation.go Outdated

samuelkarp added the status/needs-update Awaiting contributor update label Apr 22, 2026

Copilot AI review requested due to automatic review settings April 23, 2026 18:59

henry118 force-pushed the overlay branch from 68ea431 to 1f37656 Compare April 23, 2026 18:59

k8s-ci-robot removed the needs-rebase label Apr 23, 2026

Copilot started reviewing on behalf of henry118 April 23, 2026 18:59 View session

Copilot AI reviewed Apr 23, 2026

View reviewed changes

henry118 force-pushed the overlay branch from 1f37656 to 84d80f3 Compare April 23, 2026 20:30

samuelkarp reviewed Apr 23, 2026

View reviewed changes

Add parallel field to ApplyRequest proto

e04f3dc

Signed-off-by: Henry Wang <[email protected]>

Copilot AI review requested due to automatic review settings April 24, 2026 03:40

henry118 force-pushed the overlay branch from 84d80f3 to c0ad22c Compare April 24, 2026 03:40

Copilot started reviewing on behalf of henry118 April 24, 2026 03:41 View session

Copilot AI reviewed Apr 24, 2026

View reviewed changes

Comment thread plugins/services/diff/service_linux.go Outdated

Comment thread plugins/services/diff/service_linux.go

Comment thread core/diff/apply/apply.go

Comment thread core/diff/apply/apply.go

Comment thread plugins/transfer/plugin.go Outdated

Comment thread core/unpack/unpacker.go

henry118 force-pushed the overlay branch from c0ad22c to ef25bb9 Compare April 24, 2026 03:50

Copilot AI review requested due to automatic review settings April 24, 2026 04:22

henry118 force-pushed the overlay branch from ef25bb9 to 5b313b8 Compare April 24, 2026 04:22

Copilot started reviewing on behalf of henry118 April 24, 2026 04:22 View session

Copilot AI reviewed Apr 24, 2026

View reviewed changes

Comment thread core/diff/apply/apply.go

henry118 added 5 commits April 24, 2026 04:34

Customizable filesystem applier

b8d33f3

Signed-off-by: Henry Wang <[email protected]>

Add overlay specific applier

8ab3be7

Signed-off-by: Henry Wang <[email protected]>

Denylist walking differ for parallel unpack feature

b3075ec

Signed-off-by: Henry Wang <[email protected]>

Revert "Tweak mount info for overlayfs in case of parallel unpack"

675a0ae

This reverts commit 3382fb7. Signed-off-by: Henry Wang <[email protected]>

Update RELEASE.md about the change of walking differ on parallel unpack

33b2fd0

Signed-off-by: Henry Wang <[email protected]>

henry118 force-pushed the overlay branch from 5b313b8 to 33b2fd0 Compare April 24, 2026 04:34

henry118 removed the status/needs-update Awaiting contributor update label Apr 24, 2026

samuelkarp reviewed Apr 27, 2026

View reviewed changes

samuelkarp removed this from the 2.3 milestone Apr 27, 2026

k8s-ci-robot added the needs-rebase label Apr 29, 2026

Conversation

henry118 commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samuelkarp commented Mar 19, 2026

Uh oh!

samuelkarp commented Apr 15, 2026

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

henry118 commented Apr 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samuelkarp commented Apr 27, 2026

Uh oh!

k8s-ci-robot commented Apr 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

henry118 commented Mar 18, 2026 •

edited

Loading