Add `os.features` support for EROFS native container images by hsiangkao · Pull Request #13091 · containerd/containerd

hsiangkao · 2026-03-23T02:55:10Z

~~depends on #13080~~
supercedes #12784

Note that users still need to explicitly specify the EROFS snapshotter in order to run the EROFS native images by design; it only improves the user experence of the unpacking process

First, it enhances the transfer service: If no snapshotter is specified and os.features contains "erofs", unpacking should use the EROFS snapshotter and differ.

Second, if no snapshotter is specified, container run selects the default snapshotter. However, if os.features is set, we should always call checkSnapshotterSupport() so that containerd clients can report a clear error instead of the confusing layer extraction error out of overlayfs snapshotter.

Tested by the ubuntu-22.04 multi-manifest image ("linux/amd64" and "linux(+erofs)/amd64"):
ctr i pull --platform="linux(+erofs)" docker.io/hsiangkao/ubuntu:22.04-platforms
and
ctr run --platform="linux(+erofs)/amd64" --tty docker.io/hsiangkao/ubuntu:22.04-platforms test
and
ctr run --platform="linux(+erofs)/amd64" --snapshotter erofs --tty docker.io/hsiangkao/ubuntu:22.04-platforms test

Screenshot (docker.1ms.run is a connectable mirror of docker.io):

Add os.features support for EROFS native container images

Kern-- · 2026-03-23T17:34:09Z

-		}
+
+	if err := i.checkSnapshotterSupport(ctx, snapshotterName, manifest,
+		config.CheckPlatformSupported); err != nil {


It might be worth adding a note on UnpackConfig that containerd always checks snapshotter support if the image platform requires specific OSFeatures.

Kern-- · 2026-03-23T17:37:31Z

After thinking about this a bit more - this makes OSFeatures equivalent to "SnapshotterFeatures", doesn't it? Is there a use-case where an image would require a non-default/non-standard feature that the snapshotters wouldn't know or care about?

AkihiroSuda · 2026-03-23T20:10:51Z

+// Use EROFS snapshotter to extract EROFS native images
+func erofsPlatformSpec() specs.Platform {
+	p := platforms.DefaultSpec()
+	p.OSFeatures = []string{"erofs"}


Should be defined in https://github.com/opencontainers/image-spec/blob/v1.1.1/config.md?plain=1#L119-L124 ?

This was previously discussed in some OCI calls. The possible values for this field are not fully defined and its up to us as a runtime and user of the spec to experiment, validate, and propose changes to the spec. That won't happen unless there is successful adoption using this field.

hi @AkihiroSuda , as @dmcgowan said, yes and it's always a chicken-and-egg issue, but this PR is useful for user experence and multiple manifest images (e.g. one for OCI & one for erofs) for image publishers.

hsiangkao · 2026-03-23T23:03:23Z

After thinking about this a bit more - this makes OSFeatures equivalent to "SnapshotterFeatures", doesn't it? Is there a use-case where an image would require a non-default/non-standard feature that the snapshotters wouldn't know or care about?

@Kern-- I think it might have, but currently we only have one (+erofs) for now; I think eventually we should differ from snapshotter features and other features.
But first any of this should be known by the containerd itself, or we should just refuse to pull/mount unknown features since they could be incompatible and unsupported by this os (for example gpu but without any gpu; all unknown features should be refused since all features should be considered as incompatible, including unknown features); so for now it doesn't matter; and yeah, I don't think we should accept arbitary os.features now.
Anyway, we could define another set of compatible os.features too in the future, but should with another format or prefix for example, and they should be filtered out and identified in other ways instead.

@dmcgowan should we define some prefix which should be considered as incompatible features now, and ignore the random os.features without any prefixes in platforms. I'm not sure if the current users already keep garbage data in os.features now, so we need to add some customized os.feature prefix to identify our concerned features in platforms, otherwise it might cause regressions that the previous images with garbaged os.feature refuse to mount by the new version.

dmcgowan · 2026-03-23T23:25:41Z

After thinking about this a bit more - this makes OSFeatures equivalent to "SnapshotterFeatures", doesn't it? Is there a use-case where an image would require a non-default/non-standard feature that the snapshotters wouldn't know or care about?

Its more an image defines what it needs, the unpack configuration defines what the snapshotter provides. The feature could be provided by something other than a snapshotter, that is just the initial use case. Ultimately, it is the unpacker that has to make the decision about what is the best matching image for the current runtime configuration.

should we define some prefix which should be considered as incompatible features now

I don't think that is necessary right now, the feature is unambiguous and unlikely that the number of supported features would increase drastically.

dmcgowan · 2026-03-23T23:45:47Z

@hsiangkao the previous PR was merged, can you rebase

hsiangkao · 2026-03-24T00:13:38Z

@hsiangkao the previous PR was merged, can you rebase

done, thanks for reminder.

Kern-- · 2026-03-24T04:44:43Z

Its more an image defines what it needs, the unpack configuration defines what the snapshotter provides. The feature could be provided by something other than a snapshotter, that is just the initial use case. Ultimately, it is the unpacker that has to make the decision about what is the best matching image for the current runtime configuration.

This makes sense, but this current implementation requires snapshotters to be aware of every feature or it will be rejected by default. I suppose this would fail loudly and could be addressed in the future once there's a use case.

hsiangkao · 2026-03-24T05:53:25Z

Its more an image defines what it needs, the unpack configuration defines what the snapshotter provides. The feature could be provided by something other than a snapshotter, that is just the initial use case. Ultimately, it is the unpacker that has to make the decision about what is the best matching image for the current runtime configuration.

This makes sense, but this current implementation requires snapshotters to be aware of every feature or it will be rejected by default. I suppose this would fail loudly and could be addressed in the future once there's a use case.

yes, I think if there is another non-snapshotter feature, we could address it then; no real difference for now.

cpuguy83 · 2026-03-24T17:48:10Z

Is this expected?
In this case it fetched both manifests (and configs).
It also failed to unpack, I'm guessing because it tried to use overlay?

❯ ctr images pull --platform="linux(+erofs)/amd64" docker.io/hsiangkao/ubuntu:22.04-platforms
docker.io/hsiangkao/ubuntu:22.04 platfor        fetching image content
└──index (eaee1eccf13b)                         complete        |++++++++++++++++++++++++++++++++++++++|
docker.io/hsiangkao/ubuntu:22.04 platfor        fetching image content
└──index (eaee1eccf13b)                         complete        |++++++++++++++++++++++++++++++++++++++|
   ├──manifest (06ae2788312e)                   complete        |++++++++++++++++++++++++++++++++++++++|
   │  ├──config (d46432b67d0a)                  complete        |++++++++++++++++++++++++++++++++++++++|
   │  └──layer (4d7f4447d249)                   extracting      |--------------------------------------|           0.0 B/43.2 MiB
   └──manifest (1c4cc37c10c4)                   complete        |++++++++++++++++++++++++++++++++++++++|
      └──config (9fa3e2b5204f)                  complete        |++++++++++++++++++++++++++++++++++++++|
application/vnd.oci.image.index.v1+json sha256:eaee1eccf13bc5234a49d439ef25b4664e7f653330b3837501db36476fbd1bbc                                                                               Pulling from OCI Registry (docker.io/hsiangkao/ubuntu:22.04-platforms)  elapsed: 2.2 s  total:  5.8 Ki  (2.6 KiB/s)
ctr: failed to extract layer (application/vnd.erofs.layer.v1 sha256:4d7f4447d249df0fee2efde9eada7e305cd2bb5f06758338fd19ba8e37edc1a7) to overlayfs as "extract-371010066-gzXq sha256:4d7f4447d249df0fee2efde9eada7e305cd2bb5f06758338fd19ba8e37edc1a7": failed to get stream processor for application/vnd.erofs.layer.v1: no processor for media-type

Signed-off-by: Gao Xiang <[email protected]>

If no snapshotter is specified and `os.features` contains "erofs", unpacking should use the EROFS snapshotter and differ. This enhances the usability of native EROFS container images. Signed-off-by: Gao Xiang <[email protected]>

Just use apitypes.OCIPlatformFromProto(). Suggested-by: Jin Dong <[email protected]> Signed-off-by: Gao Xiang <[email protected]>

…e` is set If no snapshotter is specified, container run selects the default snapshotter. However, if `os.features` is set, we should always call `checkSnapshotterSupport()`. This ensures containerd clients report a clear error: ``` ctr: snapshotter overlayfs does not support platform {amd64 linux [erofs] } for image sha256:[] ``` instead of the confusing layer extraction error: ``` ctr: apply layer error for "": failed to extract layer sha256:[]: failed to get stream processor for application/vnd.erofs.layer.v1: no processor for media-type ``` Signed-off-by: Gao Xiang <[email protected]>

hsiangkao · 2026-04-01T05:22:59Z

Is this expected? In this case it fetched both manifests (and configs). It also failed to unpack, I'm guessing because it tried to use overlay?

Now fixed as using platforms.OnlyOS() and it works.

hsiangkao · 2026-04-01T05:23:24Z

@dmcgowan @fuweid @cpuguy83 could help review this again? thanks!

cpuguy83 · 2026-04-01T14:54:01Z

Do we know why the release CI triggered?

hsiangkao · 2026-04-01T14:56:32Z

Do we know why the release CI triggered?

... Just random flakiness, retry again.

hsiangkao · 2026-04-01T21:59:33Z

@containerd/committers could we get it merged? It's useful for end users.

github-project-automation Bot added this to Pull Request Review Mar 23, 2026

github-project-automation Bot moved this to Needs Triage in Pull Request Review Mar 23, 2026

k8s-ci-robot added the size/XL label Mar 23, 2026

dosubot Bot added area/distribution Image Distribution kind/feature labels Mar 23, 2026

hsiangkao requested review from dmcgowan and fuweid March 23, 2026 02:55

hsiangkao force-pushed the os_features branch 5 times, most recently from c9d565c to fe51250 Compare March 23, 2026 05:58

dmcgowan reviewed Mar 23, 2026

View reviewed changes

Comment thread plugins/transfer/plugin.go Outdated

hsiangkao force-pushed the os_features branch from fe51250 to 09ee6eb Compare March 23, 2026 06:50

dmcgowan added this to the 2.3 milestone Mar 23, 2026

dmcgowan moved this from Needs Triage to Needs Reviewers in Pull Request Review Mar 23, 2026

Kern-- approved these changes Mar 23, 2026

View reviewed changes

Kern-- reviewed Mar 23, 2026

View reviewed changes

AkihiroSuda reviewed Mar 23, 2026

View reviewed changes

hsiangkao force-pushed the os_features branch from 09ee6eb to c4d4f6a Compare March 24, 2026 00:12

hsiangkao force-pushed the os_features branch from c4d4f6a to 0dd2053 Compare March 24, 2026 00:17

hsiangkao requested a review from cpuguy83 March 24, 2026 01:38

hsiangkao added 4 commits April 1, 2026 13:13

api: add os_features to api/types/platform.proto

146930e

Signed-off-by: Gao Xiang <[email protected]>

client: remove toPlatforms()

f8367b8

Just use apitypes.OCIPlatformFromProto(). Suggested-by: Jin Dong <[email protected]> Signed-off-by: Gao Xiang <[email protected]>

hsiangkao force-pushed the os_features branch from 123245a to 9400764 Compare April 1, 2026 05:18

cpuguy83 approved these changes Apr 1, 2026

View reviewed changes

github-project-automation Bot moved this from Needs Reviewers to Review In Progress in Pull Request Review Apr 1, 2026

dmcgowan approved these changes Apr 1, 2026

View reviewed changes

fuweid approved these changes Apr 1, 2026

View reviewed changes

fuweid added this pull request to the merge queue Apr 1, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 1, 2026

fuweid added this pull request to the merge queue Apr 1, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 1, 2026

fuweid added this pull request to the merge queue Apr 1, 2026

Merged via the queue into containerd:main with commit 1f0b0c4 Apr 1, 2026
57 of 63 checks passed

github-project-automation Bot moved this from Review In Progress to Done in Pull Request Review Apr 1, 2026

hsiangkao deleted the os_features branch April 2, 2026 00:51

hsiangkao mentioned this pull request Apr 8, 2026

EROFS Support and Improvements #11340

Open

10 tasks

samuelkarp added impact/changelog area/snapshotters Snapshotters labels Apr 29, 2026

dmcgowan mentioned this pull request Apr 29, 2026

Prepare release notes for api/v1.11.0 #13322

Merged

samuelkarp removed the area/snapshotters Snapshotters label Apr 29, 2026

dmcgowan mentioned this pull request Apr 30, 2026

Prepare release notes for v2.3.0 #13325

Merged

BrewTestBot mentioned this pull request Apr 30, 2026

containerd 2.3.0 Homebrew/homebrew-core#280287

Merged

Conversation

hsiangkao commented Mar 23, 2026 • edited by samuelkarp Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Kern-- Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Kern-- commented Mar 23, 2026

Uh oh!

AkihiroSuda Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

dmcgowan Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

hsiangkao Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

hsiangkao commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dmcgowan commented Mar 23, 2026

Uh oh!

dmcgowan commented Mar 23, 2026

Uh oh!

hsiangkao commented Mar 24, 2026

Uh oh!

Kern-- commented Mar 24, 2026

Uh oh!

hsiangkao commented Mar 24, 2026

Uh oh!

cpuguy83 commented Mar 24, 2026

Uh oh!

hsiangkao commented Apr 1, 2026

Uh oh!

hsiangkao commented Apr 1, 2026

Uh oh!

cpuguy83 commented Apr 1, 2026

Uh oh!

hsiangkao commented Apr 1, 2026

Uh oh!

hsiangkao commented Apr 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

hsiangkao commented Mar 23, 2026 •

edited by samuelkarp

Loading

hsiangkao commented Mar 23, 2026 •

edited

Loading