Erofs snapshotter and differ #10705

hsiangkao · 2024-09-18T16:28:00Z

Implements #9361

Sorry about my limited Go skill, this version refines the previous @cardyok version (I revised it in my spare time),
I think it's clean for upstreaming.

Add an initial support for EROFS snapshotter and EROFS differ;
Support file-backed mounts which is released in Linux 6.12 https://www.phoronix.com/news/EROFS-Linux-6.12;

In order to try this:

Install erofs-utils v1.7.x or v1.8.2+ (e.g. erofs-utils 1.8 and 1.8.1 has a regression.);
modprobe erofs with Linux 5.16 or higher (for now);
configure the proper differ:

[plugins]
  [plugins."io.containerd.service.v1.diff-service"]
    default = ["erofs","walking"]

nerdctl --snapshotter=erofs run -it wordpress:latest

Some preliminary performance numbers: #10705 (comment)

TODOs:

code improvements and cleanups;
support merged fsmeta layer to avoid too many unnecessary EROFS mounts;
chattr +i and fsverity support to fully protect layer blobs from data corruption;
parallel unpacking after a proper way is found(also see: Parallel Container Layer Unpacking #8881): In principle, erofs can support parallel unpacking in a more reliable way since each container layer each erofs blob instead of each container layer each overlay dir;
other alternative form such as composefs, etc.;
some further lazy pulling enhancements (needs more extra work) discussed in Container Plumbing Days OCI meeting in April.

docs/snapshotters/README.md

hsiangkao · 2024-09-23T11:06:09Z

Some preliminary performance numbers comparing erofs and overlayfs snapshotter with the following config, both are without parallel unpacking:

version = 2

[plugins]
  [plugins."io.containerd.service.v1.diff-service"]
    default = ["erofs","walking"]

  [plugins."io.containerd.differ.v1.erofs"]
    mkfs_options = ["--sort=none"]

kernel version: Linux iZt4n84m6mwq9w1gcpsbfvZ 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux

# nerdctl image pull --snapshotter=overlayfs --unpack="false" docker.io/library/wordpress:latest
# time nerdctl container --snapshotter=overlayfs run -d docker.io/library/wordpress:latest

real 0m5.654s
user 0m0.090s
sys 0m0.013s

# nerdctl image pull --snapshotter=erofs --unpack="false" docker.io/library/wordpress:latest
# time nerdctl container --snapshotter=erofs run -d docker.io/library/wordpress:latest

real 0m4.838s
user 0m0.083s
sys 0m0.023s

and ctr image pull results

# ctr image pull --snapshotter=erofs --local docker.io/library/wordpress:latest

...
unpacking linux/amd64 sha256:a46a4a00e427168b92df0c792d0ab766bd1d4f111aaaae45fd0173ece590b6f3...
done: 4.604676715s

# ctr image pull --snapshotter=overlayfs --local docker.io/library/wordpress:latest

...
unpacking linux/amd64 sha256:a46a4a00e427168b92df0c792d0ab766bd1d4f111aaaae45fd0173ece590b6f3...
done: 5.443237423s

In this way, fsmeta merge feature is disabled as below and loop devices are still used (not the latest kernel):

ktock

Tests will be needed.

plugins/snapshots/erofs/loop.go

plugins/snapshots/erofs/plugin/plugin.go

plugins/snapshots/erofs/loop.go

ktock · 2024-09-24T20:11:19Z

plugins/diff/erofs/differ.go

+		return "", fmt.Errorf("invalid filesystem type for erofs differ: %w", errdefs.ErrNotImplemented)
+	}
+	// If the layer is not prepared by the EROFS snapshotter, fall back to the next differ
+	if _, err := os.Stat(filepath.Join(layer, ".erofslayer")); err != nil {


This looks difficult to maintain... I think applier shouldn't be aware of internal directory structure of a snapshotter.

I think applier shouldn't be aware of internal directory structure of a snapshotter.

For generic differs I think, yes. But this differ is only useful together with erofs snapshotter.
All cases are as below:
erofs snapshot layer + erofs differ : convert OCI media types (or erofs native blobs later)
erofs snapshot layer + other differs (walking for example) : just unpack to fs and behaves as overlay;
other snapshot layer (e.g. overlay or remote snapshotter) + erofs differ: move to the next differ;
other snapshot layer + other differ : don't care.

This looks difficult to maintain...

Why? Basically it needs a way to identify a layer prepared by erofs snapshotter to ensure the safety.
Do you have better ideas for this requirement?..
I've checked the previous windows differs, but they seems they just use a special mount structure. But I think this method is more clean compared to that since the mount structure can be mounted directly (so that erofs snapshotter can be used with walking differ) rather than a difficult-to-identify mount structure...

...

This looks difficult to maintain...

Why? Basically it needs a way to identify a layer prepared by erofs snapshotter to ensure the safety. Do you have better ideas for this requirement?.. I've checked the previous windows differs, but they seems they just use a special mount structure. But I think this method is more clean compared to that since the mount structure can be mounted directly (so that erofs snapshotter can be used with walking differ) rather than a difficult-to-identify mount structure...

Derek has an ongoing mount manager which could help us to clean up this with new mount[] structure too, I think after this work lands, EROFS snapshotter could have a more clean way then.
But I guess they could be done in parallel? I do hope if you could agree..

plugins/diff/erofs/differ.go

hsiangkao · 2024-09-24T23:56:23Z

Tests will be needed.

I will try to add tests to run erofs snapshotter later.

hsiangkao · 2024-12-18T03:48:35Z

Looks good, but //go:build linux line should be dropped and _linux.go suffix should be used instead

Hi @AkihiroSuda, sorry for my Go skills again, I've revised a new version and thanks for your suggestion!

docs/snapshotters/erofs.md

plugins/diff/erofs/plugin/plugin_linux.go

plugins/snapshots/erofs/erofs_linux.go

djdongjin

LGTM

AkihiroSuda · 2025-01-09T05:19:25Z

@fuweid @dmcgowan LGTY?

hsiangkao · 2025-01-09T05:41:38Z

@fuweid @dmcgowan LGTY?

Hi @AkihiroSuda, thanks for pinging. I could summerize the benefits of erofs snapshotter here again (maybe update & refine):

It doesn't need to mount each layer individually compared to SquashFS, EXT4, etc. In other words, only one mount is required for each container image instead of one mount per layer (otherwise, e.g., Wordpress could have 20+ layers → 20+ mounts). For secure containers, it's also beneficial to pass container images with a single virtio-blk to VM rather than 20+ virtio-blks.
It doesn't need loop devices to mount compared with other kernel filesystems since Linux 6.12 LTS.
Compared to EXT4, XFS and BTRFS, etc., EROFS doesn't need to estimate the total space and reserve in advance to keep container image data.
Compared to SquashFS, EROFS natively supports AUFS-style whiteout/opaque, which is useful for OCI layered images.
Compared to SquashFS, EROFS supports many advanced features like FSDAX to eliminate guest page cache for secure containers.

I do hope the kernel features above could be fully leveraged in a dedicated way for container ecosystem, that is why I worked on this. I do hope it can be upstreamed into containerd since it has been pending for almost two years since it was proposed.

fuweid · 2025-01-09T17:03:38Z

Sorry for late reply. It's on my list. Will take a look in this week.

dmcgowan

I would like to see the first version simplified a bit. If we have the merging implementation working, we should stick with that for the first version and add the overlayed erofs layers later on. We could also avoid mixing erofs layers and non-erofs layers by converting on Commit.

Let me know if that really won't work well or the merging is not the preferred solution. We still have time to make improvements and add features to this before the next release.

dmcgowan · 2025-01-09T18:49:45Z

plugins/snapshots/erofs/erofs_linux.go

+
+		_, imagePull = info.Labels[targetSnapshotLabel]
+		if !imagePull && s.mergeFsMeta {
+			s.GenerateFsMeta(ctx, snap)


Why not just do this on commit? Create snapshot should be very fast. Also this could be done outside the transaction then renamed inside.

Hi Derek, thanks for the suggestion.
The merged fsmeta generation is triggered by the non-unpacking snapshot preparation ("containerd.io/snapshot.ref").
Actually we don't want to generate merged fsmeta on every commited layer (otherwise it will impact performance, I don't want to generate fsmeta all the time), but only for all the parent layers if some layer is not using for unpacking. And fsmeta is kept into the runable layer, not every commited layers.

dmcgowan · 2025-01-09T18:50:47Z

plugins/snapshots/erofs/erofs_linux.go

+	}}, nil
+}
+
+// generate a metadata-only EROFS fsmeta.erofs if the parent layers are all EROFS blobs


Allowing some parent layers to not be EROFS blobs makes this snapshotter more complicated.

dmcgowan · 2025-01-09T18:52:30Z

plugins/snapshots/erofs/erofs_linux.go

+}
+
+func (s *snapshotter) Commit(ctx context.Context, name, key string, opts ...snapshots.Opt) error {
+	return s.ms.WithTransaction(ctx, true, func(ctx context.Context) error {


I think before we enter this transaction is the best time to generate the fsmeta blob or convert the flat directory to erofs (to avoid having a mix of erofs and non-erofs snapshots).

I've explained why generating the fsmeta blob here is ineffiencient above.
I think it's reasonable to convert the flat directory to erofs, let me try this.

dmcgowan · 2025-01-09T18:55:52Z

plugins/snapshots/erofs/erofs_linux.go

+	ovlOptions []string
+
+	// If fsmerge feature is enabled or not
+	mergeFsmeta bool


What is the performance like for this? In the initial version I think it would be best to have this as the default and possibly not even include the loopback version yet.

It's mainly used to avoid too many erofs layered mounts and used for secure container virtio-blk passthrough. In other words, it's mainly be used as an advanced feature for now.
If fsmeta merging generation is stayed into Prepare(), it will almost the same performance number against mount one-by-one;
If fsmeta merging generation is moved into Commit(), it will be performed bad against mount one-by-one.

I could reconfirm the exact performance number, but if you think fsmerge feature is still somewhat unclean, I could just drop this feature for now and only leave a very basic version without fsmerge feature.

hsiangkao · 2025-01-10T00:23:52Z

I would like to see the first version simplified a bit. If we have the merging implementation working, we should stick with that for the first version and add the overlayed erofs layers later on. We could also avoid mixing erofs layers and non-erofs layers by converting on Commit.

Yes, it's quite a clever idea to convert non-erofs layer to erofs layer on Commit (although non-erofs layer conversion will make erofs snapshotter together with the walking differ performs longer than the overlayfs snapshotter, but I could see some benefit is that mixing overlay dir is bad for VM passthrough use cases...)
But generating fsmerge on Commit as well is unacceptable for users, if you think generating fsmeta on Prepare() is unclean, I tend to drop this "fsmerge" feature for now (thus loopback device handling in the current erofs snapshotter codebase is no longer needed either.) I'm pretty fine with that.

Let me know if that really won't work well or the merging is not the preferred solution.

Although "merging is not the preferred solution" but it needs to be generated for Active Snapshot for running containers later, rather than snapshots for unpacking.

We still have time to make improvements and add features to this before the next release.

Yeah, agreed, thanks!

The EROFS differ only applies to EROFS layers which are marked by a special file `.erofslayer` generated by the EROFS snapshotter. Why it's needed? Since we'd like to parse []mount.Mount directly without actual mounting and convert OCI layers into EROFS blobs, `.erofslayer` gives a hint that the active snapshotter supports the output blob generated by the EROFS differ. I'd suggest it could be read together with the next commit. Signed-off-by: cardy.tang <[email protected]> Signed-off-by: Gao Xiang <[email protected]>

It allows us to mount each EROFS blob layer (generated by the EROFS differ) independently, or use the "unpacked" fs/ directories (if some other differ is used.) Currently, it's somewhat like the overlay snapshotter, but I tend to separate the new EROFS logic into a self-contained component, rather than keeping it tangled in the very beginning. Existing users who use the overlay snapshotter won't be impacted at all but they have a chance to use this new snapshotter to leverage the EROFS filesystem. Signed-off-by: cardy.tang <[email protected]> Signed-off-by: Gao Xiang <[email protected]>

Signed-off-by: Gao Xiang <[email protected]>

Some basic tests for now. Signed-off-by: Gao Xiang <[email protected]>

rchincha · 2025-01-14T05:08:50Z

Just a general question, is this snapshotter planned to be in-tree?

hsiangkao · 2025-01-14T05:32:54Z

Just a general question, is this snapshotter planned to be in-tree?

Yes. At least that is the plan on my side, which will provide EROFS kernel features to users much like other in-tree snapshotters.

mythi

I'd be interested in this from Kata/CoCo perspective. I did an initial check of the PR and test runs with runc.

mythi · 2025-01-31T14:41:45Z

docs/snapshotters/erofs.md

+
+```
+  [plugins."io.containerd.service.v1.diff-service"]
+    default = ["erofs","walking"]


I'm seeing this warning in the logs:

WARN[2025-01-31T14:27:36.659539529+02:00] multiple differs match for platform, set `differ` option to choose, skipping "walking"

it seems it relates to transfer service, but I don't really know the details too.
EROFS snapshotter can only work effectively with the related differ, so just
leaving "default = ["erofs"]" is fine in practice.

mythi · 2025-01-31T14:43:52Z

docs/snapshotters/erofs.md

+in the very beginning, it'd be better to be left as an independent snapshotter
+so that existing overlayfs users won't be impacted by the new behaviors and
+users could have a chance to try and develop the related ecosystems (such as
+ComposeFS, confidential containers, gVisor, Kata, gVisor, and more) together.


I'm interested in the "confidential containers" case. How the guest would know the snapshotter prepared everything correctly (and the image integrity is preserved)?

I think some changes should be made in the kata-shim, currently I don't have enough time to prepare a optimized workable implementation for this, but after this lands, I will try to sort out these.

mythi · 2025-01-31T14:45:33Z

docs/snapshotters/erofs.md

+
+ - Native EROFS layers can be pulled from registries without conversion.
+
+For VM containers, the EROFS snapshotter can efficiently pass through and share


Given the VM based use-case, it feels to me this has some overlaps with the blockfile snapshotter too.

Is there currently a way to get this tried out with Kata based setup?

in principle, the blockfile approach is ineffective if underlay fs doesn't support reflink.
Also I'd like to use vhost-user protocol (+ a merged flatten device) to pass through the container image for VM-based containers, rather than generating some loopback device on the host, which is ineffective too.

Is there currently a way to get this tried out with Kata based setup?

No, as I said above. I will not work out any further since I don't have enough time (and it's also useless) to complete all the use cases before upstreaming.
Personally I think it's a very effective approach for VM-based image pass through. but currently I don't really have a promising reason to persuade our development dept. to put more development resource on it, esp. before upstreaming.

mythi · 2025-01-31T14:50:04Z

docs/snapshotters/erofs.md

+In order to leverage EROFS-formatted blobs, the EROFS differ is needed to be
+used together to apply image layers.  Otherwise, the EROFS snapshotter will


AFAUI, this handles the whiteouts also. perhaps add a note about it (and how it's done (in mkfs.erofs?).

AFAUI, this handles the whiteouts also. perhaps add a note about it (and how it's done (in mkfs.erofs?).

I could try to mention it a bit, but that is an implementation detail, since it's a real necessary part to support OCI images. --aufs option in mkfs.erofs will handle AUFS-like whiteouts and opaques properly.

dmcgowan

Let's get this in, there is time to test and iterate on it before the 2.1 release

k8s-ci-robot added the size/XL label Sep 18, 2024

dosubot bot added snapshotters kind/feature labels Sep 18, 2024

hsiangkao mentioned this pull request Sep 18, 2024

[Feat] erofs snapshotter and differ #9362

Closed

hsiangkao force-pushed the erofs-snapshotter branch 10 times, most recently from 7b9dbc7 to ccc50bf Compare September 19, 2024 04:41

k8s-ci-robot added size/XXL and removed size/XL labels Sep 19, 2024

hsiangkao force-pushed the erofs-snapshotter branch 2 times, most recently from f57841a to e9eed08 Compare September 19, 2024 04:47

AkihiroSuda reviewed Sep 19, 2024

View reviewed changes

docs/snapshotters/README.md Show resolved Hide resolved

hsiangkao force-pushed the erofs-snapshotter branch from e9eed08 to 879e392 Compare September 23, 2024 11:03

hsiangkao force-pushed the erofs-snapshotter branch 2 times, most recently from 626a4ca to ec82d81 Compare September 23, 2024 12:14

AkihiroSuda requested review from dmcgowan and ktock September 23, 2024 14:51

ktock reviewed Sep 24, 2024

View reviewed changes

hsiangkao force-pushed the erofs-snapshotter branch 3 times, most recently from 03fc459 to 0c2285b Compare September 26, 2024 07:47

djdongjin reviewed Dec 18, 2024

View reviewed changes

hsiangkao force-pushed the erofs-snapshotter branch from 8cf5146 to 1dae4f0 Compare December 18, 2024 05:30

djdongjin approved these changes Dec 18, 2024

View reviewed changes

ruiwen-zhao approved these changes Dec 18, 2024

View reviewed changes

fuweid self-requested a review December 18, 2024 18:00

dmcgowan requested changes Jan 9, 2025

View reviewed changes

hsiangkao force-pushed the erofs-snapshotter branch from 1dae4f0 to 66f5a49 Compare January 13, 2025 08:20

hsiangkao added 3 commits January 13, 2025 16:31

Add EROFS snapshotter documentation

fd4caef

Signed-off-by: Gao Xiang <[email protected]>

Add tests for EROFS snapshotter

2f15d65

Some basic tests for now. Signed-off-by: Gao Xiang <[email protected]>

hsiangkao force-pushed the erofs-snapshotter branch from 66f5a49 to 2f15d65 Compare January 13, 2025 08:31

dmcgowan added this to the 2.1 milestone Jan 22, 2025

mythi reviewed Jan 31, 2025

View reviewed changes

dmcgowan approved these changes Feb 5, 2025

View reviewed changes

dmcgowan added this pull request to the merge queue Feb 5, 2025

Merged via the queue into containerd:main with commit 59c8cf6 Feb 5, 2025
58 checks passed

hsiangkao mentioned this pull request Feb 6, 2025

EROFS snapshotter & differ #9361

Closed

dmcgowan mentioned this pull request Feb 6, 2025

EROFS Support and Improvements #11340

Open

7 tasks

dmcgowan changed the title ~~[Feat] erofs snapshotter and differ~~ Erofs snapshotter and differ Mar 8, 2025

dmcgowan added the impact/changelog label Mar 8, 2025

dmcgowan mentioned this pull request Mar 8, 2025

Prepare release notes for v2.1.0-beta.0 #11510

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Erofs snapshotter and differ #10705

Erofs snapshotter and differ #10705

hsiangkao commented Sep 18, 2024 •

edited

Loading

hsiangkao commented Sep 23, 2024 •

edited

Loading

ktock left a comment

ktock Sep 24, 2024

hsiangkao Sep 24, 2024 •

edited

Loading

hsiangkao Dec 18, 2024

hsiangkao commented Sep 24, 2024

hsiangkao commented Dec 18, 2024

djdongjin left a comment

AkihiroSuda commented Jan 9, 2025

hsiangkao commented Jan 9, 2025 •

edited

Loading

fuweid commented Jan 9, 2025

dmcgowan left a comment

dmcgowan Jan 9, 2025

hsiangkao Jan 10, 2025

dmcgowan Jan 9, 2025

dmcgowan Jan 9, 2025

hsiangkao Jan 10, 2025

dmcgowan Jan 9, 2025

hsiangkao Jan 10, 2025

hsiangkao commented Jan 10, 2025 •

edited

Loading

rchincha commented Jan 14, 2025

hsiangkao commented Jan 14, 2025 •

edited

Loading

mythi left a comment

mythi Jan 31, 2025

hsiangkao Feb 4, 2025

mythi Jan 31, 2025

hsiangkao Feb 4, 2025

mythi Jan 31, 2025

hsiangkao Feb 4, 2025 •

edited

Loading

mythi Jan 31, 2025

hsiangkao Feb 4, 2025

dmcgowan left a comment


		- Native EROFS layers can be pulled from registries without conversion.

		For VM containers, the EROFS snapshotter can efficiently pass through and share

		In order to leverage EROFS-formatted blobs, the EROFS differ is needed to be
		used together to apply image layers. Otherwise, the EROFS snapshotter will

Erofs snapshotter and differ #10705

Erofs snapshotter and differ #10705

Conversation

hsiangkao commented Sep 18, 2024 • edited Loading

hsiangkao commented Sep 23, 2024 • edited Loading

ktock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hsiangkao Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hsiangkao commented Sep 24, 2024

hsiangkao commented Dec 18, 2024

djdongjin left a comment

Choose a reason for hiding this comment

AkihiroSuda commented Jan 9, 2025

hsiangkao commented Jan 9, 2025 • edited Loading

fuweid commented Jan 9, 2025

dmcgowan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hsiangkao commented Jan 10, 2025 • edited Loading

rchincha commented Jan 14, 2025

hsiangkao commented Jan 14, 2025 • edited Loading

mythi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hsiangkao Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmcgowan left a comment

Choose a reason for hiding this comment

hsiangkao commented Sep 18, 2024 •

edited

Loading

hsiangkao commented Sep 23, 2024 •

edited

Loading

hsiangkao Sep 24, 2024 •

edited

Loading

hsiangkao commented Jan 9, 2025 •

edited

Loading

hsiangkao commented Jan 10, 2025 •

edited

Loading

hsiangkao commented Jan 14, 2025 •

edited

Loading

hsiangkao Feb 4, 2025 •

edited

Loading