Ideally we should reexec+chdir but I fear it breaks API compatibility.

Can we keep the current API and add an overlayfs-specific chdir hack to https://github.com/containerd/containerd/blob/master/mount/mount_linux.go ?

Can we keep the current API and add an overlayfs-specific chdir hack to https://github.com/containerd/containerd/blob/master/mount/mount_linux.go ?

Agree with the API compatibility point.

I think we can use Type:overlay and Option:chdir opt to hack the chdir, like:

// https://github.com/containerd/containerd/blob/master/snapshots/overlay/overlay.go#L492
options = append(options, fmt.Sprintf("lowerdir=%s", strings.Join(parentPaths, ":")))
+ options = append(options, fmt.Sprintf("chdir=%s", filepath.Join(o.root, "snapshot"))

// https://github.com/containerd/containerd/blob/master/mount/mount_linux.go
check the option contains chdir or not

Doable but hacker. 😄

I think we can use Type:overlay and Option:chdir opt to hack the chdir, like:

That would make containerd v1.1 client unaccessible to containerd v1.2 daemon.
Can we let mount_linux.go auto-detect chdir path from options values?

I guess it'll be a while before it is generally available enough for us to rely on, but it seems there is work afoot in kernel-land which would remove the 4k limit to mount options: Six (or seven) new system calls for filesystem mounting.

That would make containerd v1.1 client unaccessible to containerd v1.2 daemon.

The client just sends the mount info into diff.apply service and the mount info comes from snapshot service. I think the lower-version client can access the higher-version daemon. Is that correct?

That's true if the client just let the daemon mount(2), but IIUC the client should be also allowed to mount(2) by itself.

Thanks for bringing this up, this was next on my 1.2 TODO of issues to look at

My plan was to implement this functionality entirely in the mount package, and preventing any API facing changes or changes to the snapshotters. Any snapshotter, not just the included overlay snapshotter should be able to return standard overlay mounts without having to add customized mount options. This puts the burden on the mount logic to figure out how to perform that mount based on the current systems limitations. I can think of 3 ways to do, 2 of which you already mentioned.

• Symlink - (1) Find a common prefix amongst the lowerdir values (2) Create a symlink in the /tmp directory to that location (3) Replace the common prefix in lowerdir values. (4) Perform mount
• 2 layer mount - (1) Remove a number of paths from end of lowerdir (2) Perform a read only overlay mount in temp (3) Replace the included lowerdir paths with the single mount (4) repeat process, excluding any directories which are already overlay since only 2 levels of overlay are allow, until the mount can be performed on the system (5) Perform mount
• re-exec - (1) Find common prefix amongst the lowerdir (2) Make lowerdir paths relative to common prefix (3) Execute subprocess (4) Chdir to common directory (5) Perform mount in subprocess

I ordered those by which I think is the easiest/best options. I think reexec is the worst option and would like to avoid it if possible (although did see some code somewhere that was able to successfully perform a real fork in Go). The second option is a bit confusing, but it would allow the deepest support for overlay by leveraging overlays support for having overlay mount in the lowerdir, but NOT if that overlay mount also has a lowerdir in its overlay mount, hence 2 level max. The first option is the fastest and easiest and the only additional systemc alls required is symlink, which is must faster than mounting or execing.

@dmcgowan Thank you for the detailed explanation!

Both Symlink and 2 layer mount are better than re-exec. But they also causes the burden about cleanup the tmpdir or symlinks after mount. In this point, I prefer to re-exec.

However, we don't need to chdir all the time. Skip the compact lowerdir option if the number of bytes of options doesn't hit the limitation. I make PR #2502 about this. how do you think about this?

@fuweid using Docker's re-exec is not an option. We don't pull in pkg from docker/docker/pkg. That being said, that re-exec approach is not really a good design and mostly working around a difficulty in Go around supporting fork. Gvisor seemed to accomplish this though in https://github.com/google/gvisor/tree/797cda301677abc8523d5a2a8d731312cc43bce4/pkg/sentry/platform.

But they also causes the burden about cleanup the tmpdir or symlinks after mount.

The mount package would also be responsible for taking care of this

@dmcgowan I will try the Gvisor way. Thank you!

Isn't the need for rexec pretty much eliminated by the changes to runtime.LockOSThread in GO v1.10? See second paragraph of Runtime in the go1.10 release notes.

It's now safe to runtime.LockOSThtread and then Chdir or flip namespace as much as you like, that OS thread is now tainted and will never be used for another goroutine after the current one exits.

It's now safe to runtime.LockOSThtread and then Chdir

There is still no way to prevent the os.Chdir from having an effect on the whole process. However if there is a way to call http://man7.org/linux/man-pages/man2/clone.2.html without CLONE_FS, that is probably the best option.

Thanks @dmcgowan , @ijc and @AkihiroSuda for the help!

There is still a limit of number of layers around 260 according to my test.