Extend mount package on macOS to use external snapshotters #5935

thehajime · 2021-09-03T01:17:26Z

(splitting from the original patchset: #4526)

This PR tries to support the native snapshotter on macOS. As macOS doesn't have union-fs like feature (nullfs nor aufs), the mount operation is simulated by using symlinks.

The snapshotter passes the test on darwin/amd64 platform.

% uname -a
Darwin sun.local 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:31 PDT 2021; root:xnu-7195.141.2~5/RELEASE_X86_64 x86_64
% pwd
~/.go/src/github.com/containerd/containerd/snapshots/native
% sudo go test -test.root -v .
(snip)
--- PASS: TestNaive (0.00s)
    --- SKIP: TestNaive/Rename (0.08s)
    --- SKIP: TestNaive/ViewReadonly (0.10s)
    --- PASS: TestNaive/RootPermission (0.76s)
    --- PASS: TestNaive/CloseTwice (0.82s)
    --- PASS: TestNaive/StatInWalk (1.44s)
    --- PASS: TestNaive/Remove (2.09s)
    --- PASS: TestNaive/PreareViewFailingtest (2.05s)
    --- PASS: TestNaive/StatActive (1.33s)
    --- PASS: TestNaive/StatComitted (1.40s)
    --- PASS: TestNaive/TransitivityTest (2.25s)
    --- PASS: TestNaive/Walk (2.89s)
    --- PASS: TestNaive/Update (4.28s)
    --- PASS: TestNaive/RemoveDirectoryInLowerLayer (4.28s)
    --- PASS: TestNaive/Basic (4.32s)
    --- PASS: TestNaive/RemoveIntermediateSnapshot (2.27s)
    --- PASS: TestNaive/DeletedFilesInChildSnapshot (4.42s)
    --- PASS: TestNaive/MoveFileFromLowerLayer (3.47s)
    --- PASS: TestNaive/Chown (2.86s)
    --- PASS: TestNaive/DirectoryPermissionOnCommit (3.29s)
    --- PASS: TestNaive/LayerFileupdate (7.12s)
    --- PASS: TestNaive/128LayersMount (22.64s)
PASS
ok      github.com/containerd/containerd/snapshots/native       22.775s

% ./bin/ctr -a /tmp/ctrd/run/containerd/containerd.sock i ls
REF                                        TYPE                                                 DIGEST                                                                  SIZE     PLATFORMS    LABELS
ghcr.io/ukontainer/runu-base:0.3-osx-extra application/vnd.docker.distribution.manifest.v2+json sha256:0a6e9d725a7a50a34dd7cf87991f7fc597b85f4b0ea591bdc5d2762776c33bf7 65.9 MiB darwin/amd64 -

k8s-ci-robot · 2021-09-03T01:17:34Z

Hi @thehajime. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

theopenlab-ci · 2021-09-03T01:43:10Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 25s (non-voting)
containerd-test-arm64 : SUCCESS in 6m 01s (non-voting)
containerd-integration-test-arm64 : SUCCESS in 24m 10s (non-voting)

kzys · 2021-09-03T20:55:46Z

mount/mount_darwin.go

Is it possible to move the symlink hack from the mount package to the native snapshotter?

macOS is basically BSD-flavored Unix and has mount(2). The current implementation may work for the snapshotter but not for other use cases.

Is it possible to move the symlink hack from the mount package to the naive snapshotter?

The hack is intended to emulate bind mount of Linux, and the snapshotter side is not aware of how it is mounted (If I understand correctly).
I understand your comment that Unmount may possibly unmount unintended target (and delete files/directories).

What about adding a check if the target is a symbolic link or not (like below) ?

Suggested change

func Unmount(mount string, flags int) error {

os.RemoveAll(mount)

return os.Mkdir(mount, 0755)

}

// UnmountAll is just a rm of symlink

func UnmountAll(mount string, flags int) error {

os.RemoveAll(mount)

return os.Mkdir(mount, 0755)

}

func Unmount(target string, flags int) error {

if isSymlink(target) {

return unmountSymlink(target, 0755)

}

return errors.Wrapf(unix.EBUSY, "failed to unmount target %s", target)

}

You are right. The snapshotter doesn't call Mount by itself.

I can agree that having the symlink check is safer than the current revision. Can you also handle non-symlink cases?

thanks for the comment,

I can agree that having the symlink check is safer than the current revision. Can you also handle non-symlink cases?

let me try.

i have updated (force-pushed) the branch adding checks on unmount while calling unmount(2) syscall in non-symlink cases. let me know how you think.

no, the updated patch doesn't work.. let me try again.

I have updated again.

because unmount is called multiple times on a single target, like during cleanup process of shim instances, it should handle properly after second unmount (but it wasn't in my previous patch). I fixed it.

theopenlab-ci · 2021-09-10T02:06:01Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 50s (non-voting)
containerd-test-arm64 : SUCCESS in 5m 26s (non-voting)
containerd-integration-test-arm64 : SUCCESS in 24m 55s (non-voting)

theopenlab-ci · 2021-09-10T06:11:35Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 21s (non-voting)
containerd-test-arm64 : SUCCESS in 5m 28s (non-voting)
containerd-integration-test-arm64 : SUCCESS in 24m 04s (non-voting)

kzys · 2021-09-13T19:11:44Z

mount/mount_darwin.go

+// Mount to the provided target path.
+//
+// Use symlink instead of union mount on darwin
+func (m *Mount) Mount(target string) error {


As like Umount, can you handle non-bind mount here as well?

I'm hesitant to handle the non-bind mount case in this Mount call as currently I have no cases to call with that. OTOH, Unmount is called without a filesystem type so, it is safe to check as you suggested (and I updated).

we already have the following check, which I think it is enough for the moment.

if m.Type != "bind" { return errors.Errorf("invalid mount type: '%s'", m.Type) }

AkihiroSuda · 2021-09-15T08:46:50Z

Symlinking is quite different from bind-mounting, so this should be a new snapshotter plugin (in an external repo, perhaps)

thehajime · 2021-09-15T13:18:11Z

Symlinking is quite different from bind-mounting, so this should be a new snapshotter plugin (in an external repo, perhaps)

I agree that symlink != bind-mount.

(Let me know if I'm wrong...)
If I understand correctly, a snapshotter implementation doesn't contain internals of how to mount, so even if we have an external snapshotter using symlinks, we anyway have to update mount/mount_darwin.go.

This is how I interpret the design document (https://github.com/containerd/containerd/blob/main/design/snapshots.md#scope).

What about having new mount type, declared in snapshots/native/default_darwin.go file, and implemented in mounts/mount_darwin.go?

snapshots/native/default_darwin.go

package native

const mountType = "symlink"

var defaultMountOptions = []string{}

mounts/mount_darwin.go

func (m *Mount) Mount(target string) error {
	if m.Type != "symlink" {
		return errors.Errorf("invalid mount type: '%s'", m.Type)
	}

	return m.mountSymlink(target)
}

thehajime · 2021-09-17T22:58:58Z

So no response for a while... (which means no good sign in this direction)

a couple of alternatives in my mind.

Use (non-open sourced) osxfuse + bindfs
Use fs.CopyDir() instead of symlink (this is how vfs graph driver does ?)
Use snapshot feature of filesystem (APFS, OpenZFS) and add new snapshotter (internally or externally): this may be able to mount without bind-like mount in a darwin way. But,
- APFS only allows me to create snapshot for TimeMachine (via tmutil command). We can create an ad-hoc snapshot admin program (like https://github.com/ahl/apfs), but may require to be installed beforehand.
- OpenZFS might be another option, but still require external programs (as FUSE).
create a disk image (or sparse-bundle image, .img or .dmg), prepare a snapshot image (by containerd), and mount it from a container. This might be a solution of non-bind mount on darwin, but might result in a similar conclusion as native snapshotter of inefficiency.. Thus testing purpose only too to this alternative..

To me none of them satisfies. If you have any good directions, I'm looking forward to seeing those.

kzys · 2021-09-17T23:11:43Z

Regarding @AkihiroSuda's comment,

Symlinking is quite different from bind-mounting, so this should be a new snapshotter plugin (in an external repo, perhaps)

As @thehajime pointed out, it would not be possible since Mount is used in the snapshotter's interface.

The forth option (making a disk image) looks most promising from my perspective. We did something similar in firecracker-containerd (see https://github.com/firecracker-microvm/firecracker-containerd/search?q=naive&type=commits).

I'm on the fence regarding Type=symlink. It wouldn't cause production issues, but supporting what OS doesn't support in the mount package doesn't seem right.

thehajime · 2021-09-21T13:27:28Z

The forth option (making a disk image) looks most promising from my perspective. We did something similar in firecracker-containerd (see https://github.com/firecracker-microvm/firecracker-containerd/search?q=naive&type=commits).

Worked on the option (using disk images) for an snapshotter.
New snapshotter, called darwin snapshotter, is backed with sparsebundle image of darwin, which can mount without bind-like mount option from mount package.

Use of sparsebundle (.sparsebundle) instead of disk image (.dmg) is because of its storage efficiency: sparsebundle images don't consume 10GB storage if the contents in image doesn't have much, while .dmg images do.

This snapshotter is not specific to runu runtime, so I thought it's worthwhile to implement as an internal snapshotter.

One of good news is we can remove a limitation of readonly mount, which symlink version cannot emulate. Bad news (it's expected though) is very poor speed.

Each layer preparation invokes hdiutil command to create/mount/detach(unmount) images, which is not cheap like calling syscalls from programs. AFAIK, DiskImage.framework of macOS is not public framework and we cannot call internal function in a stable manner.

--- PASS: TestDarwin (0.00s)
    --- SKIP: TestDarwin/Rename (0.09s)
    --- PASS: TestDarwin/ViewReadonly (9.63s)
    --- PASS: TestDarwin/StatComitted (26.88s)
    --- PASS: TestDarwin/Remove (27.78s)
    --- PASS: TestDarwin/TransitivityTest (32.46s)
    --- PASS: TestDarwin/StatActive (33.01s)
    --- PASS: TestDarwin/Walk (51.75s)
    --- PASS: TestDarwin/RemoveIntermediateSnapshot (25.88s)
    --- PASS: TestDarwin/MoveFileFromLowerLayer (63.81s)
    --- PASS: TestDarwin/RootPermission (21.18s)
    --- PASS: TestDarwin/Basic (63.96s)
    --- PASS: TestDarwin/PreareViewFailingtest (32.37s)
    --- PASS: TestDarwin/CloseTwice (13.53s)
    --- PASS: TestDarwin/Update (38.71s)
    --- PASS: TestDarwin/Chown (66.95s)
    --- PASS: TestDarwin/StatInWalk (3.51s)
    --- PASS: TestDarwin/DeletedFilesInChildSnapshot (75.64s)
    --- PASS: TestDarwin/DirectoryPermissionOnCommit (76.39s)
    --- PASS: TestDarwin/RemoveDirectoryInLowerLayer (81.14s)
    --- PASS: TestDarwin/LayerFileupdate (131.98s)
    --- PASS: TestDarwin/128LayersMount (235.10s)
PASS
ok      github.com/containerd/containerd/snapshots/darwin       286.968s

You can compare this with the (existing) native snapshotter; 286.9sec v.s. 22.7sec, which is pity...

#5935 (comment)

If this direction makes sense, I will update this PR with the new snapshotter patches. Or create new pull request ?
You can take a look at the current version of the patches,

main...ukontainer:feature-darwin-snapshotter

Looking forward to seeing broad opinions on this.

Cc: @kzys @AkihiroSuda

kzys · 2021-09-21T15:32:53Z

Thanks for the patience and the efforts! While the performance is disappointing, this looks more cleaner than extending Mount.

@AkihiroSuda or @containerd/committers Regarding the new snapshotter, are we ready to have that in-tree? This could be an out-of-tree proxy snapshotter (as like stargz snapshotter) and doesn't have to be hosted under containerd org.

thehajime · 2021-09-23T01:42:29Z

I've updated the branch:

with slightly improved the performance using cmdline options of hdiutil (e.g., -noverify, -noautofsck)
enable make root-test to test new snapshotter

(sorry, when I rename the branch, I expected to reflect this PR but accidentally close the PR; thus reopened it.)

I'm open to reorganize this PR; if there are any suggestions, those are more than welcome :)

theopenlab-ci · 2021-09-23T02:03:18Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 18s (non-voting)
containerd-test-arm64 : SUCCESS in 5m 55s (non-voting)
containerd-integration-test-arm64 : FAILURE in 24m 28s (non-voting)

theopenlab-ci · 2021-09-23T09:34:59Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 40s (non-voting)
containerd-test-arm64 : SUCCESS in 6m 00s (non-voting)
containerd-integration-test-arm64 : FAILURE in 26m 35s (non-voting)

AkihiroSuda · 2021-09-27T07:08:07Z

mount/mount_darwin.go

+// Use symlink instead of union mount on darwin
+func (m *Mount) Mount(target string) error {
+	log.L.Debugf("mount: %s, src %s", target, m.Source)
+	if m.Type != "darwin" {


I'm not sure we can reach consensus on adding this pseudo filesystem type.

I think it's better to exec thirdparty fmt.Sprintf("mount_%s", m.Type) binary.

I'm not sure we can reach consensus on adding this pseudo filesystem type.

I think it's not yet. So, this is a proposal.

Though it's a bit different one, windows snapshotter has a mount type 'windows-layer', so I was thinking it's not the first one in containerd.

I think it's better to exec thirdparty fmt.Sprintf("mount_%s", m.Type) binary.

Sorry, I didn't well understand this;
To accept unknown mount type ?

Sorry, I didn't well understand this;
To accept unknown mount type ?

mount_darwin.go should try to execute fmt.Sprintf("mount_%s", m.Type) binary when the binary is present.
This design will support arbitrary thirdparty mount types.

Thanks, now I see your point.

I'm okay to add the code, but do you really think there is a value to support various mount types without any current users ?

Mounts based on symlinks would likely have compatibility issues for runtimes that use chroot jails for filesystem isolation (as is being discussed in #5525), so making this configurable would be nice future-proofing for that use case.

@adamrehn hi, thanks for the input.

I updated the code (and tests) to be extendable for future use cases, based on the suggestion by @AkihiroSuda; it can support arbitrary mount types easily than before (I hope), though I didn't include any types at this moment.

Please take a look at around 4c61e12#diff-5777935f45ef82953bbde8ed1e50a52c43044ea1f1ac4f2042bd791342296711.

btw, mount based on symlinks no longer exists in this patch. I replaced it with a sparsebundle-based snapshotter with darwin-specific mount using hdiutil command.

Oops, I got a little tripped up there because the preview that GitHub shows for this thread still includes an older comment mentioning symlinks despite the code having changed in the meantime. The sparsebundle implementation should be compatible with all use cases, so this will immediately accelerate my progress on #5525. Thanks for your wonderful work on this @thehajime!

@AkihiroSuda How about using mount.#{type} instead of mount_#{type}? Linux is using dot.

@kzys
Mount binaries on macOS 11.6 uses mount_#{type}. Don't know how it starts though.

% uname -a Darwin sun.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64 sun:~/gitworks/ukontainer/nerdctl% ls /sbin/mount_* /sbin/mount_9p* /sbin/mount_cddafs@ /sbin/mount_hfs@ /sbin/mount_tmpfs@ /sbin/mount_acfs@ /sbin/mount_devfs* /sbin/mount_msdos@ /sbin/mount_udf@ /sbin/mount_afp* /sbin/mount_exfat@ /sbin/mount_nfs* /sbin/mount_webdav* /sbin/mount_apfs@ /sbin/mount_fdesc* /sbin/mount_ntfs@ /sbin/mount_cd9660@ /sbin/mount_ftp@ /sbin/mount_smbfs*

theopenlab-ci · 2021-10-03T04:39:52Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 42s (non-voting)
containerd-test-arm64 : SUCCESS in 5m 59s (non-voting)
containerd-integration-test-arm64 : SUCCESS in 24m 33s (non-voting)

theopenlab-ci · 2022-01-19T06:10:53Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 36s (non-voting)
containerd-test-arm64 : SUCCESS in 5m 38s (non-voting)
containerd-integration-test-arm64 : FAILURE in 30m 15s (non-voting)

theopenlab-ci · 2022-01-20T02:47:40Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 36s (non-voting)
containerd-test-arm64 : SUCCESS in 5m 43s (non-voting)
containerd-integration-test-arm64 : SUCCESS in 30m 01s (non-voting)

Darwin has no bind mount-like mount type. So, we implemented an external snapshotter plugin, called darwin sparsebundle snapshotter. This commit supports to mount a snapshot image from the snapshotter. Signed-off-by: Hajime Tazaki <[email protected]>

thehajime · 2022-01-21T00:02:18Z

fixed a bit to handle arrays properly.

theopenlab-ci · 2022-01-21T00:31:47Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 24s (non-voting)
containerd-test-arm64 : SUCCESS in 5m 38s (non-voting)
containerd-integration-test-arm64 : FAILURE in 28m 06s (non-voting)

thehajime · 2022-03-22T23:18:51Z

@dmcgowan @kzys i'd be appreciated if spending some time to review this PR. thanks.

mikedld · 2022-04-11T18:11:17Z

mount/mount_darwin.go

+		return nil
+	}
+
+	return unmountWithHelper("mount_containerd_darwin", target, flags)


Just passing by. Seems a bit odd that we use type-specific binary for mount, but a generic (?) "darwin" binary for unmount. What if my mount_containerd_foobar helper does something weird that's not actually mounting anything, in which case mount_containerd_darwin wouldn't know know to unmount it? I see that the type information is not available here but wanted to ask anyway, in case it's possible to pass it in and if you agree it makes sense.

We saw something like this in the Windows mount logic (type information not available in unmount to distinguish one of several mounting mechanisms), and the proposed solution used there is to stash the information in the filesystem. So I agree that the unmounter should be the same as the mounter to support use-cases such as this.

cguentherTUChemnitz · 2023-01-16T14:17:14Z

When i understand the comment #1881 (comment) correctly, we still need this PR merged to get native container daemon support for macos without virtual machines underneath? I am waiting for this so long, since it is at the moment quite a dealbreaker in performance and integration how docker-for-mac is handled with vscode dev-container development.

What is the status here? Is it somewhat in the future able to merge?

footlooseboss · 2023-01-17T08:30:33Z

I am waiting for this so long, since it is at the moment quite a dealbreaker in performance and integration how docker-for-mac is handled with vscode dev-container development.

tried devbox?

JensVanhooydonck · 2023-03-31T13:59:25Z

What still needs to be done for this PR to be merged?

Hokwang · 2023-05-23T05:51:46Z

@mxpv any progress?

Zamaroht · 2023-07-09T07:14:13Z

Do we have any updates on this? @mxpv @dmcgowan

slonopotamus · 2023-07-09T22:49:21Z

See #8789 for a kind of an alternative to this PR.

AkihiroSuda · 2023-07-17T17:36:10Z

mount/mount_darwin.go

+	// ErrNotImplementOnUnix is returned for methods that are not implemented
+	ErrNotImplementOnUnix = errors.New("not implemented under darwin")
+	// allowedHelperBinaries is a list of third-party executable, which can
+	// be extended in future uses, e.g., mount_nfs, mount_9p.  `mount_darwin`


Suggested change

// be extended in future uses, e.g., mount_nfs, mount_9p. `mount_darwin`

// be extended in future uses, e.g., mount_nfs, mount_9p. `mount_containerd_darwin`

github-actions · 2024-05-27T00:09:44Z

This PR is stale because it has been open 90 days with no activity. This PR will be closed in 7 days unless new comments are made or the stale label is removed.

github-actions · 2024-06-03T00:09:52Z

This PR was closed because it has been stalled for 7 days with no activity.

k8s-ci-robot added the needs-ok-to-test label Sep 3, 2021

thehajime mentioned this pull request Sep 3, 2021

darwin runtime support #4526

Closed

6 tasks

kzys requested changes Sep 3, 2021

View reviewed changes

thehajime force-pushed the feature-darwin-native-snapshotter branch from 0c3484d to 390fc95 Compare September 10, 2021 01:39

thehajime force-pushed the feature-darwin-native-snapshotter branch from 390fc95 to 8d95110 Compare September 10, 2021 05:45

kzys requested changes Sep 13, 2021

View reviewed changes

thehajime closed this Sep 23, 2021

thehajime deleted the feature-darwin-native-snapshotter branch September 23, 2021 01:33

thehajime restored the feature-darwin-native-snapshotter branch September 23, 2021 01:34

thehajime reopened this Sep 23, 2021

thehajime force-pushed the feature-darwin-native-snapshotter branch from 8d95110 to 8c8bbea Compare September 23, 2021 01:37

thehajime changed the title ~~Support the native snapshotter on macOS~~ Introduce the darwin snapshotter on macOS Sep 23, 2021

thehajime force-pushed the feature-darwin-native-snapshotter branch from 8c8bbea to d329cc4 Compare September 23, 2021 09:06

AkihiroSuda reviewed Sep 27, 2021

View reviewed changes

thehajime force-pushed the feature-darwin-native-snapshotter branch from d329cc4 to 24895b3 Compare October 3, 2021 04:13

thehajime force-pushed the feature-darwin-native-snapshotter branch from 24895b3 to 9c74b76 Compare October 4, 2021 09:30

thehajime force-pushed the feature-darwin-native-snapshotter branch 2 times, most recently from 46570b3 to 54a0d1d Compare January 20, 2022 02:15

AkihiroSuda approved these changes Jan 20, 2022

View reviewed changes

AkihiroSuda requested a review from dmcgowan January 20, 2022 02:48

thehajime force-pushed the feature-darwin-native-snapshotter branch from 54a0d1d to 190b8b3 Compare January 21, 2022 00:01

thehajime mentioned this pull request Jan 25, 2022

darwin build containerd/nerdctl#732

Closed

mikedld reviewed Apr 11, 2022

View reviewed changes

TBBle mentioned this pull request May 5, 2022

Manifesting the containerd.exe and (some) tests #6896

Closed

kzys mentioned this pull request Jun 13, 2022

Extend Snapshotter and/or Mount to support cases where mount(2) cannot handle natively #7055

Closed

AkihiroSuda requested a review from mxpv February 9, 2023 23:42

slonopotamus mentioned this pull request Jul 9, 2023

[Reverted in #8838] Add support for bind-mounts on Darwin (a.k.a. "make native snapshotter work") #8789

Merged

AkihiroSuda reviewed Jul 17, 2023

View reviewed changes

k8s-ci-robot added the needs-rebase label Nov 29, 2023

slonopotamus mentioned this pull request Mar 11, 2024

Storage drivers darwin-containers/rund#51

Closed

github-actions bot added the Stale label May 27, 2024

github-actions bot closed this Jun 3, 2024

-func Unmount(mount string, flags int) error {
-	os.RemoveAll(mount)
-	return os.Mkdir(mount, 0755)
-}
-// UnmountAll is just a rm of symlink
-func UnmountAll(mount string, flags int) error {
-	os.RemoveAll(mount)
-	return os.Mkdir(mount, 0755)
-}
+func Unmount(target string, flags int) error {
+	if isSymlink(target) {
+		return unmountSymlink(target, 0755)
+	}
+	return errors.Wrapf(unix.EBUSY, "failed to unmount target %s", target)
+}

	// be extended in future uses, e.g., mount_nfs, mount_9p. `mount_darwin`
	// be extended in future uses, e.g., mount_nfs, mount_9p. `mount_containerd_darwin`

Extend mount package on macOS to use external snapshotters #5935

Extend mount package on macOS to use external snapshotters #5935

Uh oh!

Conversation

thehajime commented Sep 3, 2021

Uh oh!

k8s-ci-robot commented Sep 3, 2021

Uh oh!

theopenlab-ci bot commented Sep 3, 2021

Uh oh!

kzys Sep 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theopenlab-ci bot commented Sep 10, 2021

Uh oh!

theopenlab-ci bot commented Sep 10, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Sep 15, 2021

Uh oh!

thehajime commented Sep 15, 2021

Uh oh!

thehajime commented Sep 17, 2021

Uh oh!

kzys commented Sep 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thehajime commented Sep 21, 2021

Uh oh!

kzys commented Sep 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thehajime commented Sep 23, 2021

Uh oh!

theopenlab-ci bot commented Sep 23, 2021

Uh oh!

theopenlab-ci bot commented Sep 23, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thehajime Sep 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theopenlab-ci bot commented Oct 3, 2021

Uh oh!

theopenlab-ci bot commented Jan 19, 2022

Uh oh!

theopenlab-ci bot commented Jan 20, 2022

Uh oh!

kzys Sep 3, 2021 •

edited

Loading

kzys commented Sep 17, 2021 •

edited

Loading

kzys commented Sep 21, 2021 •

edited

Loading

thehajime Sep 27, 2021 •

edited

Loading

thehajime commented Mar 22, 2022 •

edited

Loading

TBBle Apr 15, 2022 •

edited

Loading