Between Linux kernel v4.19.55 and v4.19.56 the containerd v1.2.7 overlayfs tests started failing with:
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474310810Z: ok github.com/containerd/containerd/archive 0.208s
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474323771Z: ok github.com/containerd/containerd/metadata 20.300s
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474343687Z: ok github.com/containerd/containerd/mount 0.345s
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474356575Z: ok github.com/containerd/containerd/snapshots/btrfs 30.088s
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474369190Z: ok github.com/containerd/containerd/snapshots/native 10.363s
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474382189Z: --- FAIL: TestOverlay (0.00s)
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474395478Z: --- FAIL: TestOverlay/Basic (0.30s)
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474408833Z: testsuite.go:253: failure reason: device or resource busy
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474426837Z: helpers_unix.go:32: unmount /var/lib/tmp/snapshot-suite-Overlay-391257545/work/nextlayer
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474439714Z: helpers_unix.go:32: unmount /var/lib/tmp/snapshot-suite-Overlay-391257545/work/preparing
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474452495Z: helpers.go:67: drwx------ 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474465674Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474483527Z: helpers.go:65: -rw------- 65536 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/metadata.db [ "\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\xed\xda\f\xed\x02\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\b\x00\x00\x00\x00\x00\x00\x00\r\x00\x00\x00\x00\x00\x00\x00" ...]
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474502336Z: helpers.go:67: drwx------ 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474516170Z: helpers.go:67: drwx------ 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/1
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474529104Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/1/fs
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474541854Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/1/fs/a
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474554751Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/1/fs/a/b
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474567599Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/1/fs/a/b/c
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474587866Z: helpers.go:65: -rwxrwxrwx 4 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/1/fs/foo [ "foo\n" ...]
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474602035Z: helpers.go:67: drwx--x--x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/1/work
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474615234Z: helpers.go:67: drwx------ 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/2
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474627814Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/2/fs
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474640840Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/2/fs/a
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474653746Z: helpers.go:67: Dc--------- 0 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/2/fs/a/b
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474672239Z: helpers.go:65: -rwxrwxrwx 4 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/2/fs/bar [ "bar\n" ...]
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474686135Z: helpers.go:65: -rwxrwxrwx 4 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/2/fs/foo [ "bar\n" ...]
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474699521Z: helpers.go:67: drwx--x--x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/2/work
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474712185Z: helpers.go:67: d--------- 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/2/work/work
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474725102Z: helpers.go:67: drwx------ 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/3
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474738107Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/root/snapshots/3/fs
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474753847Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/work
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474766521Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/work/nextlayer
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474784007Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/work/nextnextlayer
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474796958Z: helpers.go:67: drwxr-xr-x 4096 /var/lib/tmp/snapshot-suite-Overlay-391257545/work/preparing
[+ 44m 32s] [STDERR ] 2019-07-03T20:44:20.474810035Z: FAIL
In the kernel changelog there were a number of changes, which seem relevant:
0319ef1d40ff ("ovl: fix bogus -Wmaybe-unitialized warning") (2019-06-25 11:35:52 +0800) <Arnd Bergmann (N)>
639e8c2f0910 ("ovl: don't fail with disconnected lower NFS") (2019-06-25 11:35:52 +0800) <Miklos Szeredi (N)>
f1c5aa5eda08 ("ovl: detect overlapping layers") (2019-06-25 11:35:52 +0800) <Amir Goldstein (N)>
a00f405e133f ("ovl: make i_ino consistent with st_ino in more cases") (2019-06-25 11:35:52 +0800) <Amir Goldstein (N)>
d6623379d895 ("ovl: fix wrong flags check in FS_IOC_FS[SG]ETXATTR ioctls") (2019-06-25 11:35:52 +0800) <Amir Goldstein (N)>
3cb5d7fa8f7d ("ovl: support the FS_IOC_FS[SG]ETXATTR ioctls") (2019-06-25 11:35:52 +0800) <Amir Goldstein (N)>
Discovered with LinuxKit during a kernel update: linuxkit/linuxkit#3387
Between Linux kernel v4.19.55 and v4.19.56 the containerd v1.2.7 overlayfs tests started failing with:
In the kernel changelog there were a number of changes, which seem relevant:
Discovered with LinuxKit during a kernel update: linuxkit/linuxkit#3387