Reenable make test targets in GH Actions CI by estesp · Pull Request #4719 · containerd/containerd

estesp · 2020-11-10T14:38:40Z

Related to #4705: re-enable tests hidden by removal of "coverage" targets

I will drop the first commit and rebase when #4705 is merged but want to verify we can get passing tests

theopenlab-ci · 2020-11-10T14:47:05Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 6m 55s (non-voting)

thaJeztah · 2020-11-10T15:16:54Z

Are the GOPATH=$GOPATH GOPROXY=$GOPROXY needed? I think they should be preserved by -E (only PATH is a special case)

Suggested change

sudo -E PATH=$PATH GOPATH=$GOPATH GOPROXY=$GOPROXY make root-test

sudo -E PATH=$PATH make root-test

to be honest I was just copying from integration section below. :)

oh, good one. Looks like those don't use sudo -E (not sure why that was not used), so they would only get the env-vars that are explicitly set 🤔

theopenlab-ci · 2020-11-10T15:24:16Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 5m 54s (non-voting)

estesp · 2020-11-10T15:26:04Z

Hmm, don't get this locally:

--- FAIL: TestSetNegativeOomScoreAdjustmentWhenUnprivilegedHasNoEffect (0.00s)
Error:     oom_unix_test.go:66: assertion failed: 500 (adjustment int) != 0 (int)

theopenlab-ci · 2020-11-10T16:04:43Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 7m 00s (non-voting)

estesp · 2020-11-10T16:42:21Z

Windows tests now re-added and working

Linux working fine for make test after fixing OOM in GH Actions env.
Linux failing for make root-test with a unsafe pointer check in btrfs.SubvolCreate:

fatal error: checkptr: converted pointer straddles multiple allocations

If anyone has any history with this Golang unsafe pointer checking error and wants to dig in.. :)

fuweid · 2020-11-11T15:16:46Z

Windows tests now re-added and working

Linux working fine for make test after fixing OOM in GH Actions env.
Linux failing for make root-test with a unsafe pointer check in btrfs.SubvolCreate:
fatal error: checkptr: converted pointer straddles multiple allocations
If anyone has any history with this Golang unsafe pointer checking error and wants to dig in.. :)

Try to fix it https://github.com/containerd/btrfs/pull/27/files

theopenlab-ci · 2020-11-11T19:25:27Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 5m 58s (non-voting)

theopenlab-ci · 2020-11-17T02:36:37Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 6m 29s (non-voting)

estesp · 2020-11-17T03:03:22Z

Hmm. devmapper snapshotter suite tests are hanging; running locally I see that 2 pools ending in snap-1 are stuck in dmsetup remove --retry loops; trying that manually gets:

$ sudo dmsetup remove --retry /dev/mapper/containerd-snapshotter-suite-pool-35024991-snap-1
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy
device-mapper: remove ioctl on containerd-snapshotter-suite-pool-35024991-snap-1  failed: Device or resource busy

I see a mount matching that ID:

/dev/mapper/containerd-snapshotter-suite-pool-35024991-snap-1 on /tmp/snapshot-suite-devmapper-761964118/work/base type ext4 (rw,relatime,stripe=16)

If I manually unmount (as well as the other stuck dmsetup remove target) the tests complete, but are unsuccessful. I'm not sure who has the ability to poke at this a bit more, but maybe @mxpv or anyone else well versed in devmapper?

mxpv · 2020-11-18T19:34:14Z

ohh.. These are my "favorite".. Will take a look.

estesp · 2020-11-18T22:26:59Z

@mxpv one more data point; reverting 63b7587 and c383436 take away the hang (although on Ubuntu 20.04 I still get test failures); since the tests were not enabled during these PR cycles, I wonder if there are test changes that are required to align with the changes to device lifecycle in these PRs?

mxpv · 2020-11-18T22:40:34Z

@estesp How do you feel about disabling devmapper unit tests (until we figure out a proper fix, I don't want to rollback those changes) and have this PR in ? Having CI without unit tests feels like a disaster 🥇

estesp · 2020-11-18T23:19:47Z

@mxpv yes, I think disabling just devmapper to move on with re-enabling tests is good. I don't think we want to revert those fixes, I just thought it might be a clue that the tests need an update based on those PRs and since tests weren't running then, it wasn't caught.

Disable devmapper for now until test issues are fixed. Signed-off-by: Phil Estes <[email protected]>

GitHub Actions process wrapper sets score adj to 500 for any process; the OOM score adj test expected default adj to be 0 during test. Signed-off-by: Phil Estes <[email protected]>

Fixes runtime panic for testing snapshotters Signed-off-by: Phil Estes <[email protected]>

theopenlab-ci · 2020-11-19T13:57:39Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 5m 56s (non-voting)

thaJeztah · 2020-11-19T14:05:40Z

Interesting; is that depending on host configuration, or was index=off added as a default?

        --- FAIL: TestOverlay/no_opt/TestOverlayOverlayMount (0.01s)
Error:             overlay_test.go:182: expected "workdir=/tmp/overlay433793594/snapshots/2/work" but received "index=off"
Error:             overlay_test.go:182: expected "upperdir=/tmp/overlay433793594/snapshots/2/fs" but received "workdir=/tmp/overlay433793594/snapshots/2/work"
Error:             overlay_test.go:182: expected "lowerdir=/tmp/overlay433793594/snapshots/1/fs" but received "upperdir=/tmp/overlay433793594/snapshots/2/fs"

estesp · 2020-11-19T15:53:49Z

change was made while tests were not enabled in CI, so wasn't caught: d36810d#diff-22eed5233aae9c9baa1749e3ca948da07c6e2de607dbbbb00e17de3627fbcca7

Hopefully this is the last fix to get tests re-enabled, although devmapper is turned off for now

When running tests on any modern distro, this assumption will work. If we need to make it work with kernels where we don't append this option it will require some more involved changes. Signed-off-by: Phil Estes <[email protected]>

theopenlab-ci · 2020-11-19T16:08:45Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 6m 26s (non-voting)

fuweid

LGTM

estesp force-pushed the fix-shm-relabel-test branch from f3c7e05 to 44263b7 Compare November 10, 2020 15:16

thaJeztah reviewed Nov 10, 2020

View reviewed changes

estesp force-pushed the fix-shm-relabel-test branch from 4c9bdd9 to 94a7d96 Compare November 17, 2020 02:28

estesp changed the title ~~Fix shm relabel test~~ Reenable make test targets in GH Actions CI Nov 17, 2020

estesp added 2 commits November 19, 2020 08:43

Add Go test runs to GitHub Actions CI

c11472d

Disable devmapper for now until test issues are fixed. Signed-off-by: Phil Estes <[email protected]>

Allow oom adj test to run in environments with a score

af2fb4e

GitHub Actions process wrapper sets score adj to 500 for any process; the OOM score adj test expected default adj to be 0 during test. Signed-off-by: Phil Estes <[email protected]>

estesp force-pushed the fix-shm-relabel-test branch from 94a7d96 to d2d3e13 Compare November 19, 2020 13:43

Import crypto for all snapshotters during testsuite

027ee56

Fixes runtime panic for testing snapshotters Signed-off-by: Phil Estes <[email protected]>

estesp force-pushed the fix-shm-relabel-test branch from d2d3e13 to 027ee56 Compare November 19, 2020 13:50

Adjust overlay tests to expect "index=off"

85d9fe3

When running tests on any modern distro, this assumption will work. If we need to make it work with kernels where we don't append this option it will require some more involved changes. Signed-off-by: Phil Estes <[email protected]>

mxpv approved these changes Nov 19, 2020

View reviewed changes

fuweid approved these changes Nov 23, 2020

View reviewed changes

fuweid merged commit 625da6b into containerd:master Nov 23, 2020

estesp mentioned this pull request Nov 23, 2020

[release/1.4] re-enable unit tests in GitHub Actions environment #4763

Merged

fuweid added the cherry-picked/1.4.x PR commits are cherry picked into the release/1.4 branch label Nov 24, 2020

mxpv mentioned this pull request Dec 9, 2020

Enable devmapper tests #4821

Merged

This was referenced Aug 30, 2022

[20.10 backport] containerd fixes for BuildKit moby/containerd#2

Merged

[20.10] vendor: update containerd to latest of docker-20.10 branch moby/moby#44067

Merged

	sudo -E PATH=$PATH GOPATH=$GOPATH GOPROXY=$GOPROXY make root-test
	sudo -E PATH=$PATH make root-test

Conversation

estesp commented Nov 10, 2020

Uh oh!

theopenlab-ci Bot commented Nov 10, 2020

Uh oh!

thaJeztah Nov 10, 2020

Choose a reason for hiding this comment

Uh oh!

estesp Nov 10, 2020

Choose a reason for hiding this comment

Uh oh!

thaJeztah Nov 10, 2020

Choose a reason for hiding this comment

Uh oh!

theopenlab-ci Bot commented Nov 10, 2020

Uh oh!

estesp commented Nov 10, 2020

Uh oh!

theopenlab-ci Bot commented Nov 10, 2020

Uh oh!

estesp commented Nov 10, 2020

Uh oh!

fuweid commented Nov 11, 2020

Uh oh!

theopenlab-ci Bot commented Nov 11, 2020

Uh oh!

theopenlab-ci Bot commented Nov 17, 2020

Uh oh!

estesp commented Nov 17, 2020

Uh oh!

mxpv commented Nov 18, 2020

Uh oh!

estesp commented Nov 18, 2020

Uh oh!

mxpv commented Nov 18, 2020

Uh oh!

estesp commented Nov 18, 2020

Uh oh!

theopenlab-ci Bot commented Nov 19, 2020

Uh oh!

thaJeztah commented Nov 19, 2020

Uh oh!

estesp commented Nov 19, 2020

Uh oh!

theopenlab-ci Bot commented Nov 19, 2020

Uh oh!

fuweid left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants