Skip to content

Enable OOM Kill events integration test#36229

Closed
arm64b wants to merge 1 commit intomoby:masterfrom
arm64b:enable-Oom-ci
Closed

Enable OOM Kill events integration test#36229
arm64b wants to merge 1 commit intomoby:masterfrom
arm64b:enable-Oom-ci

Conversation

@arm64b
Copy link
Copy Markdown
Contributor

@arm64b arm64b commented Feb 7, 2018

Now that the -m flag can work in case of cgroup swap memory limit
isn't enabled, also we have PR #36201 merged, so now we can remove
the swapMemorySupport from testRequires() order to trigger the real
test on those platforms which don't enable cgroup swap mem limit.

Signed-off-by: Dennis Chen [email protected]

- What I did
Trigger OOM Kill events integration tests
- How I did it
Remove swapMemorySupport from testRequires()
- How to verify it
make test-integration
- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

Now that the `-m` flag can work in case of cgroup swap memory limit
isn't enabled, also we have PR moby#36201 merged, so now we can remove
the `swapMemorySupport` from `testRequires()` order to trigger the real
test on those platforms which don't enable cgroup swap mem limit.

Signed-off-by: Dennis Chen <[email protected]>
@arm64b
Copy link
Copy Markdown
Contributor Author

arm64b commented Feb 7, 2018

Relevant commit for this PR: commit 3abf2a77414195d5b0c15f7fc88b7fd9aa4edaa8

And we snapshot one CI log on janky node before this PR to see what will happen, curiously🤗:
https://jenkins.dockerproject.org/job/Docker-PRs/47879/console

@arm64b
Copy link
Copy Markdown
Contributor Author

arm64b commented Feb 7, 2018

From the CI result, seems the -m flag can't work in case of the swap limit is not enabled on powerpc platform only (I have a local test on AArch64, works), so before I add NotPpc64le to the testRequires() to skip these tests on powerpc, maybe @tophj-ibm can take a look at this issue on that platform to find the root cause...

Anyway this change will put those OOM kill event tests into practice while not just simply skip it on all platforms, for example, those tests output on janky node after this PR:

08:40:15 PASS: docker_cli_events_unix_test.go:51: DockerSuite.TestEventsOOMDisableFalse	0.465s
08:40:15 PASS: docker_cli_events_unix_test.go:81: DockerSuite.TestEventsOOMDisableTrue	0.523s

@tophj-ibm
Copy link
Copy Markdown
Contributor

tophj-ibm commented Feb 7, 2018

I don't think removing swapMemorySupport is the correct solution here, I think instead we want to enable swap limiting capabilities in all the CI machines so this test is run as is.

I think the issue with one of the powerpc tests is that the swap size on those machines is large enough that it doesn't run out of memory in time, because of the way --memory works with swap caps disabled. Here is a good comment that details what is going on #20627 (comment)

@arm64b
Copy link
Copy Markdown
Contributor Author

arm64b commented Feb 8, 2018

Right! So I am thinking about if we can figure out a method to trigger the OOM state quickly in case of CI machines don't enable the cgroup swap limit configuration? swapoff -a can get that result, but it's global.
Hmm, now we can leave the decision to the maintainers @thaJeztah 😄 : 1. we need to configure the kernel in CI machines to enable the cgroup swap limit Or 2. Those test cases can't play its role as they should before we can figure out a reasonable way to trigger OOM in a very short time if we don't enable swap limit cgroup on CI machines 😭

@tophj-ibm
Copy link
Copy Markdown
Contributor

@arm64b I sent a message to one of the people who maintains these machines and asked to get it enabled.

@arm64b
Copy link
Copy Markdown
Contributor Author

arm64b commented Feb 9, 2018

Hello @tophj-ibm I agree that will be final solution to do that, although we do have the intention to put those test cases into practice to trigger the real issues asap. I think now we can close this PR...

@arm64b arm64b closed this Feb 9, 2018
@arm64b arm64b deleted the enable-Oom-ci branch February 9, 2018 07:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants