3961 fixes testing memory bug and revert the workaround#3978
Merged
wyli merged 6 commits intoProject-MONAI:devfrom Mar 22, 2022
Merged
3961 fixes testing memory bug and revert the workaround#3978wyli merged 6 commits intoProject-MONAI:devfrom
wyli merged 6 commits intoProject-MONAI:devfrom
Conversation
Signed-off-by: Wenqi Li <[email protected]>
…roject-MONAI#3949)" This reverts commit 6ea9742. Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
wyli
force-pushed
the
3961-testing-memory-issue
branch
from
0bb07fb to
4295969
Compare
Signed-off-by: Wenqi Li <[email protected]>
Contributor
Author
|
/integration-test |
Nic-Ma
approved these changes
Mar 22, 2022
Contributor
Nic-Ma
left a comment
Nic-Ma
left a comment
There was a problem hiding this comment.
Thanks for the deep investigation for the issue!
Looks good to me.
Contributor
Author
|
/build |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3961
Description
Multiprocessing in the 'spawn' mode would trigger the
has_cupyevaluation for each subprocess. the evaluations will cause multple cuda context loading which takes a good amount of memory. The issue becomes obvious when 'num_workers' is large.The solution here is to skip cupy check if it's in subprocess. (the main testing process is named by default "MainProcess"). A stress test is added to avoid this bug in the future.
Status
Ready
Types of changes
./runtests.sh -f -u --net --coverage../runtests.sh --quick --unittests --disttests.make htmlcommand in thedocs/folder.