Skip to content

Generate fake/simulated images in Auto3DSeg tests based on mGPU test env#6347

Merged
wyli merged 4 commits intoProject-MONAI:devfrom
mingxin-zheng:fix-6342
Apr 12, 2023
Merged

Generate fake/simulated images in Auto3DSeg tests based on mGPU test env#6347
wyli merged 4 commits intoProject-MONAI:devfrom
mingxin-zheng:fix-6342

Conversation

@mingxin-zheng
Copy link
Copy Markdown
Contributor

@mingxin-zheng mingxin-zheng commented Apr 12, 2023 •
edited
Loading

Fixes #6342 .

Description

In some testing environment, there are more GPUs than the simulated dataset image number. This PR is to modify the tests so that it will generate images enough to finish the testing.

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.

@mingxin-zheng mingxin-zheng requested a review from wyli April 12, 2023 10:45
wyli
wyli reviewed Apr 12, 2023
View reviewed changes
@wyli
Copy link
Copy Markdown
Contributor

wyli commented Apr 12, 2023 •
edited
Loading

@mingxin-zheng please consider adding a skip for test_autorunner_gpu_customization for torch 1.9

(updated the PR, please let me know if there's any concern)

[2023-04-12T13:41:11.671Z] ======================================================================
[2023-04-12T13:41:11.671Z] ERROR: test_autorunner_gpu_customization (__main__.TestAutoRunner)
[2023-04-12T13:41:11.671Z] ----------------------------------------------------------------------
[2023-04-12T13:41:11.671Z] Traceback (most recent call last):
[2023-04-12T13:41:11.671Z]   File "tests/test_integration_autorunner.py", line 149, in test_autorunner_gpu_customization
[2023-04-12T13:41:11.671Z]     runner.run()
[2023-04-12T13:41:11.671Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/auto_runner.py", line 749, in run
[2023-04-12T13:41:11.671Z]     bundle_generator.generate(
[2023-04-12T13:41:11.671Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 530, in generate
[2023-04-12T13:41:11.671Z]     gen_algo.export_to_disk(
[2023-04-12T13:41:11.671Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 152, in export_to_disk
[2023-04-12T13:41:11.671Z]     self.fill_records = self.fill_template_config(self.data_stats_files, self.output_path, **kwargs)
[2023-04-12T13:41:11.671Z]   File "/tmp/tmpasxr2duu/work_dir/algorithm_templates/dints/scripts/algo.py", line 225, in fill_template_config
[2023-04-12T13:41:11.671Z]   File "/tmp/tmpasxr2duu/work_dir/algorithm_templates/dints/scripts/algo.py", line 261, in customize_param_for_gpu
[2023-04-12T13:41:11.671Z]   File "/tmp/tmpasxr2duu/work_dir/algorithm_templates/dints/scripts/algo.py", line 31, in get_mem_from_visible_gpus
[2023-04-12T13:41:11.671Z] AttributeError: module 'torch.cuda' has no attribute 'mem_get_info'
[2023-04-12T13:41:11.671Z] 
[2023-04-12T13:41:11.671Z] ----------------------------------------------------------------------

@wyli
Copy link
Copy Markdown
Contributor

wyli commented Apr 12, 2023 •
edited
Loading

/integration-test
/build

@wyli wyli enabled auto-merge (squash) April 12, 2023 17:33
@wyli wyli merged commit 21cef45 into Project-MONAI:dev Apr 12, 2023
@mingxin-zheng mingxin-zheng deleted the fix-6342 branch April 14, 2023 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wyli wyli wyli approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Integration failure with unmerged ALGO_HASH

2 participants

@mingxin-zheng @wyli