Skip to content

Segresnet algo updates#211

Merged
mingxin-zheng merged 2 commits intoProject-MONAI:mainfrom
myron:segres
Apr 14, 2023
Merged

Segresnet algo updates#211
mingxin-zheng merged 2 commits intoProject-MONAI:mainfrom
myron:segres

Conversation

@myron
Copy link
Copy Markdown
Collaborator

@myron myron commented Mar 24, 2023

Segresnet algo updates...

@myron
Copy link
Copy Markdown
Collaborator Author

myron commented Mar 27, 2023

@mingxin-zheng can you review it please

mingxin-zheng
mingxin-zheng reviewed Mar 28, 2023
View reviewed changes
mingxin-zheng
mingxin-zheng reviewed Mar 28, 2023
View reviewed changes
@mingxin-zheng
Copy link
Copy Markdown
Contributor

mingxin-zheng commented Mar 28, 2023 •
edited
Loading

I have the following error when running the tests/test_algo_templates.py

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "auto3dseg/tests/test_algo_templates.py", line 164, in test_sim
    preds = auto_run(work_dir, data_src_cfg, self.algos)
  File "auto3dseg/tests/test_algo_templates.py", line 132, in auto_run
    preds = builder.get_ensemble()(pred_param)
  File "/workspace/monai/monai-in-dev/monai/apps/auto3dseg/ensemble_builder.py", line 166, in __call__
    outputs.append(self.ensemble_pred(preds, sigmoid=sigmoid))
  File "/workspace/monai/monai-in-dev/monai/apps/auto3dseg/ensemble_builder.py", line 111, in ensemble_pred
    prob = MeanEnsemble()(preds)
  File "/workspace/monai/monai-in-dev/monai/transforms/post/array.py", line 628, in __call__
    img_ = self.get_stacked_torch(img)
  File "/workspace/monai/monai-in-dev/monai/transforms/post/array.py", line 589, in get_stacked_torch
    out: torch.Tensor = torch.stack(img) if isinstance(img, Sequence) else img  # type: ignore
  File "/workspace/monai/monai-in-dev/monai/data/meta_tensor.py", line 276, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/_tensor.py", line 1241, in __torch_function__
    ret = func(*args, **kwargs)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument tensors in method wrapper_cat)

Detail log in a3d.log

@mingxin-zheng
Copy link
Copy Markdown
Contributor

mingxin-zheng commented Mar 28, 2023

Hi @myron thank you for the PR! It would be nice if you can add a little bit more details in either the title or the description of the issue ticket. So that next time we update the ALGO_HASH in MONAI, reviewers can quickly remember what happened with Auto3DSeg as in this PR

@myron
Copy link
Copy Markdown
Collaborator Author

myron commented Apr 14, 2023

prob = MeanEnsemble()(preds)

it should be fixed for this test case, but this actually uncovers a bug we need to handle. Different algorithms may return results on different devices (CPU or GPU) , e.g. if algo didn't have enough gpu memory it may do CPU based slidingwindow inference. So than when we do Ensembling, some internal function concatinate the predictions and it fails, since the predictions are on different devices. @mingxin-zheng we should open an issue to track it

@myron
Copy link
Copy Markdown
Collaborator Author

myron commented Apr 14, 2023

I've updated the branch, please check @mingxin-zheng

There is still clean up required for print() statements to transition to loggin,
I'll try to do it fast, or we can do another PR

@mingxin-zheng
Copy link
Copy Markdown
Contributor

mingxin-zheng commented Apr 14, 2023

@myron let's do it in a separate PR, since logging won't affect benchmarking

@mingxin-zheng mingxin-zheng mentioned this pull request Apr 14, 2023
Closed
@mingxin-zheng mingxin-zheng merged commit 8b10ad5 into Project-MONAI:main Apr 14, 2023
@mingxin-zheng mingxin-zheng mentioned this pull request Apr 14, 2023
Merged
7 tasks
wyli pushed a commit to Project-MONAI/MONAI that referenced this pull request Apr 14, 2023
@mingxin-zheng
Update ALGO_HASH/skip slow tests 6359 (#6367)
c5b1127 
### Description

The new ALGO_HASH has two updates for better performance of Auto3DSeg
algo templates:
- Project-MONAI/research-contributions#193
- Project-MONAI/research-contributions#211


### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: Mingxin Zheng <[email protected]>
heyufan1995 pushed a commit to heyufan1995/research-contributions that referenced this pull request Apr 17, 2023
@myron @pre-commit-ci @heyufan1995
Segresnet algo updates (Project-MONAI#211)
cec032b 
* SegResNet algo update

Signed-off-by: am <am>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: am <am>
Co-authored-by: am <am>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mingxin-zheng mingxin-zheng mingxin-zheng left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@myron @mingxin-zheng