Skip to content

Added HoverNet Loss#5199

Merged
bhashemian merged 24 commits intoProject-MONAI:devfrom
JHancox:dev
Oct 24, 2022
Merged

Added HoverNet Loss#5199
bhashemian merged 24 commits intoProject-MONAI:devfrom
JHancox:dev

Conversation

@JHancox
Copy link
Copy Markdown
Contributor

@JHancox JHancox commented Sep 22, 2022

Fixes #4672

Description

Adds the Hovernet Loss, which takes the raw predictions from the HoverNet net and produces a Loss Tensor.

Hovernet Loss added along with test script that verfies same results as original TIA version of Hovernet loss. Test script also include synthetic image generation (fixed to deterministic behaviour)

N.B. Currently does not include a test function for torch script since the test includes code that torchscipt cannot compile - mainly the SobelGradient and associated functions. Need to resolve this.

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Sep 23, 2022

Hi @JHancox ,

I would suggest to put this loss implementation in:
https://github.com/Project-MONAI/MONAI/tree/dev/monai/apps/pathology
/losses/hovernet_loss.py

What do you think?

Thanks.

@JHancox
Copy link
Copy Markdown
Contributor Author

JHancox commented Sep 23, 2022

Hi @JHancox ,

I would suggest to put this loss implementation in: https://github.com/Project-MONAI/MONAI/tree/dev/monai/apps/pathology /losses/hovernet_loss.py

What do you think?

Thanks.

Thanks @Nic-Ma - I see your logic and it would make sense. On the other hand HoverNet is already in the monai/networks/nets branch and so we'd probably need to move this too.
@drbeh - what do you think?

@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Sep 23, 2022

Hi @JHancox ,

I think the monai/networks/nets/hovernet is OK.
I requested to move the loss to apps/pathology because it seems not a general loss function, it can only work for the particular HoverNet training pipeline.
Please feel free to correct me if I misunderstand anything.

Thanks.

@JHancox
Copy link
Copy Markdown
Contributor Author

JHancox commented Sep 23, 2022

Thanks @Nic-Ma.
You are right - the loss is specific to Hovernet. I have no problem redirecting this to apps but will leave the decision to @drbeh

@bhashemian
Copy link
Copy Markdown
Member

@JHancox @Nic-Ma
I agree. This is better suited for monai/apps/pathology/loss/.

@bhashemian
Copy link
Copy Markdown
Member

Hi @Nic-Ma @wyli @rijobro @ericspod @yiheng-wang-nv,
Do you have any insight why torchscript is failing here? https://github.com/Project-MONAI/MONAI/actions/runs/3190912645/jobs/5206609165#step:7:30985

It seems that it is related to SobelGradients but do you know what's the issue?

Thanks for your help.

ERROR: test_script (tests.test_hovernet_loss.TestHoverNetLoss)
tests finished, printing completed times >10.0s in ascending order...

test_senet_shape_0 (tests.test_senet.TestPretrainedSENET) (10.9s)
test_cpu_precise_backwards_7_3_dimension_1_channel_high_spatial_sigma_high_color_sigma (tests.test_bilateral_precise.BilateralFilterTestCaseCpuPrecise) (11.3s)
test_invert (tests.test_invertd.TestInvertd) (12.7s)
test_script_0 (tests.test_senet.TestSENET) (15.9s)
test_cuda_1_1_batches_1_dimensions_5_channels_2_classes_1_mixtures (tests.test_gmm.GMMTestCase) (16.5s)
test_cuda_2_1_batches_2_dimensions_2_channels_4_classes_4_mixtures (tests.test_gmm.GMMTestCase) (16.7s)
test_cuda_3_1_batches_3_dimensions_1_channels_2_classes_1_mixtures (tests.test_gmm.GMMTestCase) (17.2s)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/MONAI/MONAI/tests/test_hovernet_loss.py", line 181, in test_script
    test_script_save(loss, prediction, target)
  File "/__w/MONAI/MONAI/tests/utils.py", line 704, in test_script_save
    atol=atol,
  File "/__w/MONAI/MONAI/monai/networks/utils.py", line 593, in convert_to_torchscript
    script_module = torch.jit.script(model, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_script.py", line 1287, in script
    obj, torch.jit._recursive.infer_methods_to_compile
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py", line 458, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py", line 524, in create_script_module_impl
    create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py", line 375, in create_methods_and_properties_from_stubs
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py", line 876, in compile_unbound_method
    create_methods_and_properties_from_stubs(concrete_type, (stub,), ())
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py", line 375, in create_methods_and_properties_from_stubs
test_cuda_0_2_batches_1_dimensions_1_channels_2_classes_2_mixtures (tests.test_gmm.GMMTestCase) (17.6s)
test_dints_inference_3 (tests.test_dints_network.TestDints) (17.7s)
test_dints_inference_1 (tests.test_dints_network.TestDints) (19.0s)
test_dints_inference_2 (tests.test_dints_network.TestDints) (19.7s)
test_run_algo_after_move_files (tests.test_auto3dseg_hpo.TestHPO) (25.9s)
test_run_algo (tests.test_auto3dseg_hpo.TestHPO) (26.0s)
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py", line 876, in compile_unbound_method
    create_methods_and_properties_from_stubs(concrete_type, (stub,), ())
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_recursive.py", line 375, in create_methods_and_properties_from_stubs
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
RuntimeError: 
Module 'HoVerNetLoss' has no attribute 'sobel' (This attribute exists on the Python module, but we failed to convert Python type: 'monai.transforms.post.array.SobelGradients' to a TorchScript type. Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type SobelGradients.. Its type was inferred; try adding a type annotation for the attribute.):
  File "/__w/MONAI/MONAI/monai/apps/pathology/losses/hovernet_loss.py", line 56
    
        batch_size = image.shape[0]
        result_h = self.sobel(torch.squeeze(image[:, 0], dim=1))[batch_size:]
                   ~~~~~~~~~~ <--- HERE
        result_v = self.sobel(torch.squeeze(image[:, 1], dim=1))[:batch_size]
    
'HoVerNetLoss._compute_sobel' is being compiled since it was called from 'HoVerNetLoss._mse_gradient_loss'
  File "/__w/MONAI/MONAI/monai/apps/pathology/losses/hovernet_loss.py", line 63
    def _mse_gradient_loss(self, prediction: torch.Tensor, target: torch.Tensor, focus: torch.Tensor) -> torch.Tensor:
    
        pred_grad = self._compute_sobel(prediction)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        true_grad = self._compute_sobel(target)
    
'HoVerNetLoss._mse_gradient_loss' is being compiled since it was called from 'HoVerNetLoss.forward'
  File "/__w/MONAI/MONAI/monai/apps/pathology/losses/hovernet_loss.py", line 118
        # Use the nuclei class, one hot encoded, as the mask
        loss_hv_mse_grad = (
            self._mse_gradient_loss(
            ~~~~~~~~~~~~~~~~~~~~~~~~
                prediction[HoVerNetBranch.HV.value],
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                target[HoVerNetBranch.HV.value],
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                target[HoVerNetBranch.NP.value][:, 1],
                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
            )
            * self.lambda_hv_mse_grad


----------------------------------------------------------------------

@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Oct 7, 2022

Hi @benjamin-gorman ,

As your error message shows:
Module 'HoVerNetLoss' has no attribute 'sobel' (This attribute exists on the Python module, but we failed to convert Python type: 'monai.transforms.post.array.SobelGradients' to a TorchScript type. Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type SobelGradients.. Its type was inferred; try adding a type annotation for the attribute.
Maybe you can have a try?

Thanks.

@JHancox
Copy link
Copy Markdown
Contributor Author

JHancox commented Oct 7, 2022

Hi @benjamin-gorman ,

As your error message shows: Module 'HoVerNetLoss' has no attribute 'sobel' (This attribute exists on the Python module, but we failed to convert Python type: 'monai.transforms.post.array.SobelGradients' to a TorchScript type. Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type SobelGradients.. Its type was inferred; try adding a type annotation for the attribute. Maybe you can have a try?

Thanks.

I think the problem is that SobelGradients is a Transform, which is a Callable and I don't think TS supports the Callable type. Therefore, perhaps we might refactor SobelGradients so that it can be used as a Tensor-returning class and, perhaps wrap that in the Transform interface. ? Oh, and did you mean @drbeh ? :)

@bhashemian
Copy link
Copy Markdown
Member

bhashemian commented Oct 7, 2022

As your error message shows: Module 'HoVerNetLoss' has no attribute 'sobel' (This attribute exists on the Python module, but we failed to convert Python type: 'monai.transforms.post.array.SobelGradients' to a TorchScript type. Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type SobelGradients.. Its type was inferred; try adding a type annotation for the attribute. Maybe you can have a try?
Thanks.

I think the problem is that SobelGradients is a Transform, which is a Callable and I don't think TS supports the Callable type. Therefore, perhaps we might refactor SobelGradients so that it can be used as a Tensor-returning class and, perhaps wrap that in the Transform interface. ? Oh, and did you mean @drbeh ? :)

Hi @Nic-Ma,
As @JHancox mentioned, we have tried that and didn't work. Even removing sobel from being an attribute and apply it in the call does not work and give other different complaints. In my experience torch script error messages are not that useful although their statement is correct.

@JHancox, this transform is already a Tensor-returning class but the issue with torch script is that it is very limited in the scope of what it can support. https://pytorch.org/docs/stable/jit_unsupported.html#jit-unsupported

@Nic-Ma do we need the loss function to be torch scriptable? I am not sure if it is worth spending more time on this to make it to work with torch script (if possible at all). If it is not required, I would suggest to move forward without it.

@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Oct 8, 2022

Hi @drbeh ,

Thanks for the quick update.
I didn't see any TorchScript use case of loss function, so I feel we can remove the TorchScript tests so far.
But I see we have TorchScript compatible tests for almost all the losses:
https://github.com/Project-MONAI/MONAI/blob/dev/tests/test_dice_loss.py#L187
@wyli @ericspod Do you know the use case or reason to support that?

Thanks.

@wyli
Copy link
Copy Markdown
Contributor

wyli commented Oct 9, 2022

agree with @Nic-Ma to remove test_script from TestHoverNetLoss (previously to have torchscript is potentially speed up the training #1592 (comment))

@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Oct 12, 2022

/build

@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Oct 12, 2022

Is it ready for review now?

Thanks.

@JHancox
Copy link
Copy Markdown
Contributor Author

JHancox commented Oct 12, 2022

Is it ready for review now?

Thanks.

I just need to check with @drbeh whether there was anything else. Will try to get an answer today (in Sweden at the moment).

@bhashemian
Copy link
Copy Markdown
Member

I try to resolve some failing tests and some formatting, and then it should be ready. I'll request to review afterwards.

@bhashemian bhashemian marked this pull request as ready for review October 13, 2022 20:29
@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Oct 14, 2022

/build

@Nic-Ma Nic-Ma enabled auto-merge (squash) October 14, 2022 00:47
@Nic-Ma Nic-Ma disabled auto-merge October 17, 2022 03:28
bhashemian and others added 5 commits October 20, 2022 15:27
…arge images (Project-MONAI#5297)

SlidingWindowInferer: option to adaptively stitch in cpu memory for
large images.

This adds an option to provide maximum input image volume (number of
elements) to dynamically change stitching to cpu memory (to avoid gpu
memory crashes). For example with `cpu_thresh=400*400*400`, all input
images with large volume will be stitched on cpu.

At the moment, a user must decide beforehand, to stitch ALL images on
cpu or gpu (by specifying the 'device' parameter). But in many datasets,
only a few large images require device==cpu, and running inference on
cpu for ALL will be unnecessary slow.

It's related to 
Project-MONAI#4625
Project-MONAI#4495
Project-MONAI#3497
Project-MONAI#4726
Project-MONAI#4588 


### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: myron <[email protected]>
Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Behrooz <[email protected]>
…5280)

Fixes MONAI/apps/Auto3DSeg/bundle_gen.py

### Description

Commands ran in subprocess currently cause issues with string formatting
and backslashes not being escaped properly. Changing from Back Flash to
Forward Slash solves the issue.

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: Maxime <[email protected]>
Signed-off-by: Mingxin Zheng <[email protected]>
Signed-off-by: Mingxin Zheng <[email protected]>
Co-authored-by: Maxime <[email protected]>
Co-authored-by: Mingxin Zheng <[email protected]>
Co-authored-by: Mingxin Zheng <[email protected]>
Co-authored-by: Mingxin Zheng <[email protected]
Signed-off-by: Behrooz <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>

Fixes Project-MONAI#5133

### Description
adds a test

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [x] New tests added to cover the changes.
- [x] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [x] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: monai-bot <[email protected]>
Co-authored-by: monai-bot <[email protected]>
Signed-off-by: Behrooz <[email protected]>
JHancox and others added 2 commits October 20, 2022 15:27
Added the missing docstring entry from the HoVerNetLoss class

Signed-off-by: Behrooz <[email protected]>
Copy link
Copy Markdown
Member

@bhashemian bhashemian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @JHancox, I believe this PR is almost ready to be merged. I left few minor comments inline. Thanks

mingxin-zheng and others added 4 commits October 21, 2022 16:57
…5367)

Signed-off-by: Mingxin Zheng
<[email protected]>

Fixes Project-MONAI#5201 
Fixes Project-MONAI#5332 

### Description

- Remove deprecated meta_dict usage from Auto3DSeg. 
- Fix affine -> spacing conversion
- Update docstring 
- Change "pixel_percentage" to "foreground_percentage" to unify
foreground "pixel"/"voxel" naming

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [x] In-line docstrings updated.

Signed-off-by: Mingxin Zheng <[email protected]>
Signed-off-by: Behrooz <[email protected]>
### Description
adding tests for python 3.10

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [x] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Behrooz <[email protected]>
JHancox and others added 3 commits October 21, 2022 18:42
Moved variables and assigned test inputs to a list

Signed-off-by: JHancox <[email protected]>
Signed-off-by: Behrooz <[email protected]>
@bhashemian bhashemian requested a review from Nic-Ma October 21, 2022 18:15
@bhashemian bhashemian enabled auto-merge (squash) October 21, 2022 18:18
@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Oct 24, 2022

/build

@bhashemian bhashemian merged commit 4b41ad9 into Project-MONAI:dev Oct 24, 2022
wyli pushed a commit that referenced this pull request Oct 24, 2022
Fixes #4672 

### Description

Adds the Hovernet Loss, which takes the raw predictions from the
HoverNet net and produces a Loss Tensor.

Hovernet Loss added along with test script that verfies same results as
original TIA version of Hovernet loss. Test script also include
synthetic image generation (fixed to deterministic behaviour)


N.B. Currently does not include a test function for torch script since
the test includes code that torchscipt cannot compile - mainly the
SobelGradient and associated functions. Need to resolve this.



### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [x] New tests added to cover the changes.
- [x] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [x] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [x] In-line docstrings updated.
- [x] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: Behrooz <[email protected]>
wyli pushed a commit that referenced this pull request Oct 24, 2022
Fixes #4672 

### Description

Adds the Hovernet Loss, which takes the raw predictions from the
HoverNet net and produces a Loss Tensor.

Hovernet Loss added along with test script that verfies same results as
original TIA version of Hovernet loss. Test script also include
synthetic image generation (fixed to deterministic behaviour)


N.B. Currently does not include a test function for torch script since
the test includes code that torchscipt cannot compile - mainly the
SobelGradient and associated functions. Need to resolve this.



### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [x] New tests added to cover the changes.
- [x] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [x] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [x] In-line docstrings updated.
- [x] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: Behrooz <[email protected]>
@wyli
Copy link
Copy Markdown
Contributor

wyli commented Oct 25, 2022

this PR causes some integration errors, could you please take a look? #5393 @drbeh @Nic-Ma

@bhashemian
Copy link
Copy Markdown
Member

Hi @wyli, sure, I'll look into it.

KumoLiu pushed a commit that referenced this pull request Nov 2, 2022
Fixes #4672 

### Description

Adds the Hovernet Loss, which takes the raw predictions from the
HoverNet net and produces a Loss Tensor.

Hovernet Loss added along with test script that verfies same results as
original TIA version of Hovernet loss. Test script also include
synthetic image generation (fixed to deterministic behaviour)


N.B. Currently does not include a test function for torch script since
the test includes code that torchscipt cannot compile - mainly the
SobelGradient and associated functions. Need to resolve this.



### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [x] New tests added to cover the changes.
- [x] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [x] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [x] In-line docstrings updated.
- [x] Documentation updated, tested `make html` command in the `docs/`
folder.

Signed-off-by: Behrooz <[email protected]>
Signed-off-by: KumoLiu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HoverNet Loss

10 participants