Occlusion sensitivity to use slidiing_window_inference#5230
Merged
wyli merged 8 commits intoProject-MONAI:devfrom Sep 30, 2022
Merged
Occlusion sensitivity to use slidiing_window_inference#5230wyli merged 8 commits intoProject-MONAI:devfrom
wyli merged 8 commits intoProject-MONAI:devfrom
Conversation
Signed-off-by: Richard Brown <[email protected]>
Signed-off-by: Richard Brown <[email protected]>
Signed-off-by: Richard Brown <[email protected]>
wyli
reviewed
Sep 30, 2022
wyli
reviewed
Sep 30, 2022
wyli
reviewed
Sep 30, 2022
ericspod
approved these changes
Sep 30, 2022
Member
|
This can close #3032 though one thing mentioned there is determining the importance of individual channels, this may be important to support with an example of how it can be shown to work. I'm not sure if cats and dogs would be suitable unless the network fixates on specific coat colours or something like that. |
Contributor
Author
|
I don't think this solves #3032 – it still remains in the domain of interpretability of image classification, not segmentation. The output of |
Signed-off-by: Richard Brown <[email protected]>
Contributor
|
I see, for unet, it'll require some spatial pooling after this line to make it from BCHWD -> BC... could be a very interesting future PR :) |
Contributor
|
/build |
Contributor
|
/build |
wyli
pushed a commit
that referenced
this pull request
Oct 10, 2022
### Description
Occlusion sensitivity currently works by setting the logits at the
center of the occluded region. Even though neighbouring voxels are
occluded at the same time, this is not recorded. A better way to do it
would be to do an average each time a voxel is occluded. This is taken
care of by switching to using `sliding_window_inference`.
We had to do a bit of hacking to get this to work.
`sliding_window_inference` normally takes a subset of the whole image
and infers that. We want `sliding_window_inference` to tell us which
part of the image we should occlude, then we occlude it and infer the
**whole** image. To that end, we use a meshgrid to tell us the
coordinates of the region `sliding_window_inference` wants us to
occlude. We occlude, infer and then crop back to the size of the region
that was requested by `sliding_window_inference`.
This PR also allows for different occlusion kernels. We currently have:
- gaussian (actually inverse gaussian): the center of the occluded
region is zero, and towards the edge of the image is unchanged. This
doesn't introduce hard edges into the image, which might undermine the
visualisation process.
- mean_patch: the occluded region is replaced with the mean of the patch
it is occluding.
- mean_img: the occluded region is replaced with the mean of the whole
image (current implementation).
## Changes to input arguments
This PR is backwards incompatible, as using `sliding_window_inference`
means changing the API significantly.
- pad_val: now determined by `mode`
- stride: `overlap` used instead
- per_channel: all channels are done simultaneously
- upsampler: image is no longer downsampled
## Changes to output
Output previously had the shape `B,C,H,W,[D],N` where `C` and `N` were
the number of input and output channels of the network, respectively.
Now, we output the shape `B,N,H,W,[D]` as the `per_channel` feature is
no longer present.
Columns 2-4 are occlusion sensitivity done with Gaussian, mean of patch
and mean of image:

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [x] Breaking change (fix or new feature that would cause existing
functionality to change).
- [x] New tests added to cover the changes.
- [x] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [x] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [x] In-line docstrings updated.
- [x] Documentation updated, tested `make html` command in the `docs/`
folder.
Signed-off-by: Richard Brown <[email protected]>
KumoLiu
pushed a commit
that referenced
this pull request
Nov 2, 2022
### Description
Occlusion sensitivity currently works by setting the logits at the
center of the occluded region. Even though neighbouring voxels are
occluded at the same time, this is not recorded. A better way to do it
would be to do an average each time a voxel is occluded. This is taken
care of by switching to using `sliding_window_inference`.
We had to do a bit of hacking to get this to work.
`sliding_window_inference` normally takes a subset of the whole image
and infers that. We want `sliding_window_inference` to tell us which
part of the image we should occlude, then we occlude it and infer the
**whole** image. To that end, we use a meshgrid to tell us the
coordinates of the region `sliding_window_inference` wants us to
occlude. We occlude, infer and then crop back to the size of the region
that was requested by `sliding_window_inference`.
This PR also allows for different occlusion kernels. We currently have:
- gaussian (actually inverse gaussian): the center of the occluded
region is zero, and towards the edge of the image is unchanged. This
doesn't introduce hard edges into the image, which might undermine the
visualisation process.
- mean_patch: the occluded region is replaced with the mean of the patch
it is occluding.
- mean_img: the occluded region is replaced with the mean of the whole
image (current implementation).
## Changes to input arguments
This PR is backwards incompatible, as using `sliding_window_inference`
means changing the API significantly.
- pad_val: now determined by `mode`
- stride: `overlap` used instead
- per_channel: all channels are done simultaneously
- upsampler: image is no longer downsampled
## Changes to output
Output previously had the shape `B,C,H,W,[D],N` where `C` and `N` were
the number of input and output channels of the network, respectively.
Now, we output the shape `B,N,H,W,[D]` as the `per_channel` feature is
no longer present.
Columns 2-4 are occlusion sensitivity done with Gaussian, mean of patch
and mean of image:

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [x] Breaking change (fix or new feature that would cause existing
functionality to change).
- [x] New tests added to cover the changes.
- [x] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [x] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [x] In-line docstrings updated.
- [x] Documentation updated, tested `make html` command in the `docs/`
folder.
Signed-off-by: Richard Brown <[email protected]>
Signed-off-by: KumoLiu <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Occlusion sensitivity currently works by setting the logits at the center of the occluded region. Even though neighbouring voxels are occluded at the same time, this is not recorded. A better way to do it would be to do an average each time a voxel is occluded. This is taken care of by switching to using
sliding_window_inference.We had to do a bit of hacking to get this to work.
sliding_window_inferencenormally takes a subset of the whole image and infers that. We wantsliding_window_inferenceto tell us which part of the image we should occlude, then we occlude it and infer the whole image. To that end, we use a meshgrid to tell us the coordinates of the regionsliding_window_inferencewants us to occlude. We occlude, infer and then crop back to the size of the region that was requested bysliding_window_inference.This PR also allows for different occlusion kernels. We currently have:
- gaussian (actually inverse gaussian): the center of the occluded region is zero, and towards the edge of the image is unchanged. This doesn't introduce hard edges into the image, which might undermine the visualisation process.
- mean_patch: the occluded region is replaced with the mean of the patch it is occluding.
- mean_img: the occluded region is replaced with the mean of the whole image (current implementation).
Changes to input arguments
This PR is backwards incompatible, as using
sliding_window_inferencemeans changing the API significantly.- pad_val: now determined by
mode- stride:
overlapused instead- per_channel: all channels are done simultaneously
- upsampler: image is no longer downsampled
Changes to output
Output previously had the shape
B,C,H,W,[D],NwhereCandNwere the number of input and output channels of the network, respectively. Now, we output the shapeB,N,H,W,[D]as theper_channelfeature is no longer present.Columns 2-4 are occlusion sensitivity done with Gaussian, mean of patch and mean of image:
Types of changes
./runtests.sh -f -u --net --coverage../runtests.sh --quick --unittests --disttests.make htmlcommand in thedocs/folder.