5867 add tjbf layer by faebstn96 · Pull Request #5876 · Project-MONAI/MONAI

faebstn96 · 2023-01-19T09:27:11Z

Description

I integrated the trainable joint bilateral filter layer (TJBF) in the MONAI repository as a new PyTorch filter layer. The TJBF contains an analytical gradient derivation toward its filter parameters, its guidance image, and its noisy input image which enables gradient-based optimization within the PyTorch graph. See here for more details on the gradient derivation. Unit tests were added that check the filter output as well as the gradient computations.

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: Fabian Wagner <[email protected]>

wyli · 2023-01-20T08:44:52Z

/black

Signed-off-by: monai-bot <[email protected]>

wyli

thanks, do you think the component names should have _3d in them? all the logic seems to be coupled with three spatial dimensions.

e.g.

MONAI/monai/csrc/filtering/trainable_joint_bilateral/trainable_joint_bilateral.cpp

Lines 38 to 44 in 30b2b11

    
           TrainableJointBilateralFilterForward( 
        
               torch::Tensor inputTensor, 
        
               torch::Tensor guidanceTensor, 
        
               float sigma_x, 
        
               float sigma_y, 
        
               float sigma_z, 
        
               float colorSigma) {

MONAI/monai/networks/layers/filtering.py

Lines 117 to 121 in 30b2b11

    
                   sigma x: trainable standard deviation of the spatial filter kernel in x direction. 
        
                   sigma y: trainable standard deviation of the spatial filter kernel in y direction. 
        
                   sigma z: trainable standard deviation of the spatial filter kernel in z direction.

monai/csrc/filtering/trainable_joint_bilateral/jbf_layer_cpu_backward.cpp

Signed-off-by: Fabian Wagner <[email protected]>

faebstn96 · 2023-01-20T14:21:34Z

Do you mean renaming them to, e.g., sigma_3d_x etc? If you think that makes the code more readable I can do that.

wyli · 2023-01-20T14:41:34Z

I think in general the question is, as a user looking at this set of components

MONAI/monai/networks/layers/filtering.py

Lines 18 to 20 in 2bcb52c

    
           _C, _ = optional_import("monai._C") 
        
           __all__ = ["BilateralFilter", "PHLFilter", "TrainableBilateralFilter", "TrainableJointBilateralFilter"]

the underlying concepts are very similar but the input argument styles are different...

ideally the signatures should be unified, but maybe that's out of the scope of this PR. so I'm thinking alternatively if we could rename the new components as BilateralFilter3D and JointBilateralFilter3D (do the same renaming for the c++ functions), and in the docstring clarify the implementation differences among the components in file monai/networks/layers/filtering.py

faebstn96 · 2023-01-20T15:03:19Z

So, the inputs to the TrainableBilateralFilter and TrainableJointBilateralFilter can also be [B, C, X] or [B, C, X, Y] tensors as there is some unsqueezing done:

MONAI/monai/networks/layers/filtering.py

Lines 234 to 238 in 5c778ad

    
           # C++ extension so far only supports 5-dim inputs. 
        
           if len_input == 3: 
        
               input_tensor = input_tensor.unsqueeze(3).unsqueeze(4) 
        
           elif len_input == 4: 
        
               input_tensor = input_tensor.unsqueeze(4)

However, you are right, the underlying C++/Cuda code operates on three dimensions.
Though, I just recognized we should maybe make the spatial sigma parameters a tuple input like sigma_spatial=(sigma_x, sigma_y, sigma_z) and then check during layer initialization if len(sigma_spatial) equals the spatial dimensions of the input image/guidance tensors, right? This would make the layer flexible for 1D, 2D, and 3D inputs.

wyli · 2023-01-20T15:17:58Z

agreed, if it's renamed to spatial_sigma, then it's consistent with the existing BilateralFilter

Signed-off-by: Fabian Wagner <[email protected]>

faebstn96 · 2023-01-20T16:19:53Z

Great, I just refactored the spatial_sigma initializations accordingly.

wyli

thanks, I'll merge this once the tests are ok (I'll push a workaround for the premerge test issue #5859)

Signed-off-by: monai-bot <[email protected]>

Signed-off-by: Wenqi Li <[email protected]>

wyli · 2023-01-20T16:59:35Z

/build

wyli · 2023-01-20T17:00:58Z

/build

faebstn96 and others added 3 commits January 18, 2023 16:03

Integrate trainable joint bilateral filter with unittests.

59d7bb3

Include filter in docs.

e90e1ae

Signed-off-by: Fabian Wagner <[email protected]>

Merge branch 'Project-MONAI:dev' into 5867-add_tjbf_layer

42491fa

[MONAI] code formatting

30b2b11

Signed-off-by: monai-bot <[email protected]>

wyli reviewed Jan 20, 2023

View reviewed changes

monai/csrc/filtering/trainable_joint_bilateral/jbf_layer_cpu_backward.cpp Outdated Show resolved Hide resolved

faebstn96 and others added 2 commits January 20, 2023 14:16

Merge branch 'Project-MONAI:dev' into 5867-add_tjbf_layer

467371f

Refactor tensor indexer.

2bcb52c

Signed-off-by: Fabian Wagner <[email protected]>

Refactor spatial_sigma initialization.

829af54

Signed-off-by: Fabian Wagner <[email protected]>

wyli approved these changes Jan 20, 2023

View reviewed changes

monai-bot and others added 3 commits January 20, 2023 16:33

[MONAI] code formatting

a5201ec

Signed-off-by: monai-bot <[email protected]>

workaround Project-MONAI#5859

cbc293f

Signed-off-by: Wenqi Li <[email protected]>

style fixes

4e377d3

Signed-off-by: Wenqi Li <[email protected]>

wyli enabled auto-merge (squash) January 20, 2023 16:59

wyli merged commit 2d0e021 into Project-MONAI:dev Jan 20, 2023

faebstn96 deleted the 5867-add_tjbf_layer branch January 31, 2023 09:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

5867 add tjbf layer#5876

5867 add tjbf layer#5876
wyli merged 10 commits intoProject-MONAI:devfrom
faebstn96:5867-add_tjbf_layer

faebstn96 commented Jan 19, 2023

Uh oh!

wyli commented Jan 20, 2023 •

edited

Loading

Uh oh!

wyli left a comment •

edited

Loading

Uh oh!

Uh oh!

faebstn96 commented Jan 20, 2023

Uh oh!

wyli commented Jan 20, 2023 •

edited

Loading

Uh oh!

faebstn96 commented Jan 20, 2023

Uh oh!

wyli commented Jan 20, 2023

Uh oh!

faebstn96 commented Jan 20, 2023

Uh oh!

wyli left a comment

Uh oh!

wyli commented Jan 20, 2023

Uh oh!

wyli commented Jan 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	TrainableJointBilateralFilterForward(
	torch::Tensor inputTensor,
	torch::Tensor guidanceTensor,
	float sigma_x,
	float sigma_y,
	float sigma_z,
	float colorSigma) {

	sigma x: trainable standard deviation of the spatial filter kernel in x direction.

	sigma y: trainable standard deviation of the spatial filter kernel in y direction.

	sigma z: trainable standard deviation of the spatial filter kernel in z direction.

Conversation

faebstn96 commented Jan 19, 2023

Description

Types of changes

Uh oh!

wyli commented Jan 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wyli left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

faebstn96 commented Jan 20, 2023

Uh oh!

wyli commented Jan 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

faebstn96 commented Jan 20, 2023

Uh oh!

wyli commented Jan 20, 2023

Uh oh!

faebstn96 commented Jan 20, 2023

Uh oh!

wyli left a comment

Choose a reason for hiding this comment

Uh oh!

wyli commented Jan 20, 2023

Uh oh!

wyli commented Jan 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wyli commented Jan 20, 2023 •

edited

Loading

wyli left a comment •

edited

Loading

wyli commented Jan 20, 2023 •

edited

Loading