Skip to content

1198 efficient differentiable bilateral filter#1375

Merged
charliebudd merged 57 commits intoProject-MONAI:masterfrom
charliebudd:1198-efficient-differentiable-bilateral-filter
Dec 18, 2020
Merged

1198 efficient differentiable bilateral filter#1375
charliebudd merged 57 commits intoProject-MONAI:masterfrom
charliebudd:1198-efficient-differentiable-bilateral-filter

Conversation

@charliebudd
Copy link
Copy Markdown
Collaborator

@charliebudd charliebudd commented Dec 16, 2020 •
edited
Loading

Fixes #1198 .

Description

Adds two bilateral filter algorithms to the c++ extention module. Each have both a cpu and a cuda implementation.
One works by a bruteforce kernel, the other using a more refined approximate solution presented here...https://graphics.stanford.edu/papers/permutohedral/
The desired implementation is chosen via a boolean flag at the python interface level.

There are improvements which could be made but this is long overdue and already a very large PR.
There are also two issues I'm aware of but both are imperceptible at normal scales...

  • The approximate cuda implementation shows some small variablity in its output.
  • Both the approximate cpu and approximate cuda implementations show slight asymetry in the filtered output (This may be an artifact of the approximation method)

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh --codeformat --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@charliebudd charliebudd changed the title 1198 efficient differentiable bilateral filter [WIP]1198 efficient differentiable bilateral filter Dec 16, 2020
@charliebudd
Initial cpu implementation
e382b3f 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
hue speed increase moving away from torch tensors
a04bcc6 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
allowing generic channel dimension size
80ced0e 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
allowing generic dimensionality
3b98676 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
manual testing script for bilateral filter
ecdf753 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
Initial commit of CRF using permutohedral lattice to optimise message…
98f9940 
… passing, implementation from https://github.com/SamuelJoutard/Permutohedral_attention_module

Signed-off-by: charliebudd <[email protected]>
@charliebudd
temporary example scripts demonstrating the usage of crf and phl, and…
193c8d1 
… some data to process

Signed-off-by: charliebudd <[email protected]>
@charliebudd
alternate phl implementation using a more efficient cuda hashtable, c…
19e2c1c 
…ode currently untested due to windows build issues, implementation from https://github.com/SamuelJoutard/Permutohedral_attention_module

Signed-off-by: charliebudd <[email protected]>
@charliebudd
c++ cpu permutohedral lattice implementation
eecd36c 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
comparison script
f534b52 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
REVERSE THIS COMMIT
abfb80f 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
cleaning
3c97af1 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
backwards pass
7bcce81 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
initial cuda brute force kernel
ab85503 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
reverting setup.py
b0097b4 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
working cuda kernel for brute force 2d kernel
a52f496 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
fixing unresolved symbols when compiling without cuda
5f7304c 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
removing macros file
9c19f48 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
placeholder cuda phl implementation
a025f57 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
changing test case color sigma
24a7b9b 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
Initial import of reference cuda phl implementation
32eb18b 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
fixing cuda kernel color weight error
566c43f 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
initial edits to cuda phl code
3a8add5 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
fixing errors in cuda phl implementation
3276ee6 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
removing logging from cuda phl
72b8d35 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
using template data_ptr function
c225505 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
fixed cpu phl implementation
cc6bd4a 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
updating cpu phl to run on arbitrary input dimensions and channles
1d44988 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
2d and 3d testing scripts
9c4a4a3 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
fixing gaussian kernel function for cpu and cuda bruteforce
133ede9 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
ensuring kernel is an odd numbered size
77f2b4a 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
adding tests for precised implementation
523003b 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
adding approximate imlpementation test
2c8d1b7 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
templating implementations based on scalar_type
c5dfbd1 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
cleaning up
80677c8 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
Merge branch 'master' of https://github.com/charliebudd/MONAI into 11…
20af55b 
…98-efficient-differentiable-bilateral-filter
@charliebudd
code formatting
e526ce8 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
removing half precision implementation for permutohedral due to error…
1ca9762 
…s caused on windows

Signed-off-by: charliebudd <[email protected]>
@charliebudd
skipping cuda tests if cuda missing
e12c49f 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
Merge branch 'master' into 1198-efficient-differentiable-bilateral-fi…
3018c6a 
…lter
@charliebudd
reformating
67baedd 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
adding unit test skip based on cpp extention availablility
3014534 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
removing unused import
83d1d03 
Signed-off-by: charliebudd <[email protected]>
@charliebudd
changing to use of THCatomic add
978a80a 
Signed-off-by: charliebudd <[email protected]>
wyli
wyli reviewed Dec 18, 2020
View reviewed changes
@charliebudd
adding missing licenses
39b007e 
Signed-off-by: charliebudd <[email protected]>
@wyli
Copy link
Copy Markdown
Contributor

wyli commented Dec 18, 2020 •
edited
Loading

/black

@charliebudd
clang reformat
5b557bf 
Signed-off-by: charliebudd <[email protected]>
@charliebudd charliebudd changed the title [WIP]1198 efficient differentiable bilateral filter 1198 efficient differentiable bilateral filter Dec 18, 2020
wyli
wyli approved these changes Dec 18, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @charliebudd this is very nice, would be great to have a short notebook demo

@charliebudd charliebudd merged commit 70be9ab into Project-MONAI:master Dec 18, 2020
@charliebudd charliebudd deleted the 1198-efficient-differentiable-bilateral-filter branch December 18, 2020 14:24
@wyli wyli mentioned this pull request Dec 18, 2020
Closed
@charliebudd charliebudd restored the 1198-efficient-differentiable-bilateral-filter branch December 18, 2020 15:00
@charliebudd charliebudd deleted the 1198-efficient-differentiable-bilateral-filter branch December 18, 2020 15:18
@faebstn96
Copy link
Copy Markdown
Contributor

faebstn96 commented Mar 29, 2021

Hi @charliebudd ,
Thanks for your implementation. Unfortunately, I am failing to get the layer running. When trying to call the forward pass of the bilateral filter with
filter_layer = monai.networks.layers.BilateralFilter(spatial_sigma=5, color_sigma=0.5, fast_approx=True)
filter_layer.apply(tensor_in) # tensor_in: arbitrary 3,4 or 5 dim torch tensor

I get problems with the save_for_backward function, as it does not accept the filter parameters (I think save_for_backward is actually made for storing the input tensor and not for the parameters):
TypeError: save_for_backward can only save variables, but argument 0 is of type int

Wouldn't it be easier to implement the bilateral filter class as a nn.Module instead of torch.autograd.Function (like, e.g., the Gaussian filter)? Then, one could store the parameters in an instance variable.

Thanks for your help!

@faebstn96
Copy link
Copy Markdown
Contributor

faebstn96 commented Mar 29, 2021

I just recognized, you can just assign the parameters directly to ctx instead of using save_for_backward. E.g., like
ctx.spatial_sigma = spatial_sigma

Then the layer at least does not throw an error.

@charliebudd
Copy link
Copy Markdown
Collaborator Author

charliebudd commented Mar 29, 2021 •
edited
Loading

Indeed, thanks for pointing out this out. I have made the changes and have drafted a PR #1888. Sorry for the error.

@faebstn96
Copy link
Copy Markdown
Contributor

faebstn96 commented Mar 29, 2021

Thank you so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wyli wyli wyli approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Efficient, differentiable bilateral filter

3 participants

@charliebudd @wyli @faebstn96