Skip to content

add softmax version to focal_loss#6544

Merged
wyli merged 13 commits intoProject-MONAI:devfrom
qingpeng9802:add-softmax
May 27, 2023
Merged

add softmax version to focal_loss#6544
wyli merged 13 commits intoProject-MONAI:devfrom
qingpeng9802:add-softmax

Conversation

@qingpeng9802
Copy link
Copy Markdown
Contributor

@qingpeng9802 qingpeng9802 commented May 23, 2023 •
edited
Loading

Fixes #6510 .

Description

Add softmax version to Focal loss

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@qingpeng9802
Copy link
Copy Markdown
Contributor Author

qingpeng9802 commented May 23, 2023

2
1

For further reference, the implementation reference (pytorch/detectron) has been visualized above.
It is worth to notice that current MONAI's sigmoid focal loss is kind of different from detectron's. Detectron's sigmoid implementation does not include background class by default. See, https://github.com/pytorch/pytorch/blob/6e0c741105b4022fc7c3f8c2919aad9f971beb62/modules/detectron/sigmoid_focal_loss_op.cc#L35-L36
Other implementations also include background class by default like MONAI so this might not be a big issue.
https://github.com/kornia/kornia/blob/edbcae696909bca677c71615a71cb89ba87321e1/kornia/losses/focal.py#L144 and
https://github.com/qubvel/segmentation_models.pytorch/blob/e7cbea0bbd546167b9d1757b2cfeca3fbe26a33e/segmentation_models_pytorch/losses/_functional.py#L36

@wyli Please help review the current implementation. Some unit tests will be added after the review.

@wyli
Copy link
Copy Markdown
Contributor

wyli commented May 23, 2023

thanks @qingpeng9802, it looks good to me, one minor thing is that if we assume target is a one-hot binary tensor, then I think - (1 - input_ls.exp()).pow(gamma) * input_ls * target could be simplified to compute for target==1 only.

qingpeng9802 and others added 2 commits May 26, 2023 21:11
@qingpeng9802
add unittests for focal_loss.py
37adb2d 
DCO Remediation Commit for Qingpeng Li <[email protected]>

I, Qingpeng Li <[email protected]>, hereby add my Signed-off-by to this commit: 5db9e58

Signed-off-by: Qingpeng Li <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
fca52fa 
for more information, see https://pre-commit.ci
@qingpeng9802
Copy link
Copy Markdown
Contributor Author

qingpeng9802 commented May 26, 2023

/black

@monai-bot
[MONAI] code formatting
99f0284 
Signed-off-by: monai-bot <[email protected]>
@qingpeng9802
Copy link
Copy Markdown
Contributor Author

qingpeng9802 commented May 26, 2023

thanks @qingpeng9802, it looks good to me, one minor thing is that if we assume target is a one-hot binary tensor, then I think - (1 - input_ls.exp()).pow(gamma) * input_ls * target could be simplified to compute for target==1 only.

loss: torch.Tensor = torch.where(target>0.5, - (1 - input_ls.exp()).pow(gamma) * input_ls, 0)

and

input_ls[target>0.5]=0.
loss: torch.Tensor = - (1 - input_ls.exp()).pow(gamma) * input_ls

are benchmarked. Both are significantly slower than the original one. It looks like torch.jit.script can fuse element-wise operators efficiently so torch.where or indexing might break the optimization. Therefore, the original one is kept.

@qingpeng9802 qingpeng9802 marked this pull request as ready for review May 26, 2023 14:36
@wyli
Copy link
Copy Markdown
Contributor

wyli commented May 26, 2023

sure, thanks for looking into these @qingpeng9802!

@qingpeng9802
Copy link
Copy Markdown
Contributor Author

qingpeng9802 commented May 27, 2023

/black

@wyli
Copy link
Copy Markdown
Contributor

wyli commented May 27, 2023

/build

@wyli wyli enabled auto-merge (squash) May 27, 2023 18:18
@wyli wyli merged commit 9d6ccce into Project-MONAI:dev May 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wyli wyli wyli approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

implement a softmax version of focal loss

3 participants

@qingpeng9802 @wyli @monai-bot