Add Weighted Loss Functions to PyTorch : WMSE, WMAE, and Weighted Huber Loss #132049

tolleybot · 2024-07-29T15:11:52Z

Summary

This pull request introduces new weighted loss functions to the PyTorch library: weighted_huber_loss, wmse_loss, and wmae_loss. These functions allow for precise control over the influence of each sample during training, important for imbalanced data or when certain samples are more significant than others.

Changes

weighted_huber_loss: Huber loss modified to incorporate weights, providing a balance between L1 and L2 loss based on the delta parameter.
wmse_loss (Weighted Mean Squared Error): Applies weights to the standard MSE loss, useful for emphasizing certain samples in regression tasks.
wmae_loss (Weighted Mean Absolute Error): Adjusts MAE loss calculation by including weights, ideal for datasets with outliers.

Code Details

Input Validation: Ensures input, target, and weights tensors match in size to prevent broadcasting errors.
Reduction Options: Supports none, mean, and sum reductions to suit various computational needs.
Backward Compatibility: Maintains support for deprecated arguments size_average and reduce, while encouraging use of the reduction argument.

Usage Example

import torch
input = torch.tensor([0.5, 2.5, 2.0], dtype=torch.float32)
target = torch.tensor([0.0, 2.0, 1.5], dtype=torch.float32)
weights = torch.tensor([1.0, 0.5, 1.5], dtype=torch.float32)

loss = weighted_huber_loss(input, target, weights, delta=1.0)
print(loss)

Feedback on these implementations is welcome; please let me know if further modifications are required.

Resolves #132465

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki

pytorch-bot · 2024-07-29T15:11:55Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/132049

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7167ef8 with merge base 44257c0 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

linux-foundation-easycla · 2024-07-29T15:11:56Z

The committers listed above are authorized under a signed CLA.

✅ login: tolleybot / name: Donald Tolley (a498940, 2e53ebd, 03f7292, 85cd4d4, 04bbfba, d468180, 50d3f18, 2d30e77, 931d90c, 1595b32, fe8a39d, 4a2f323, 6380b0f, b889a1d, bee496c, ae1dd31, b2ef2b3, 353de04, 7167ef8, 6dde75b)

jbschlosser · 2024-07-30T15:36:44Z

Hi @tolleybot, is there an associated issue for this PR?

tolleybot · 2024-08-02T01:21:52Z

Hi @tolleybot, is there an associated issue for this PR?

Yes, I added it the comments above: #132465

mikaylagawarecki

See comment on issue #132465 (comment)

mikaylagawarecki · 2024-08-14T18:53:12Z

seems like tests are failing

torch/nn/functional.py

mikaylagawarecki · 2024-08-26T20:54:30Z

@pytorchbot rebase

pytorchmergebot · 2024-08-26T20:56:01Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-08-26T20:56:05Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict pull/132049/head returned non-zero exit code 1

Rebasing (1/8)
Rebasing (2/8)
Rebasing (3/8)
Auto-merging test/test_nn.py
CONFLICT (content): Merge conflict in test/test_nn.py
Auto-merging torch/nn/functional.py
error: could not apply 065dd99e88... adding weighted huber loss to functional, and adding a new test for weighted huber loss under test_nn.py
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Could not apply 065dd99e88... adding weighted huber loss to functional, and adding a new test for weighted huber loss under test_nn.py

Raised by https://github.com/pytorch/pytorch/actions/runs/10567120792

adamreeve · 2024-09-12T23:32:22Z

Hi @mikaylagawarecki, can you please take another look at this when you have time, is there anything else needed before this can be merged?

mikaylagawarecki

Apologies for the delay, this PR fell out of my queue

torch/nn/functional.py

test/test_nn.py

torch/nn/functional.py

…eighted huber loss under test_nn.py

…and just added "weights" as an optional param to the mse, mae, and huber_loss. I udpated tests to reflect changes

2. removed test_mae_loss and created test_weighted_l1_loss_with_weights 3. fixed linting issues

Co-authored-by: mikaylagawarecki <[email protected]>

…moved by mistake

Add mse_loss to overrides.py

…_variadic signature to include weight updated overrides.py to remove duplicate mse_loss entry

tolleybot · 2024-10-30T17:21:00Z

@mikaylagawarecki I am working to find the correct way to add weights to the l1_loss function. It seems just adding the optional weight parameter is causing issues when running tests

It looks like after rebasing and some small change things have cleared upl.

mikaylagawarecki

Thanks

mikaylagawarecki · 2024-10-31T19:34:57Z

@pytorchbot merge

pytorchmergebot · 2024-10-31T19:36:40Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…ber Loss (pytorch#132049) #### Summary This pull request introduces new weighted loss functions to the PyTorch library: `weighted_huber_loss`, `wmse_loss`, and `wmae_loss`. These functions allow for precise control over the influence of each sample during training, important for imbalanced data or when certain samples are more significant than others. #### Changes - **`weighted_huber_loss`**: Huber loss modified to incorporate weights, providing a balance between L1 and L2 loss based on the `delta` parameter. - **`wmse_loss`** (Weighted Mean Squared Error): Applies weights to the standard MSE loss, useful for emphasizing certain samples in regression tasks. - **`wmae_loss`** (Weighted Mean Absolute Error): Adjusts MAE loss calculation by including weights, ideal for datasets with outliers. #### Code Details - **Input Validation**: Ensures `input`, `target`, and `weights` tensors match in size to prevent broadcasting errors. - **Reduction Options**: Supports `none`, `mean`, and `sum` reductions to suit various computational needs. - **Backward Compatibility**: Maintains support for deprecated arguments `size_average` and `reduce`, while encouraging use of the `reduction` argument. #### Usage Example ```python import torch input = torch.tensor([0.5, 2.5, 2.0], dtype=torch.float32) target = torch.tensor([0.0, 2.0, 1.5], dtype=torch.float32) weights = torch.tensor([1.0, 0.5, 1.5], dtype=torch.float32) loss = weighted_huber_loss(input, target, weights, delta=1.0) print(loss) ``` --- Feedback on these implementations is welcome; please let me know if further modifications are required. Resolves pytorch#132465 Pull Request resolved: pytorch#132049 Approved by: https://github.com/mikaylagawarecki Co-authored-by: mikaylagawarecki <[email protected]>

tolleybot requested review from albanD, jbschlosser and mikaylagawarecki as code owners July 29, 2024 15:11

pytorch-bot bot added the release notes: nn release notes category label Jul 29, 2024

albanD removed their request for review July 29, 2024 15:12

pytorchbot added the open source label Jul 29, 2024

jbschlosser added module: nn Related to torch.nn triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Jul 30, 2024

mikaylagawarecki reviewed Aug 2, 2024

View reviewed changes

mikaylagawarecki reviewed Aug 15, 2024

View reviewed changes

torch/nn/functional.py Outdated Show resolved Hide resolved

tolleybot force-pushed the wmse_wmae branch from 7bd906c to e180221 Compare August 28, 2024 16:55

mikaylagawarecki reviewed Sep 13, 2024

View reviewed changes

torch/nn/functional.py Outdated Show resolved Hide resolved

torch/nn/functional.py Outdated Show resolved Hide resolved

test/test_nn.py Outdated Show resolved Hide resolved

torch/nn/functional.py Outdated Show resolved Hide resolved

torch/nn/functional.py Outdated Show resolved Hide resolved

mikaylagawarecki reviewed Sep 13, 2024

View reviewed changes

torch/nn/functional.py Outdated Show resolved Hide resolved

tolleybot requested review from EikanWang, angelayi, avikchaudhuri, eqy, syed-ahmed, tugsbayasgalan, ydwu4 and zhxchen17 as code owners September 14, 2024 21:02

tolleybot and others added 18 commits October 30, 2024 13:39

adding weighted huber loss to functional, and adding a new test for w…

fe8a39d

…eighted huber loss under test_nn.py

removed unwanted formatting

03f7292

I removed seperate functions for wmse, wmae, and weighted_huber_loss …

85cd4d4

…and just added "weights" as an optional param to the mse, mae, and huber_loss. I udpated tests to reflect changes

Updated misspelling in test_nn.py

931d90c

updated with lintrunner

d468180

1. removed mae_loss, and modified l1_loss to take in optional weights.

b889a1d

2. removed test_mae_loss and created test_weighted_l1_loss_with_weights 3. fixed linting issues

Update functional.py

a498940

Co-authored-by: mikaylagawarecki <[email protected]>

Renaming weights to weight in huber_loss, l1_loss and mse_loss

2d30e77

updated reduction "mean" in huber_loss.

6dde75b

re-added allow_mps=True to instantiate_device_type_tests which was re…

50d3f18

…moved by mistake

Cleaned up linting issues

b2ef2b3

making sure "weights" are canged to "weight"

bee496c

Adding weight to l1_loss and huber_loss in overrides.py.

353de04

Add mse_loss to overrides.py

updated signature of l1_loss, mse_loss, and huber_loss , has_function…

4a2f323

…_variadic signature to include weight updated overrides.py to remove duplicate mse_loss entry

Updated l1_loss signature to use kwargs for weight.

6380b0f

removing kwargs from l1_loss

1595b32

fixing linting for functional.py

ae1dd31

removing optional argument from l1_loss

7167ef8

tolleybot force-pushed the wmse_wmae branch from 01825f3 to 7167ef8 Compare October 30, 2024 13:39

mikaylagawarecki approved these changes Oct 31, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 31, 2024

pytorchmergebot added the merging label Oct 31, 2024

pytorchmergebot added the Merged label Oct 31, 2024

pytorchmergebot closed this in c1e7d85 Oct 31, 2024

pytorchmergebot removed the merging label Oct 31, 2024

Add Weighted Loss Functions to PyTorch : WMSE, WMAE, and Weighted Huber Loss #132049

Add Weighted Loss Functions to PyTorch : WMSE, WMAE, and Weighted Huber Loss #132049

Uh oh!

Conversation

tolleybot commented Jul 29, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Code Details

Usage Example

Uh oh!

pytorch-bot bot commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/132049

✅ No Failures

Uh oh!

linux-foundation-easycla bot commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbschlosser commented Jul 30, 2024

Uh oh!

tolleybot commented Aug 2, 2024

Uh oh!

mikaylagawarecki left a comment

Choose a reason for hiding this comment

Uh oh!

mikaylagawarecki commented Aug 14, 2024

Uh oh!

Uh oh!

mikaylagawarecki commented Aug 26, 2024

Uh oh!

pytorchmergebot commented Aug 26, 2024

Uh oh!

pytorchmergebot commented Aug 26, 2024

Uh oh!

adamreeve commented Sep 12, 2024

Uh oh!

mikaylagawarecki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tolleybot commented Oct 30, 2024

Uh oh!

mikaylagawarecki left a comment

Choose a reason for hiding this comment

Uh oh!

mikaylagawarecki commented Oct 31, 2024

Uh oh!

pytorchmergebot commented Oct 31, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

tolleybot commented Jul 29, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jul 29, 2024 •

edited

Loading

linux-foundation-easycla bot commented Jul 29, 2024 •

edited

Loading