Absent class miou fix by Isalia20 · Pull Request #2892 · Lightning-AI/torchmetrics

Isalia20 · 2025-01-03T20:18:44Z

What does this PR do?

This is not the final version. I wanted to discuss the expected outputs for different edge cases and how to deal with them. I haven't included tests for these edge cases because monAI deals with such cases differently than what is proposed in this PR/issue, therefore adding a test and checking it against the monAI value would lead to failing tests. I'll post different inputs and what kind of outputs we should expect in the followup comment.

Before submitting

Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
Did you read the contributor guideline, Pull Request section?
Did you make sure to update the docs?
Did you write any new necessary tests? See above

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun? partly 😄

Make sure you had fun coding 🙃

📚 Documentation preview 📚: https://torchmetrics--2892.org.readthedocs.build/en/2892/

Isalia20 · 2025-01-03T20:24:51Z

Case 1: Perfect prediction

import torch
from torchmetrics.segmentation import MeanIoU
from torchmetrics.functional.segmentation.mean_iou import mean_iou

num_classes = 3
metric_one = MeanIoU(num_classes=num_classes, per_class=False, input_format='index')
metric_two = MeanIoU(num_classes=num_classes, per_class=True, input_format='index')
target = torch.tensor([
    [0, 1],  # Ground truth: class 0, class 1
    [1, 0],  # Ground truth: class 1, class 0
    [2, 2],  # Ground truth: class 2, class 2
])
preds = torch.tensor([
    [0, 1],  # Predictions: class 0, class 1
    [1, 0],  # Predictions: class 1, class 0
    [2, 2],  # Predictions: class 2, class 2
])
metric_one.update(preds, target)
miou_per_class_one = metric_one.compute()
metric_two.update(preds, target)
miou_per_class_two = metric_two.compute()

print(miou_per_class_one)
print(miou_per_class_two)

Returns:

tensor(1.)
tensor([1., 1., 1.])

Case 2 Perfect prediction but one completely absent class

import torch
from torchmetrics.segmentation import MeanIoU
from torchmetrics.functional.segmentation.mean_iou import mean_iou

num_classes = 4
metric_one = MeanIoU(num_classes=num_classes, per_class=False, input_format='index')
metric_two = MeanIoU(num_classes=num_classes, per_class=True, input_format='index')
target = torch.tensor([
    [0, 1],  # Ground truth: class 0, class 1
    [1, 0],  # Ground truth: class 1, class 0
    [2, 2],  # Ground truth: class 2, class 2
])
preds = torch.tensor([
    [0, 1],  # Predictions: class 0, class 1
    [1, 0],  # Predictions: class 1, class 0
    [2, 2],  # Predictions: class 2, class 2
])
metric_one.update(preds, target)
miou_per_class_one = metric_one.compute()
metric_two.update(preds, target)
miou_per_class_two = metric_two.compute()

print(miou_per_class_one)
print(miou_per_class_two)

Returns:

tensor(nan)
tensor([1., 1., 1., nan])

Case 3: Completely wrong predictions(Probably same as the old one)

import torch
from torchmetrics.segmentation import MeanIoU

num_classes = 3
metric_one = MeanIoU(num_classes=num_classes, per_class=False, input_format='index')
metric_two = MeanIoU(num_classes=num_classes, per_class=True, input_format='index')
target = torch.tensor([
    [0, 1],  # Ground truth: class 0, class 1
    [1, 0],  # Ground truth: class 1, class 0
    [2, 2],  # Ground truth: class 2, class 2
])
preds = torch.tensor([
    [1, 2],  # Predictions: all wrong
    [2, 1],  # Predictions: all wrong
    [0, 1],  # Predictions: all wrong
])
metric_one.update(preds, target)
miou_per_class_one = metric_one.compute()
metric_two.update(preds, target)
miou_per_class_two = metric_two.compute()

print(miou_per_class_one)
print(miou_per_class_two)

Returns

tensor(0.)
tensor([0., 0., 0.])

codecov · 2025-01-03T20:38:58Z

Codecov Report

Attention: Patch coverage is 78.78788% with 7 lines in your changes missing coverage. Please review.

Project coverage is 35%. Comparing base (9e08ff2) to head (124c377).
Report is 75 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (9e08ff2) and HEAD (124c377). Click for more details.

HEAD has 6 uploads less than BASE

Flag BASE (9e08ff2) HEAD (124c377)

gpu 3 0

unittest 3 0

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #2892     +/-   ##
========================================
- Coverage      70%     35%    -35%     
========================================
  Files         355     341     -14     
  Lines       19662   19495    -167     
========================================
- Hits        13782    6757   -7025     
- Misses       5880   12738   +6858

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Isalia20 · 2025-01-06T19:24:08Z

While it's marked WIP I still need comments from you @Borda . I posted some cases and wonder how I should proceed

Isalia20 · 2025-01-16T19:02:42Z

Need a review on this @Borda

Borda · 2025-02-14T22:45:02Z

Need a review on this @Borda

sure, let me have look next week :)

Borda · 2025-03-10T13:12:24Z

@SkafteNicki mind have look, pls ^^ 🐿️

SkafteNicki · 2025-03-10T13:49:03Z

@Isalia20 no strong opinions from my side, but here is my take:

Agree that if there is a perfect match in input/output then the scores should be 1
Agree that the correct value for the absent class in principal should be nan because we end up computing 0/0 in this case. However, we received feedback time and time again (for other metrics) that they want these cases to be ignored because one nan may kill a training process. So I am thinking that maybe the output should still be tensor(1) and instead a warning should be given to the users that nan values were computed but have been filtered
Agree that the value should be zero in this case

Borda · 2025-03-18T20:50:49Z

@Isalia20 @SkafteNicki do you think we could finish it for the next release this week? 🦩

Isalia20 · 2025-03-19T07:15:09Z

One point here is that MonAI(which is used for validating the outputs of this) is used in tests. For such cases the behaviour is different from what I have listed in above cases. So my question is, can I use direct value comparison for such cases and disregard MonAI's output?

Borda · 2025-03-19T08:20:59Z

One point here is that MonAI(which is used for validating the outputs of this) is used in tests. For such cases the behaviour is different from what I have listed in above cases. So my question is, can I use direct value comparison for such cases and disregard MonAI's output?

in general yes, we use validation with standard references or manually computed values...
do you have a specific one in mind?

Isalia20 · 2025-03-19T10:05:10Z

Yes, I've put the cases in the above comment.

Here:

Case 1: Perfect prediction

import torch
from torchmetrics.segmentation import MeanIoU
from torchmetrics.functional.segmentation.mean_iou import mean_iou

num_classes = 3
metric_one = MeanIoU(num_classes=num_classes, per_class=False, input_format='index')
metric_two = MeanIoU(num_classes=num_classes, per_class=True, input_format='index')
target = torch.tensor([
    [0, 1],  # Ground truth: class 0, class 1
    [1, 0],  # Ground truth: class 1, class 0
    [2, 2],  # Ground truth: class 2, class 2
])
preds = torch.tensor([
    [0, 1],  # Predictions: class 0, class 1
    [1, 0],  # Predictions: class 1, class 0
    [2, 2],  # Predictions: class 2, class 2
])
metric_one.update(preds, target)
miou_per_class_one = metric_one.compute()
metric_two.update(preds, target)
miou_per_class_two = metric_two.compute()

print(miou_per_class_one)
print(miou_per_class_two)

Returns:

tensor(1.)
tensor([1., 1., 1.])

Case 2 Perfect prediction but one completely absent class

import torch
from torchmetrics.segmentation import MeanIoU
from torchmetrics.functional.segmentation.mean_iou import mean_iou

num_classes = 4
metric_one = MeanIoU(num_classes=num_classes, per_class=False, input_format='index')
metric_two = MeanIoU(num_classes=num_classes, per_class=True, input_format='index')
target = torch.tensor([
    [0, 1],  # Ground truth: class 0, class 1
    [1, 0],  # Ground truth: class 1, class 0
    [2, 2],  # Ground truth: class 2, class 2
])
preds = torch.tensor([
    [0, 1],  # Predictions: class 0, class 1
    [1, 0],  # Predictions: class 1, class 0
    [2, 2],  # Predictions: class 2, class 2
])
metric_one.update(preds, target)
miou_per_class_one = metric_one.compute()
metric_two.update(preds, target)
miou_per_class_two = metric_two.compute()

print(miou_per_class_one)
print(miou_per_class_two)

Returns:

tensor(nan)
tensor([1., 1., 1., nan])

Case 3: Completely wrong predictions(Probably same as the old one)

import torch
from torchmetrics.segmentation import MeanIoU

num_classes = 3
metric_one = MeanIoU(num_classes=num_classes, per_class=False, input_format='index')
metric_two = MeanIoU(num_classes=num_classes, per_class=True, input_format='index')
target = torch.tensor([
    [0, 1],  # Ground truth: class 0, class 1
    [1, 0],  # Ground truth: class 1, class 0
    [2, 2],  # Ground truth: class 2, class 2
])
preds = torch.tensor([
    [1, 2],  # Predictions: all wrong
    [2, 1],  # Predictions: all wrong
    [0, 1],  # Predictions: all wrong
])
metric_one.update(preds, target)
miou_per_class_one = metric_one.compute()
metric_two.update(preds, target)
miou_per_class_two = metric_two.compute()

print(miou_per_class_one)
print(miou_per_class_two)

Returns

tensor(0.)
tensor([0., 0., 0.])

Isalia20 · 2025-03-22T23:22:42Z

Ready for review, I think failing tests are unrelated. Instead of choosing to go with Nan's, I chose to go with -1.0's. If the user is requesting a single number(not per_class) then the class which wasn't present is ignored and iou is calculated for all other classes

…orchmetrics into absent-class-miou-fix

Isalia20 · 2025-03-24T16:45:01Z

And the tests are failing due to PyTDC here as well :/

Copilot

Pull Request Overview

A PR to fix the handling of absent classes in the MeanIoU metric, ensuring that classes with no presence return a value of -1 instead of 0 and the state tensors are properly initialized.

Updated state initialization and update logic in the MeanIoU class to use correctly shaped tensors.
Adjusted the computation in both the class-based and functional implementations to reflect the desired behavior for absent classes.
Added new tests covering absent-class and perfect prediction scenarios using the "index" input format.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
tests/unittests/segmentation/test_mean_iou.py	Added tests verifying absent-class handling and perfect predictions with index tensor inputs.
src/torchmetrics/segmentation/mean_iou.py	Revised state initialization and update/compute logic to correctly handle absent classes.
src/torchmetrics/functional/segmentation/mean_iou.py	Refactored computation and argument reshaping to align functional output with the class API.
CHANGELOG.md	Updated changelog to document the fix for absent class miou.

Comments suppressed due to low confidence (3)

src/torchmetrics/segmentation/mean_iou.py:158

Replacing the per_class parameter with a fixed zero_division value in _mean_iou_compute changes the behavior compared to the previous implementation. Please verify that this modification produces the expected IoU values for both per-class and aggregated cases.

score = _mean_iou_compute(intersection, union, zero_division=0.0)

src/torchmetrics/functional/segmentation/mean_iou.py:169

The reduction operation in the non-per_class branch assumes that scores and valid_classes have an extra dimension. Verify that this reduction remains valid for all expected input tensor shapes, or consider adding explicit handling for different shapes.

return scores.nansum(dim=-1) / valid_classes.sum(dim=-1)

tests/unittests/segmentation/test_mean_iou.py:141

[nitpick] The use of .mean(dim=0) assumes that the functional implementation returns a tensor with a batch dimension. Consider adding a clarifying comment or explicit verification that this reduction is appropriate for all tested input scenarios.

functional_miou = mean_iou(preds, target, num_classes=3, per_class=True, input_format="index").mean(dim=0)

…orchmetrics into absent-class-miou-fix

Isalia20 · 2025-03-31T09:11:10Z

resolved the comments, should be good to merge. Those 2 tests failed on purging cached refs stage

* first working version, some errots on tests * functional part rewritten as well * remove extra code added in tests for now * proper mean calculation * update docstring of a function * fix some bugs * add device to the num batches * PR comments --------- Co-authored-by: Nicki Skafte Detlefsen <[email protected]> (cherry picked from commit 5746b7a)

Isalia20 added 3 commits January 3, 2025 23:44

first working version, some errots on tests

8596130

functional part rewritten as well

68a1d06

remove extra code added in tests for now

998a2bc

Isalia20 requested review from Borda, SkafteNicki, justusschock and stancld as code owners January 3, 2025 20:18

proper mean calculation

b67a5b6

Borda changed the title ~~Absent class miou fix~~ Absent class miou fix [WIP] Jan 6, 2025

Merge branch 'master' into absent-class-miou-fix

e6dcbf6

Merge branch 'master' into absent-class-miou-fix

7e52f05

SkafteNicki assigned Isalia20 Mar 10, 2025

SkafteNicki added the bug / fix Something isn't working label Mar 10, 2025

SkafteNicki added this to the v1.6.x milestone Mar 10, 2025

Merge branch 'master' into absent-class-miou-fix

ec4bcfb

SkafteNicki modified the milestones: v1.6.x, future Mar 19, 2025

mergify Bot added the has conflicts label Mar 19, 2025

Borda force-pushed the master branch from cc76a81 to 4b07b79 Compare March 20, 2025 11:57

Isalia20 added 5 commits March 23, 2025 02:11

more bugfixes

cb2f776

doctest fixes

2092e0d

-1 instead of outputting nan

b873b32

doctest -1 as well

df65b1f

normalize whitespaces

b6d40e2

Borda force-pushed the master branch from 444126b to 627121a Compare March 24, 2025 08:01

mergify Bot added the has conflicts label Mar 24, 2025

Merge branch 'master' into absent-class-miou-fix

31140f1

mergify Bot removed the has conflicts label Mar 24, 2025

Isalia20 added 2 commits March 24, 2025 20:41

add device to the num batches

c4c5af6

Merge branch 'absent-class-miou-fix' of https://github.com/Isalia20/t…

2e7de19

…orchmetrics into absent-class-miou-fix

Merge branch 'master' into absent-class-miou-fix

d55aba8

Borda requested a review from Copilot March 26, 2025 21:15

Copilot AI reviewed Mar 26, 2025

View reviewed changes

mergify Bot added the has conflicts label Mar 31, 2025

Merge branch 'master' into absent-class-miou-fix

be03889

mergify Bot removed the has conflicts label Mar 31, 2025

SkafteNicki reviewed Mar 31, 2025

View reviewed changes

Isalia20 added 2 commits March 31, 2025 12:51

PR comments

17ba136

Merge branch 'absent-class-miou-fix' of https://github.com/Isalia20/t…

124c377

…orchmetrics into absent-class-miou-fix

SkafteNicki approved these changes Mar 31, 2025

View reviewed changes

SkafteNicki changed the title ~~Absent class miou fix [WIP]~~ Absent class miou fix Mar 31, 2025

mergify Bot added the ready label Mar 31, 2025

Borda approved these changes Mar 31, 2025

View reviewed changes

Borda merged commit 5746b7a into Lightning-AI:master Mar 31, 2025

Uh oh!

Conversation

Isalia20 commented Jan 3, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Did you have fun? partly 😄

Uh oh!

Isalia20 commented Jan 3, 2025 • edited by Borda Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Case 1: Perfect prediction

Returns:

Case 2 Perfect prediction but one completely absent class

Returns:

Case 3: Completely wrong predictions(Probably same as the old one)

Returns

Uh oh!

codecov Bot commented Jan 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Isalia20 commented Jan 6, 2025

Uh oh!

Isalia20 commented Jan 16, 2025

Uh oh!

Borda commented Feb 14, 2025

Uh oh!

Borda commented Mar 10, 2025

Uh oh!

SkafteNicki commented Mar 10, 2025

Uh oh!

Borda commented Mar 18, 2025

Uh oh!

Isalia20 commented Mar 19, 2025

Uh oh!

Borda commented Mar 19, 2025

Uh oh!

Isalia20 commented Mar 19, 2025

Case 1: Perfect prediction

Returns:

Case 2 Perfect prediction but one completely absent class

Returns:

Case 3: Completely wrong predictions(Probably same as the old one)

Returns

Uh oh!

Isalia20 commented Mar 22, 2025

Uh oh!

Isalia20 commented Mar 24, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Isalia20 commented Mar 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Isalia20 commented Jan 3, 2025 •

edited by github-actions Bot

Loading

Isalia20 commented Jan 3, 2025 •

edited by Borda

Loading

codecov Bot commented Jan 3, 2025 •

edited

Loading