Skip to content

Implement FROC metric#1509

Merged
bhashemian merged 14 commits intoProject-MONAI:masterfrom
yiheng-wang-nv:froc
Mar 18, 2021
Merged

Implement FROC metric#1509
bhashemian merged 14 commits intoProject-MONAI:masterfrom
yiheng-wang-nv:froc

Conversation

@yiheng-wang-nv
Copy link
Copy Markdown
Contributor

@yiheng-wang-nv yiheng-wang-nv commented Jan 26, 2021

Signed-off-by: yiheng-wang-nv [email protected]

Implement FROC metric which is used in https://camelyon16.grand-challenge.org/Evaluation/.

Status

Hold

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh --codeformat --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: yiheng-wang-nv <[email protected]>
@yiheng-wang-nv
Copy link
Copy Markdown
Contributor Author

Hi @behxyz , in this MR, I remove the part of compute evaluation mask and ITC, since they are only focusing on the challenge based data format, I'm not sure if they should be added into MONAI. Could you please help to review this MR and if needed, let's discuss about the next step for the implementation. Thanks!

Copy link
Copy Markdown
Member

@bhashemian bhashemian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yiheng-wang-nv Very neat code, I liked it! I proposed few changes. On the other hand, we should think of what else is needed to make this metric usable in MONAI, and maybe starting from the pathology use case. Any thought?

@yiheng-wang-nv
Copy link
Copy Markdown
Contributor Author

yiheng-wang-nv commented Feb 4, 2021

Hi @Nic-Ma @wyli , I met an error like this:

    if torch.is_tensor(probs):
        probs = probs.detach().cpu().numpy()

The above part got the mypy error:
monai/metrics/froc.py:52:17: error: Item "ndarray" of "Union[ndarray, Tensor]" has no attribute "detach" [union-attr]
However, it used mypy 0.8, which is the same as my local environment, and I did not get this error locally. Could you please help to take a look at this issue? Thanks!

The code is in here.

@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Feb 4, 2021

Hi @Nic-Ma @wyli , I met an error like this:

    if torch.is_tensor(probs):
        probs = probs.detach().cpu().numpy()

The above part got the mypy error:
monai/metrics/froc.py:52:17: error: Item "ndarray" of "Union[ndarray, Tensor]" has no attribute "detach" [union-attr]
However, it used mypy 0.8, which is the same as my local environment, and I did not get this error locally. Could you please help to take a look at this issue? Thanks!

The code is in here.

2 points:

  1. you can try to use isinstance(probs, torch.Tensor) instead of torch.is_tensor() and try to submit PR again.
  2. The reason why your local test is OK: your numpy is old, if you update to the latest numpy, you will get the error in mypy.

Thanks.

@yiheng-wang-nv
Copy link
Copy Markdown
Contributor Author

Hi @Nic-Ma @wyli , I met an error like this:

    if torch.is_tensor(probs):
        probs = probs.detach().cpu().numpy()

The above part got the mypy error:
monai/metrics/froc.py:52:17: error: Item "ndarray" of "Union[ndarray, Tensor]" has no attribute "detach" [union-attr]
However, it used mypy 0.8, which is the same as my local environment, and I did not get this error locally. Could you please help to take a look at this issue? Thanks!
The code is in here.

2 points:

  1. you can try to use isinstance(probs, torch.Tensor) instead of torch.is_tensor() and try to submit PR again.
  2. The reason why your local test is OK: your numpy is old, if you update to the latest numpy, you will get the error in mypy.

Thanks.

Thanks Nic,

My running environment is the latest MONAI docker, and the numpy version is 1.19.1, I upgraded it into 1.19.5 but still cannot get the same error.

Signed-off-by: yiheng-wang-nv <[email protected]>
@wyli
Copy link
Copy Markdown
Contributor

wyli commented Feb 4, 2021

the latest numpy 1.20 doesn't support python 3.6 which is the default python of the docker image...

Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I put some comments on the docstring and variable names

Copy link
Copy Markdown
Member

@bhashemian bhashemian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the suggestions are related to name changing to make it more generic and not specific to pathology use cases. The general flow seems to be generic enough but I need to find a couple of use cases to verify. I will add my comments once I find the use cases to check.

@bhashemian bhashemian marked this pull request as ready for review March 18, 2021 17:05
Copy link
Copy Markdown
Member

@bhashemian bhashemian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good to me. Well done, @yiheng-wang-nv!
After this PR, it's time to make this FROC work for pathology inference pipeline.

@bhashemian bhashemian merged commit 466a0bf into Project-MONAI:master Mar 18, 2021
@yiheng-wang-nv yiheng-wang-nv deleted the froc branch March 19, 2021 05:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants