Tumor FROC Evaluation#1878

Merged

bhashemian merged 47 commits intoProject-MONAI:masterfrom

bhashemian:calculate_froc

Mar 31, 2021

Member

bhashemian commented Mar 28, 2021 •

edited

Loading

Fixes #1872

Description

It a reimplementation of #1873 that

Uses WSIReader to support both OpenSlide and CuCIM
Updates class, methods, parameters, and variables's names to be more representative
Restructures the process for more modularity
Receives the probability map of inference outputs
Applies non-maximal suppression
Adds level and pixel_pacing to the input
Removes redundant calculations
Rewrites compute_evaluation_mask, now is compute_multi_instance_mask
Rewrites compute_itc_list, now is compute_isolated_tumor_cells

Status

Ready

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

bhashemian added 22 commits

March 26, 2021 20:54


          Implement MaskedInferenceWSIDataset for pathology inference

8bb3d6f

Signed-off-by: Behrooz <[email protected]>


          Update pathology init

52e28da

Signed-off-by: Behrooz <[email protected]>


          Update docs

d0fb5d3

Signed-off-by: Behrooz <[email protected]>


          Remove last elemnt of cum_num_patches

53d3edd

Signed-off-by: Behrooz <[email protected]>


          Add unittest with multiple cases for MaskedInferenceWSIDataset

dcba583

Signed-off-by: Behrooz <[email protected]>


          Merge branch 'master' into pathology_inference

096d3ab


          sort imports in init

93d51ea

Signed-off-by: Behrooz <[email protected]>


          Merge branch 'pathology_inference' of https://github.com/behxyz/MONAI …

bce2fa4

…into pathology_inference


          Remove list dataset

8d7e353

Signed-off-by: Behrooz <[email protected]>


          Merge branch 'master' into pathology_inference

2ea8874


          Remove try/except and add type hint

d0abacd

Signed-off-by: Behrooz <[email protected]>


          Convert the sample output to a list

d450da2

Signed-off-by: Behrooz <[email protected]>


          Merge branch 'master' into pathology_inference

de2ac9a


          Remove some type hints

1d02f04

Signed-off-by: Behrooz <[email protected]>


          Implement FROC calcualtion for pathology

3105c82

Signed-off-by: Behrooz <[email protected]>


          Update ProbNMS doctring

56a002d

Signed-off-by: Behrooz <[email protected]>


          Update docs and change namings

3d776e4

Signed-off-by: Behrooz <[email protected]>


          Fix a bug and minor changes

c3cd6af

Signed-off-by: Behrooz <[email protected]>


          Minor changes

bddcb4f

Signed-off-by: Behrooz <[email protected]>


          Fix docstring formatting

d6eca07

Signed-off-by: Behrooz <[email protected]>


          Add a type hint

44dd729

Signed-off-by: Behrooz <[email protected]>


          Implement unittests for EvaluateTumorFROC

90d86dc

Signed-off-by: Behrooz <[email protected]>

bhashemian changed the title ~~[WIP] Tumor FROC Evaluation~~ Tumor FROC Evaluation

bhashemian requested review from Nic-Ma, wyli and yiheng-wang-nv

March 28, 2021 23:56

bhashemian added 3 commits

March 28, 2021 20:13


          Ignore type for np.amax

11a8b6d

Signed-off-by: Behrooz <[email protected]>


          Remove space

c9e5a3f

Signed-off-by: Behrooz <[email protected]>


          Ignore type for range instead of np.amax

486399a

Signed-off-by: Behrooz <[email protected]>

yiheng-wang-nv mentioned this pull request

[WIP] 1872 Implement pathology eval froc class #1873

Closed

6 tasks

bhashemian and others added 2 commits

March 29, 2021 19:27


          Remove print

5d8de3e

Signed-off-by: Behrooz <[email protected]>


          Merge branch 'master' into calculate_froc

340087f

yiheng-wang-nv reviewed

View reviewed changes

monai/apps/pathology/datasets.py Outdated Show resolved Hide resolved

monai/utils/prob_nms.py Outdated Show resolved Hide resolved

monai/apps/pathology/evaluators.py Outdated Show resolved Hide resolved

yiheng-wang-nv and others added 8 commits

March 30, 2021 15:56


          Merge branch 'master' into calculate_froc

721f88f


          Rename TumorFROC and add few type hints

fd7c161

Signed-off-by: Behrooz <[email protected]>


          Rename evaluators to metrics

e83dcda

Signed-off-by: Behrooz <[email protected]>


          Merge branch 'master' of https://github.com/Project-MONAI/MONAI into …

39c3ef8

…calculate_froc


          Remove non-relevant files

57d86ef

Signed-off-by: Behrooz <[email protected]>


          Rename to LesionFROC and minor changes

2e4b5fd

Signed-off-by: Behrooz <[email protected]>


          Update test

f130796

Signed-off-by: Behrooz <[email protected]>


          Merge branch 'calculate_froc' of https://github.com/behxyz/MONAI into…

695be38

… calculate_froc

Member Author

bhashemian commented Mar 30, 2021

@yiheng-wang-nv, is there any other issue that I can address before merging?

bhashemian enabled auto-merge (squash)

March 30, 2021 22:47


          Merge branch 'master' into calculate_froc

wyli reviewed

View reviewed changes

Contributor

wyli left a comment

thanks I put some comments inline

monai/apps/pathology/metrics.py Outdated Show resolved Hide resolved

monai/apps/pathology/metrics.py

+                          box_size=48,
+                      )
+                  def _load_data(self, file_path: str) -> List[Dict]:

Contributor

wyli Mar 31, 2021

could you remove _load_data, this data list should be read before this metric computation, to be consistent with the other metric modules

Member Author

bhashemian Mar 31, 2021

This metrics supports the case that that is loaded too. _load_data is optional in case the data is not loaded and can provide the json path only.

Contributor

wyli Mar 31, 2021

this is to be consistent across the project. the data working group is proposing prototypes for data representation, before we have a concrete design we shouldn't assume a JSON encoded file here

monai/apps/pathology/metrics.py Outdated

+                          threshold: the threashold to fill holes
+                      """
+                      # make sure it is between 0 and 1
+                      assert 0 <= mask.max() <= 1, "The input mask should be a binary mask!"

Contributor

wyli Mar 31, 2021

remove the assert https://deepsource.io/gh/Project-MONAI/MONAI/run/2804f1da-5ea2-4956-940e-a217daca55b5/python/BAN-B101

monai/apps/pathology/metrics.py Outdated


		return froc_score

		def compute_multi_instance_mask(self, mask: np.ndarray, threshold: float):

Contributor

wyli Mar 31, 2021

this should be a generic postprocessing utility as self is not used.

monai/apps/pathology/metrics.py Outdated


		return multi_instance_mask

		def compute_isolated_tumor_cells(self, tumor_mask: np.ndarray, threshold: float) -> List[int]:

Contributor

wyli Mar 31, 2021

this should be a generic postprocessing utility as self is not used.

monai/apps/pathology/metrics.py Outdated

+                  Evaluate with Free Response Operating Characteristic (FROC) score.
+                  Args:
+                      data: either the list of dictionaries containg probability maps (inference result) and

Contributor

wyli Mar 31, 2021

typo "containg"

monai/apps/pathology/metrics.py Outdated

+                  def evaluate(self):
+                      """
+                      Evalaute the detection performance of a model based on the model probability map output,

Contributor

wyli Mar 31, 2021

typo Evalaute

monai/apps/pathology/metrics.py Outdated

+                      Args:
+                          mask: the binary mask array
+                          threshold: the threashold to fill holes

Contributor

wyli Mar 31, 2021

typo threashold

tests/test_lesion_froc.py Show resolved Hide resolved

bhashemian added 5 commits

March 31, 2021 10:15


          Merge branch 'master' into calculate_froc

68e033f


          Address PR comments

9bfbc75

Signed-off-by: Behrooz <[email protected]>


          Update nms

Signed-off-by: Behrooz <[email protected]>


          Merge branch 'master' into calculate_froc

e7baa7a


          Update docs

7e7fe68

Signed-off-by: Behrooz <[email protected]>

wyli approved these changes

View reviewed changes

Contributor

wyli left a comment

thanks!


          Merge branch 'master' into calculate_froc

07190f4

bhashemian merged commit 1314701 into Project-MONAI:master

wyli added a commit to wyli/MONAI that referenced this pull request


          followup of Project-MONAI#1878, fixes tests, remove json loading

5a621b9

Signed-off-by: Wenqi Li <[email protected]>

wyli mentioned this pull request

followup of #1878 #1913

Merged

1 task

bhashemian added a commit that referenced this pull request


          followup of #1878 (#1913)

debc561

* followup of #1878, fixes tests, remove json loading

Signed-off-by: Wenqi Li <[email protected]>

* Update test ordinal numbers

Signed-off-by: Behrooz <[email protected]>

Co-authored-by: Behrooz <[email protected]>

bhashemian deleted the calculate_froc branch

April 12, 2021 20:22

nsrivathsa pushed a commit to nsrivathsa/MONAI that referenced this pull request


          followup of Project-MONAI#1878 (Project-MONAI#1913)

* followup of Project-MONAI#1878, fixes tests, remove json loading

Signed-off-by: Wenqi Li <[email protected]>

* Update test ordinal numbers

Signed-off-by: Behrooz <[email protected]>

Co-authored-by: Behrooz <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>

kevinkwshin commented May 21, 2021

can this module apply to 3d FROC?

Member Author

bhashemian commented Jul 6, 2021

can this module apply to 3d FROC?

Hi @kevinkwshin, thank for your question. This FROC is particular to pathology applications so it only supports 2D FROC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet