Skip to content

Tumor FROC Evaluation #1878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 47 commits into from
Mar 31, 2021
Merged

Tumor FROC Evaluation #1878

merged 47 commits into from
Mar 31, 2021

Conversation

bhashemian
Copy link
Member

@bhashemian bhashemian commented Mar 28, 2021 •
edited
Loading

Fixes #1872

Description

It a reimplementation of #1873 that

  • Uses WSIReader to support both OpenSlide and CuCIM
  • Updates class, methods, parameters, and variables's names to be more representative
  • Restructures the process for more modularity
  • Receives the probability map of inference outputs
  • Applies non-maximal suppression
  • Adds level and pixel_pacing to the input
  • Removes redundant calculations
  • Rewrites compute_evaluation_mask, now is compute_multi_instance_mask
  • Rewrites compute_itc_list, now is compute_isolated_tumor_cells

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

bhashemian added 22 commits March 26, 2021 20:54
@bhashemian
Implement MaskedInferenceWSIDataset for pathology inference
8bb3d6f 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Update pathology init
52e28da 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Update docs
d0fb5d3 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Remove last elemnt of cum_num_patches
53d3edd 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Add unittest with multiple cases for MaskedInferenceWSIDataset
dcba583 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Merge branch 'master' into pathology_inference
096d3ab
@bhashemian
sort imports in init
93d51ea 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Merge branch 'pathology_inference' of https://github.com/behxyz/MONAI …
bce2fa4 
…into pathology_inference
@bhashemian
Remove list dataset
8d7e353 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Merge branch 'master' into pathology_inference
2ea8874
@bhashemian
Remove try/except and add type hint
d0abacd 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Convert the sample output to a list
d450da2 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Merge branch 'master' into pathology_inference
de2ac9a
@bhashemian
Remove some type hints
1d02f04 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Implement FROC calcualtion for pathology
3105c82 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Update ProbNMS doctring
56a002d 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Update docs and change namings
3d776e4 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Fix a bug and minor changes
c3cd6af 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Minor changes
bddcb4f 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Fix docstring formatting
d6eca07 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Add a type hint
44dd729 
Signed-off-by: Behrooz <[email protected]>
@bhashemian
Implement unittests for EvaluateTumorFROC
90d86dc 
Signed-off-by: Behrooz <[email protected]>
@bhashemian bhashemian changed the title [WIP] Tumor FROC Evaluation Tumor FROC Evaluation Mar 28, 2021
@yiheng-wang-nv yiheng-wang-nv mentioned this pull request Mar 29, 2021
Closed
6 tasks
yiheng-wang-nv
yiheng-wang-nv reviewed Mar 30, 2021
View reviewed changes
@bhashemian
Copy link
Member Author

@yiheng-wang-nv, is there any other issue that I can address before merging?

@bhashemian bhashemian enabled auto-merge (squash) March 30, 2021 22:47
wyli
wyli reviewed Mar 31, 2021
View reviewed changes
Copy link
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks I put some comments inline

monai/apps/pathology/metrics.py
box_size=48,
)

def _load_data(self, file_path: str) -> List[Dict]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you remove _load_data, this data list should be read before this metric computation, to be consistent with the other metric modules

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This metrics supports the case that that is loaded too. _load_data is optional in case the data is not loaded and can provide the json path only.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is to be consistent across the project. the data working group is proposing prototypes for data representation, before we have a concrete design we shouldn't assume a JSON encoded file here

monai/apps/pathology/metrics.py Outdated
threshold: the threashold to fill holes
"""
# make sure it is between 0 and 1
assert 0 <= mask.max() <= 1, "The input mask should be a binary mask!"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the assert https://deepsource.io/gh/Project-MONAI/MONAI/run/2804f1da-5ea2-4956-940e-a217daca55b5/python/BAN-B101

monai/apps/pathology/metrics.py Outdated

return froc_score

def compute_multi_instance_mask(self, mask: np.ndarray, threshold: float):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be a generic postprocessing utility as self is not used.

monai/apps/pathology/metrics.py Outdated

return multi_instance_mask

def compute_isolated_tumor_cells(self, tumor_mask: np.ndarray, threshold: float) -> List[int]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be a generic postprocessing utility as self is not used.

monai/apps/pathology/metrics.py Outdated
Evaluate with Free Response Operating Characteristic (FROC) score.

Args:
data: either the list of dictionaries containg probability maps (inference result) and
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo "containg"

monai/apps/pathology/metrics.py Outdated

def evaluate(self):
"""
Evalaute the detection performance of a model based on the model probability map output,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo Evalaute

monai/apps/pathology/metrics.py Outdated

Args:
mask: the binary mask array
threshold: the threashold to fill holes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo threashold

wyli
wyli approved these changes Mar 31, 2021
View reviewed changes
Copy link
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@bhashemian bhashemian merged commit 1314701 into Project-MONAI:master Mar 31, 2021
wyli added a commit to wyli/MONAI that referenced this pull request Apr 1, 2021
@wyli
followup of Project-MONAI#1878, fixes tests, remove json loading
5a621b9 
Signed-off-by: Wenqi Li <[email protected]>
@wyli wyli mentioned this pull request Apr 1, 2021
Merged
1 task
bhashemian added a commit that referenced this pull request Apr 1, 2021
@wyli @bhashemian
followup of #1878 (#1913)
debc561 
* followup of #1878, fixes tests, remove json loading

Signed-off-by: Wenqi Li <[email protected]>

* Update test ordinal numbers

Signed-off-by: Behrooz <[email protected]>

Co-authored-by: Behrooz <[email protected]>
@bhashemian bhashemian deleted the calculate_froc branch April 12, 2021 20:22
nsrivathsa pushed a commit to nsrivathsa/MONAI that referenced this pull request Apr 12, 2021
@wyli @bhashemian @nsrivathsa
followup of Project-MONAI#1878 (Project-MONAI#1913)
5113396 
* followup of Project-MONAI#1878, fixes tests, remove json loading

Signed-off-by: Wenqi Li <[email protected]>

* Update test ordinal numbers

Signed-off-by: Behrooz <[email protected]>

Co-authored-by: Behrooz <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
@kevinkwshin
Copy link

can this module apply to 3d FROC?

@bhashemian
Copy link
Member Author

can this module apply to 3d FROC?

Hi @kevinkwshin, thank for your question. This FROC is particular to pathology applications so it only supports 2D FROC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiheng-wang-nv yiheng-wang-nv yiheng-wang-nv left review comments

@wyli wyli wyli approved these changes

@Nic-Ma Nic-Ma Awaiting requested review from Nic-Ma

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add Pathology based FROC metric calculation class
4 participants
@bhashemian @kevinkwshin @wyli @yiheng-wang-nv