Skip to content

[WIP] Adding Stain Normalization Transform for Pathology #1912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 55 commits into from

Conversation

nsrivathsa
Copy link
Contributor

Fixes # .

Description

Added a Transform for CuPy-based Stain Normalization for Pathology.

Status

Work in progress

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@nsrivathsa nsrivathsa changed the title [Draft] Adding Stain Normalization Transform for Pathology [WIP] Adding Stain Normalization Transform for Pathology Mar 31, 2021
@bhashemian bhashemian self-requested a review April 1, 2021 00:17
Copy link
Member

@bhashemian bhashemian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done! It's a great job!
I left few comments but once you add the unittests, I will review it again. Thanks

Copy link
Member

@bhashemian bhashemian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left few comments on cupy. Please let me know if you have any question on that.

Copy link
Member

@bhashemian bhashemian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @nsrivathsa, thanks for addressing all the comments. So far so good! Have you had a chance to work on the test cases?

bhashemian and others added 25 commits April 12, 2021 15:42
Signed-off-by: Behrooz <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Behrooz <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* Rename the prob map producer unittest to match the module

Signed-off-by: Behrooz <[email protected]>

* Change probs_map to prob_map

Signed-off-by: Behrooz <[email protected]>

* Prepend image_inference_outputs with temp to be ignored

Signed-off-by: Behrooz <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Yiheng Wang <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* followup of Project-MONAI#1878, fixes tests, remove json loading

Signed-off-by: Wenqi Li <[email protected]>

* Update test ordinal numbers

Signed-off-by: Behrooz <[email protected]>

Co-authored-by: Behrooz <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* clip variance to be >= 0

Signed-off-by: kate-sann5100 <[email protected]>

* max for torch1.6

Signed-off-by: Wenqi Li <[email protected]>

Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* update to use pytorch2103

Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
…MONAI#1922)

* [DLMED] SaveImage supports patch data

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix flake8 issue

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix flake8 issue

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix flake8 issue

Signed-off-by: Nic Ma <[email protected]>

Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
This reverts commit 4bd26f9.

Signed-off-by: Wenqi Li <[email protected]>

Co-authored-by: Nic Ma <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* [DLMED] add EarlyStop handler

Signed-off-by: Nic Ma <[email protected]>

* [MONAI] python code formatting

Signed-off-by: monai-bot <[email protected]>

* [DLMED] enhance validation handler

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add set_trainer support

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add more check

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix flake8 issue

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] update according to comments

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix flake8 issue

Signed-off-by: Nic Ma <[email protected]>

Co-authored-by: monai-bot <[email protected]>
Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* [DLMED] change to save patch_index in meta_dict

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] remove error import

Signed-off-by: Nic Ma <[email protected]>

* [MONAI] python code formatting

Signed-off-by: monai-bot <[email protected]>

* [DLMED] fix CI tests

Signed-off-by: Nic Ma <[email protected]>

Co-authored-by: monai-bot <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* Enabled partial checkpoint loading

Allowing partial loading of a model via strict=False.

Signed-off-by: Petru-Daniel Tudosiu <[email protected]>

* [MONAI] python code formatting

Signed-off-by: monai-bot <[email protected]>

* [DLMED] simplify strict arg

Signed-off-by: Nic Ma <[email protected]>

Co-authored-by: monai-bot <[email protected]>
Co-authored-by: Nic Ma <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* less warning msg; remove PILImage types

Signed-off-by: Wenqi Li <[email protected]>

* remove engine type

Signed-off-by: Wenqi Li <[email protected]>

* temp tests

Signed-off-by: Wenqi Li <[email protected]>

* add quick py36 37 tests

Signed-off-by: Wenqi Li <[email protected]>

* temp tests

Signed-off-by: Wenqi Li <[email protected]>

* Revert "temp tests"

This reverts commit deaed40.

Signed-off-by: Wenqi Li <[email protected]>

* Revert "temp tests"

This reverts commit 3e8d8f4.

Signed-off-by: Wenqi Li <[email protected]>

* min test exclude senet test

Signed-off-by: Wenqi Li <[email protected]>

* update

Signed-off-by: Wenqi Li <[email protected]>

* temp test

Signed-off-by: Wenqi Li <[email protected]>

* Revert "temp test"

This reverts commit 31f40a0.

Signed-off-by: Wenqi Li <[email protected]>

* update get gpu id

Signed-off-by: Wenqi Li <[email protected]>

* update docsstrings

Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* Add ProbNMS to transforts/post/array

Signed-off-by: Behrooz <[email protected]>

* Implement ProbNMSDict

Signed-off-by: Behrooz <[email protected]>

* Update the usage and add unittests

Signed-off-by: Behrooz <[email protected]>

* Update docs

Signed-off-by: Behrooz <[email protected]>

* Correct docs

Signed-off-by: Behrooz <[email protected]>

* Correct test case

Signed-off-by: Behrooz <[email protected]>

Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* Implement garbage collector handler

Signed-off-by: Behrooz <[email protected]>

* Make trigger_event lower case

Signed-off-by: Behrooz <[email protected]>

* Add unittest for garbage collector

Signed-off-by: Behrooz <[email protected]>

* Update docs

Signed-off-by: Behrooz <[email protected]>

* Exclude from min test

Signed-off-by: Behrooz <[email protected]>

* Fix a typo

Signed-off-by: Behrooz <[email protected]>

* Fix a bug

Signed-off-by: Behrooz <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* [DLMED] add strict_shape option

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add unit tests

Signed-off-by: Nic Ma <[email protected]>

* update test case

Signed-off-by: Wenqi Li <[email protected]>

* fixes test config

Signed-off-by: Wenqi Li <[email protected]>

Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
wyli and others added 27 commits April 12, 2021 15:42
* refactor docker building

Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* Update garbage collection assertion to a more reliable one

Signed-off-by: Behrooz <[email protected]>

* Remove print

Signed-off-by: Behrooz <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* Working ParameterScheduler 

Added a new ParameterScheduler handler and the required tests.

Signed-off-by: Petru-Daniel Tudosiu <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Nic Ma <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* [DLMED] add support for addtional events

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add unit tests

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix typehints

Signed-off-by: Nic Ma <[email protected]>

* [MONAI] python code formatting

Signed-off-by: monai-bot <[email protected]>

* fixes typos

Signed-off-by: Wenqi Li <[email protected]>

Co-authored-by: monai-bot <[email protected]>
Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* fixes Project-MONAI#1965

Signed-off-by: Wenqi Li <[email protected]>

* adds docstring

Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
…#1969)

* [DLMED] enhance doc-strings

Signed-off-by: Nic Ma <[email protected]>

* [MONAI] python code formatting

Signed-off-by: monai-bot <[email protected]>

Co-authored-by: monai-bot <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Alvin Ihsani <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* adding init efficientnet support

Signed-off-by: masadcv <[email protected]>

* fixing flake8 and further refactoring

Signed-off-by: masadcv <[email protected]>

* adding unittests for efficiennet

Signed-off-by: masadcv <[email protected]>

* making unittests backwards compatible python<3.8

Signed-off-by: masadcv <[email protected]>

* fixed kitty unittests file path

Signed-off-by: masadcv <[email protected]>

* adding docstrings and minor refactoring

Signed-off-by: masadcv <[email protected]>

* fix flake8-py3 failing test

Signed-off-by: masadcv <[email protected]>

* generalize drop_connect for n-dim, fix/add unittests, remove assert

Signed-off-by: masadcv <[email protected]>

* fix failing unittest, CC0-license image for test

Signed-off-by: masadcv <[email protected]>

* refactoring code for review

Signed-off-by: masadcv <[email protected]>

* WIP fix mypy type hint errors

Signed-off-by: masadcv <[email protected]>

* fix cuda test error

Signed-off-by: masadcv <[email protected]>

* WIP fix test errors

Signed-off-by: masadcv <[email protected]>

* adding non-default shape tests

Signed-off-by: masadcv <[email protected]>

* remove 3d case from non-default shape test

Signed-off-by: masadcv <[email protected]>

* refactoring and updating docs

Signed-off-by: masadcv <[email protected]>

Co-authored-by: Yiheng Wang <[email protected]>
Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* [DLMED] add TransformInverter handler

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix typo

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add support in SegmentationSaver handler

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix flake8 issue

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix flake8 issue

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix CI test

Signed-off-by: Nic Ma <[email protected]>

* [MONAI] python code formatting

Signed-off-by: monai-bot <[email protected]>

Co-authored-by: monai-bot <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* changelog for 0.5.0

Signed-off-by: Wenqi Li <[email protected]>

* update changelog

Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* [DLMED] add TransformInverter handler

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix typo

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add support in SegmentationSaver handler

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix flake8 issue

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix flake8 issue

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix CI test

Signed-off-by: Nic Ma <[email protected]>

* [MONAI] python code formatting

Signed-off-by: monai-bot <[email protected]>

* [DLMED] save mode into inverse dict

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add unit tests

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix ToTensor inverse issue

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] change the replacement logic into util function

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add more tests

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix flake8

Signed-off-by: Nic Ma <[email protected]>

* [MONAI] python code formatting

Signed-off-by: monai-bot <[email protected]>

Co-authored-by: monai-bot <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* [DLMED] update highlights for 0.5

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add invert transforms

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add more highlights

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add checkpointloader

Signed-off-by: Nic Ma <[email protected]>

* downscale image

Signed-off-by: Wenqi Li <[email protected]>

* downscale image

Signed-off-by: Wenqi Li <[email protected]>

* update docs

Signed-off-by: Wenqi Li <[email protected]>

* update desc

Signed-off-by: Wenqi Li <[email protected]>

* update

Signed-off-by: Wenqi Li <[email protected]>

Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* update convs docstrings

Signed-off-by: Wenqi Li <[email protected]>

* increase tol for cuda crf

Signed-off-by: Wenqi Li <[email protected]>

* update requirements according to docs/requirements

Signed-off-by: Wenqi Li <[email protected]>

* prune image

Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>

Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
…NAI#1988)

* [DLMED] metrics support a list of tensor

Signed-off-by: Nic Ma <[email protected]>

* update workflow test

Signed-off-by: Wenqi Li <[email protected]>

* [DLMED] fix engine.state.output dict copy issue

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] add num_workers

Signed-off-by: Nic Ma <[email protected]>

* fixes typos

Signed-off-by: Wenqi Li <[email protected]>

* update tests

Signed-off-by: Wenqi Li <[email protected]>

* update docstrings

Signed-off-by: Wenqi Li <[email protected]>

* tests

Signed-off-by: Wenqi Li <[email protected]>

Co-authored-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* handling error

Signed-off-by: Wenqi Li <[email protected]>

* temp test

Signed-off-by: Wenqi Li <[email protected]>

* fixes tests

Signed-off-by: Wenqi Li <[email protected]>

* Revert "temp test"

This reverts commit 60661ae.

Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
* update dev version

Signed-off-by: Wenqi Li <[email protected]>

* typo fix

Signed-off-by: Wenqi Li <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Richard Brown <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Behrooz <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
Signed-off-by: Neha Srivathsa <[email protected]>
@bhashemian
Copy link
Member

closing in favor of #1998

@bhashemian bhashemian closed this Apr 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants