Skip to content

ClassesToIndicesd limit the cache memory#6284

Merged
wyli merged 6 commits intoProject-MONAI:devfrom
myron:class_ind
Apr 6, 2023
Merged

ClassesToIndicesd limit the cache memory#6284
wyli merged 6 commits intoProject-MONAI:devfrom
myron:class_ind

Conversation

@myron
Copy link
Copy Markdown
Collaborator

@myron myron commented Apr 4, 2023

to solve: #6285

Currently ClassesToIndicesd can be used to pre-compute and cache the locations of all background/foreground classes.
This significantly speedups the cropping transform RandCropByLabelClassesd, since the coordinates of class voxels do not need to be computed every time.

Unfortunately for an average dataset it requires ~80gb of extra RAM, with an average cache (per image) is on the order of image size itself. (most of it due to background coordinates, or by some large segmentation classes).

This PR clips (optionally) the number of coordinates in each class to a parameter max_indices_per_class. (e.g. 5000), which greatly reduces cache size requirements only to a few MB. Generally these coordinates are used to random choose a cropping center at each epoch, e.g. for 1000 epochs -> 1000 centers per image, so we don't need to store ALL the foreground coordinates, it's sufficient to store just some large random subset.

PS: I wasn't sure if it's better to also change ClassesToIndices (not ClassesToIndicesD), feel free edit this PR.

thank you

Description

A few sentences describing the changes proposed in this pull request.

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

wyli added 2 commits April 5, 2023 11:32
Signed-off-by: Wenqi Li <[email protected]>
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated based on your comments @myron, please have a look, let's merge this soon.

@myron
Copy link
Copy Markdown
Collaborator Author

myron commented Apr 5, 2023

looks great! let's merge. thank you

@wyli wyli enabled auto-merge (squash) April 6, 2023 06:54
@wyli
Copy link
Copy Markdown
Contributor

wyli commented Apr 6, 2023

/build

@wyli wyli merged commit 332bfc0 into Project-MONAI:dev Apr 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants