Skip to content

3632 Enhance GridPatchDataset for iterable API and data source#3636

Merged
wyli merged 13 commits intoProject-MONAI:devfrom
Nic-Ma:update-grid-patch
Jan 11, 2022
Merged

3632 Enhance GridPatchDataset for iterable API and data source#3636
wyli merged 13 commits intoProject-MONAI:devfrom
Nic-Ma:update-grid-patch

Conversation

@Nic-Ma
Copy link
Copy Markdown
Contributor

@Nic-Ma Nic-Ma commented Jan 10, 2022

Fixes #3632 .

Description

This PR enhanced the GridPatchDataset to easily follow the new IterableDataset API and added support for iterable data source which was marked TODO.

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jan 10, 2022

/black

@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jan 10, 2022

/build

@Nic-Ma Nic-Ma requested review from ericspod, rijobro and wyli January 10, 2022 17:02
@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jan 11, 2022

/black

@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jan 11, 2022

/build

@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jan 11, 2022

/black

@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jan 11, 2022

/build

1 similar comment
@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jan 11, 2022

/build

@Nic-Ma
[DLMED] update test
a2eea6b 
Signed-off-by: Nic Ma <[email protected]>
@Nic-Ma Nic-Ma force-pushed the update-grid-patch branch from 635bffa to a2eea6b Compare January 11, 2022 14:21
@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jan 11, 2022

/build

wyli
wyli approved these changes Jan 11, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, it looks good to me.

@wyli
Copy link
Copy Markdown
Contributor

wyli commented Jan 11, 2022

for the record the way multiprocessing works changed in this PR, previously was

MONAI/monai/data/grid_dataset.py

Lines 147 to 150 in 83f8b06

# split workload
per_worker = int(np.ceil((iter_end - iter_start) / float(worker_info.num_workers)))
iter_start += worker_info.id * per_worker
iter_end = min(iter_start + per_worker, iter_end)

vs now:

MONAI/monai/data/iterable_dataset.py

Lines 53 to 58 in 83f8b06

num_workers = info.num_workers if info is not None else 1
id = info.id if info is not None else 0
self.source = iter(self.data)
for i, item in enumerate(self.source):
if i % num_workers == id:

@wyli wyli merged commit be1c362 into Project-MONAI:dev Jan 11, 2022
wyli pushed a commit that referenced this pull request Jan 21, 2022
@Nic-Ma @wyli
3632 Enhance GridPatchDataset for iterable API and data source (#3636)
c77c0b1 
* [DLMED] update GridPatchDataset

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix test typo

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] update for windows

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] fix windows test

Signed-off-by: Nic Ma <[email protected]>

* [DLMED] update test

Signed-off-by: Nic Ma <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wyli wyli wyli approved these changes

@ericspod ericspod Awaiting requested review from ericspod ericspod is a code owner

@rijobro rijobro Awaiting requested review from rijobro

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Enhance GridPatchDataset for iterable dataset

2 participants

@Nic-Ma @wyli