Skip to content

2290 Add support to update data in Datasets#2436

Merged
Nic-Ma merged 17 commits intoProject-MONAI:devfrom
Nic-Ma:2290-rerun-cachedataset
Jun 28, 2021
Merged

2290 Add support to update data in Datasets#2436
Nic-Ma merged 17 commits intoProject-MONAI:devfrom
Nic-Ma:2290-rerun-cachedataset

Conversation

@Nic-Ma
Copy link
Copy Markdown
Contributor

@Nic-Ma Nic-Ma commented Jun 24, 2021 •
edited
Loading

Fixes #2290 .

Description

This PR added support to update data after several epochs in Datasets.

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jun 24, 2021

/black

@Nic-Ma Nic-Ma changed the title [WIP] 2290 Add support to update data in Datasets 2290 Add support to update data in Datasets Jun 24, 2021
@Nic-Ma Nic-Ma marked this pull request as ready for review June 24, 2021 09:32
@Nic-Ma Nic-Ma requested review from ericspod, rijobro and wyli June 24, 2021 09:32
@Nic-Ma Nic-Ma changed the title 2290 Add support to update data in Datasets [WIP] 2290 Add support to update data in Datasets Jun 25, 2021
@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jun 25, 2021

/black

@Nic-Ma Nic-Ma changed the title [WIP] 2290 Add support to update data in Datasets 2290 Add support to update data in Datasets Jun 25, 2021
@Nic-Ma Nic-Ma mentioned this pull request Jun 25, 2021
Closed
wyli
wyli reviewed Jun 25, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the new method could be set_data instead of update_data? not sure about the overall stability in the multiprocess loading because there will be concurrent read and write for the cache. maybe the PersistentDataset is ok but how about the other datasets?

@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jun 25, 2021

/black

@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jun 25, 2021

Hi @wyli ,

Thanks for your review and suggestions.
I updated the PR according to your comments.
Could you please help review it again?
I already added multi-processing test based on workers of DataLoader:
https://github.com/Project-MONAI/MONAI/pull/2436/files#diff-eea046ab7338e5acd9b172b24ec5a704a755dd10bfae2056b046d598d064ecf9R97

Thanks in advance.

@Nic-Ma
Copy link
Copy Markdown
Contributor Author

Nic-Ma commented Jun 25, 2021

Hi @wyli ,

BTW, I already added the doc-string in CacheDataset to highlight that it requires persistent_workers=False in DataLoader:
https://github.com/Project-MONAI/MONAI/pull/2436/files#diff-e682e654b07b9d753cac73f5a782e70644c82a84d30ebc757ec294a261845108R541

Thanks.

ericspod
ericspod reviewed Jun 25, 2021
View reviewed changes
ericspod
ericspod reviewed Jun 25, 2021
View reviewed changes
ericspod
ericspod reviewed Jun 25, 2021
View reviewed changes
@Nic-Ma Nic-Ma requested review from ericspod and wyli June 25, 2021 23:35
wyli
wyli approved these changes Jun 28, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, just need to change the docstring update_data() to set_data()

@Nic-Ma Nic-Ma enabled auto-merge (squash) June 28, 2021 16:31
@Nic-Ma Nic-Ma merged commit ad80597 into Project-MONAI:dev Jun 28, 2021
@Nic-Ma Nic-Ma deleted the 2290-rerun-cachedataset branch July 2, 2021 23:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wyli wyli wyli approved these changes

@rijobro rijobro Awaiting requested review from rijobro

@ericspod ericspod Awaiting requested review from ericspod ericspod is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

support dynamic data list for the dataset APIs (2/July)

3 participants

@Nic-Ma @wyli @ericspod