Skip to content

Permanent dataset hashes transforms#4633

Merged
wyli merged 7 commits intoProject-MONAI:devfrom
rijobro:PermanentDataset_hash_transform
Jul 23, 2022
Merged

Permanent dataset hashes transforms#4633
wyli merged 7 commits intoProject-MONAI:devfrom
rijobro:PermanentDataset_hash_transform

Conversation

@rijobro
Copy link
Copy Markdown
Contributor

@rijobro rijobro commented Jul 5, 2022

Description

With this change, PermanentDataset creates a hash of the deterministic transforms and adds this to the filename. Hence, as transforms change, so to should the filename. This reduces errors due to transforms changing during experiments.

fixes #999

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@rijobro rijobro requested review from Nic-Ma, ericspod and wyli July 5, 2022 12:26
rijobro added 2 commits July 5, 2022 14:34
Signed-off-by: Richard Brown <[email protected]>
Signed-off-by: Richard Brown <[email protected]>
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, it looks good to me, I guess it'll close #999?

@rijobro
Copy link
Copy Markdown
Contributor Author

rijobro commented Jul 6, 2022

Fixes #999

@ChenglongWang
Copy link
Copy Markdown

I'm wondering what's the definition of "transforms change". Is it possible to hash the argument changes? Perhaps more detailed docs/examples are required. I think users need to know exactly the data they are processing.

@KumoLiu
Copy link
Copy Markdown
Contributor

KumoLiu commented Jul 7, 2022

Hi @ChenglongWang,
The same transforms with different argument would also make hash change!
Thanks!

@Nic-Ma
Copy link
Copy Markdown
Contributor

Nic-Ma commented Jul 7, 2022

Thanks @KumoLiu for your explanation.
I think most of the transforms are JSON serializable, here only the exceptions will only hash with the transform name.

Thanks.

@wyli
Copy link
Copy Markdown
Contributor

wyli commented Jul 23, 2022

/build

@wyli wyli enabled auto-merge (squash) July 23, 2022 10:10
Signed-off-by: Wenqi Li <[email protected]>
@wyli
Copy link
Copy Markdown
Contributor

wyli commented Jul 23, 2022

/build

@wyli wyli merged commit 6338ea4 into Project-MONAI:dev Jul 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Control Consistency of PersistentDataset

5 participants