-
Notifications
You must be signed in to change notification settings - Fork 133
Fix LsdirHR download #866
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix LsdirHR download #866
Conversation
|
Note 5: |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #866 +/- ##
==========================================
+ Coverage 84.51% 84.53% +0.02%
==========================================
Files 208 208
Lines 21179 21175 -4
Branches 2885 2883 -2
==========================================
+ Hits 17899 17900 +1
+ Misses 2362 2358 -4
+ Partials 918 917 -1 ☔ View full report in Codecov by Sentry. |
|
Better solution for train and ensuring correct behavior of But this would break backward compatibility for users who previously downloaded the dataset.. |
Andrewwango
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @nucli-vicky for this, and for closing an old issue! Just one minor coment.
Thanks for the details - I think a large portion of users will want to anyway download the dataset themselves via GUI / have it already downloaded, since it's so big. The auto download is more necessary for the smaller, quick-debug datasets.
|
This should now also fix the broken doc build from #869 |
Andrewwango
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
/gpu-tests |
|
🚀 GPU test workflows successfully triggered! |
|
✅ GPU docs passed |
|
❌ GPU tests failed |
* 3D Total variation priors (#745) * 3D denoising variational * comments * comments jeremy * Remove concurrency settings from gpu_docs.yml (#850) * Remove concurrency settings from gpu_docs.yml final CI fixes * Remove concurrency settings from test_gpu.yml Removed concurrency settings from the GPU test workflow. * Refine Codecov upload condition for Linux only * Update Codecov upload condition in workflow Restrict Codecov upload to Linux with Python 3.10 only. * Fix astra test failing on GPU (#841) * fix normalize threshold in astra tests * black * Fix test transform on GPU (#852) Co-authored-by: Andrew Wang <[email protected]> * Fix broken download Set14 (#845) * . * fix * adapt docs & use utils * changelog * Fix py3.9 not supported by recent coverage version (#856) * . * . * Follow up to #856 (#857) * Robustify CI for Python 3.9 (#858) * Robustify CI for Python 3.9 * Update test_pr.yml * Fix pip install command for Python 3.9 * Update test_pr.yml * Update CI workflow for deepinv installation Refactor installation steps for deepinv in CI workflow to handle Python 3.9 compatibility. * Add compatibility step for Python 3.9 in workflow Add step to modify pyproject.toml for Python 3.9 compatibility. * Refactor deepinv installation steps in workflow Removed conditional installation for Python 3.9 and adjusted the installation step for deepinv dependencies. * Support 3D patches datasets (#806) * Nibabel test hotfix (#863) * Nibabel test hotfix * Require 'blosc2' for the test dataset Added a dependency check for 'blosc2' in the test. * Typing optim (#827) * distance typed * dpir * data fidelity * fix callable * bregman * fix callable distance * epll * start optimizers.py * phase retrieval * start utils * utils fixes * fix circular import from physics * bregman returns * data fid phsycis & return * distance returns * epll returns * opt & phase retrieval * return spectral wrapper * future annotations * _reconstruction_step * utils * add suggestions * finish optimizers.py * Gentler loader check (#871) * Gentler loader check * remove codecov from test * black * add wee comment * Fix error handling for CBSD68 dataset (#868) * fix error handling * fix * suggestions * add PR #723 in CHANGELOG; fix some minor typos (#860) * add PR #723 in CHANGELOG; fix some minor typos * update CHANGELOG * Fix DDRM citation to reference Kawar et al. 2022 (#876) * Initial plan * Fix DDRM reference to cite Kawar et al. 2022 Co-authored-by: tachella <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: tachella <[email protected]> * Allow for complex data inside Wavelets (#738) * Update mri.py * Update mri.py * Added Complex support * Added support for complex data * reformat * Undo MRI change * More fixes * More fixes * yet more fixes, gosh why am i like this * Finally done * Take in comments * fix psnr and add performance checks --------- Co-authored-by: German Shâma Wache <[email protected]> Co-authored-by: Jérémy Scanvic <[email protected]> * add self-sup eval + history fixes (#777) * add self-sup eval + history fixes * changelog * avoid using losses as metrics + add explicit model inference with .eval + trainer tests * add docs warning * remove loss docstring from metric * add missing docstrings * small fix * doc fixes * doc fixes * fixes * fix adversarial * fixes * fix bugs * black * reblack * jeremy's comments * improve docs explanation * unsure + community + generate_datasets fix * early_stop int + fix doc typo * fix docs * add pragma * andrew's comments * docstring compute_metrics * change names + test * still some fixes to go * black * add train_metrics_history * fix wandb bug * black + docs fixes * black + docs fixes * fix * fix class link * re-order * Update deepinv/training/trainer.py Co-authored-by: Andrew Wang <[email protected]> * andrew's comments * andrew's comments * fix julian issue * docstring fixes --------- Co-authored-by: Andrew Wang <[email protected]> * Add 3D variants of CNN networks (#869) * dncnn * gan * . * ruff check * fix ordering test_pad * maxpool unet nd * icnn * dsccp * PDNet * fix icnn * fix dim drunet * pragma * fix dip * test dsccp 3D * icnn 3D test * more tests * test PDNet * tests GAN * . * test all modes drunet + fix bug from main * changelog * clean dummy * remove c variable * fix doc rician noise * 3d models -> denoisers * warnings * fixdim doc * userguide * syntax * Fix LsdirHR download (#866) * . * ruff + fix docs * add proposed train glob * changelog * fix docstring randompatchsampler * fix sphynx? * pragma * pragma * fix denoiser docs * `dinv.io` module and clean up utils (#768) * start develpoing io submodule + modify examples * add scipy load mat * fix doc * add references to DIP demo (idk why this hasn't been added before) * add raster * io -> io_utils, signal -> signals * checks * ISMRMD loading * add all tests * fix optional deps * add new example * fix example * changelog * fix docs * oops * fix doc * update typing * remove load_np and load_nifit in preparation for merge from vicky PR * black * test assert in doctest * add nifti vicky comment + io_utils -> io * pydicom rescale + lidc uses load_dicom * fix test * fix typing in io and add kwargs * fix sphinx typing * fix test * fix doc * vicky comments * vicky comment * fix doc * blosc2 test * PET comment * Minor README fix (#882) * fix (#885) Co-authored-by: mh-nguyen712 <[email protected]> * Minor trainer docstring fixes (#884) * io -> utils.io * improve docstring * very minor fix * Fix formatting in contributing section of README (#888) * Fix #887 (#890) * Move changelog.rst to docs (#889) * init * fix links * fix links * fix double changelog * change pytorch link * remove llm bs * black * fix docs * Update .github/workflows/release.yml Co-authored-by: Minh Hai Nguyen <[email protected]> * fix ghs * fix ghs --------- Co-authored-by: Minh Hai Nguyen <[email protected]> * Enable ruff rule N805 (#892) * enable ruff rule N805 Co-authored-by: Jérémy Scanvic <[email protected]> * ruff check --fix --unsafe-fixes Co-authored-by: Jérémy Scanvic <[email protected]> --------- Co-authored-by: Andrewwango <[email protected]> * Fix announcement text in conf.py (#901) * Update link to changelog in PR template (#903) * Fix docstring formatting in BaseSDE class (#902) * Refactor release workflow (#914) * Refactor release workflow by removing CI checks Remove CI status check step from release workflow * fixes * fixes * black * Release v0.3.6 (#917) Co-authored-by: tachella <[email protected]> * Fix license formatting in CITATION.cff (#920) * Fix license formatting in CITATION.cff * Create workflow to validate CITATION.cff Add GitHub Actions workflow to validate CITATION.cff file * Add check_citation.yml workflow file * Fix license format in CITATION.cff * Add DOI and version to CITATION.cff * Warn inconsistent upsampling parameters in SwinIR (#909) * add a warning to SwinIR * update the changelog * let test be skipped when timm is absent (#922) * Enable ruff rule E402 (#910) * enable rule E402 Co-authored-by: Jérémy Scanvic <[email protected]> * ignore E402 in the examples Co-authored-by: Jérémy Scanvic <[email protected]> * disable E402 in deepinv/__init__.py Co-authored-by: Jérémy Scanvic <[email protected]> * pull changes from original PR * fix the import location * put back the weird named import --------- Co-authored-by: Thibaut Modrzyk <[email protected]> * Add demo to connect spyrit and deepinv (#865) * Add demo to connect spyrit and deepinv * implement changes * lint * add spyrit optional dependency * fix toml file * fix attribute to /images/ * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> --------- Co-authored-by: Thibaut Modrzyk <[email protected]> Co-authored-by: Jérémy Scanvic <[email protected]> * add youtube (#926) * Add center_crop parameter to Metric class (#913) * Initial plan * Add center_crop functionality to Metric class and update all child classes Co-authored-by: tachella <[email protected]> * fix metric docs + black + test for mixed signs * fix docstrings --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: tachella <[email protected]> Co-authored-by: Julián Tachella <[email protected]> * fix contributing doc * remove auto labeller --------- Co-authored-by: Matthieu Terris <[email protected]> Co-authored-by: Julián Tachella <[email protected]> Co-authored-by: romainvo <[email protected]> Co-authored-by: Andrew Wang <[email protected]> Co-authored-by: Vicky De Ridder <[email protected]> Co-authored-by: Jérémy Scanvic <[email protected]> Co-authored-by: Johannes Hertrich <[email protected]> Co-authored-by: Brayan Monroy <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Chaithya G R <[email protected]> Co-authored-by: German Shâma Wache <[email protected]> Co-authored-by: Jérémy Scanvic <[email protected]> Co-authored-by: Minh Hai Nguyen <[email protected]> Co-authored-by: mh-nguyen712 <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Thibaut Modrzyk <[email protected]> Co-authored-by: tbaudier <[email protected]> Co-authored-by: Julián Tachella <[email protected]>
* enable blank issues * PR status labels * readme/index/community references contributing * add label system * Labelling (#931) * 3D Total variation priors (#745) * 3D denoising variational * comments * comments jeremy * Remove concurrency settings from gpu_docs.yml (#850) * Remove concurrency settings from gpu_docs.yml final CI fixes * Remove concurrency settings from test_gpu.yml Removed concurrency settings from the GPU test workflow. * Refine Codecov upload condition for Linux only * Update Codecov upload condition in workflow Restrict Codecov upload to Linux with Python 3.10 only. * Fix astra test failing on GPU (#841) * fix normalize threshold in astra tests * black * Fix test transform on GPU (#852) Co-authored-by: Andrew Wang <[email protected]> * Fix broken download Set14 (#845) * . * fix * adapt docs & use utils * changelog * Fix py3.9 not supported by recent coverage version (#856) * . * . * Follow up to #856 (#857) * Robustify CI for Python 3.9 (#858) * Robustify CI for Python 3.9 * Update test_pr.yml * Fix pip install command for Python 3.9 * Update test_pr.yml * Update CI workflow for deepinv installation Refactor installation steps for deepinv in CI workflow to handle Python 3.9 compatibility. * Add compatibility step for Python 3.9 in workflow Add step to modify pyproject.toml for Python 3.9 compatibility. * Refactor deepinv installation steps in workflow Removed conditional installation for Python 3.9 and adjusted the installation step for deepinv dependencies. * Support 3D patches datasets (#806) * Nibabel test hotfix (#863) * Nibabel test hotfix * Require 'blosc2' for the test dataset Added a dependency check for 'blosc2' in the test. * Typing optim (#827) * distance typed * dpir * data fidelity * fix callable * bregman * fix callable distance * epll * start optimizers.py * phase retrieval * start utils * utils fixes * fix circular import from physics * bregman returns * data fid phsycis & return * distance returns * epll returns * opt & phase retrieval * return spectral wrapper * future annotations * _reconstruction_step * utils * add suggestions * finish optimizers.py * Gentler loader check (#871) * Gentler loader check * remove codecov from test * black * add wee comment * Fix error handling for CBSD68 dataset (#868) * fix error handling * fix * suggestions * add PR #723 in CHANGELOG; fix some minor typos (#860) * add PR #723 in CHANGELOG; fix some minor typos * update CHANGELOG * Fix DDRM citation to reference Kawar et al. 2022 (#876) * Initial plan * Fix DDRM reference to cite Kawar et al. 2022 Co-authored-by: tachella <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: tachella <[email protected]> * Allow for complex data inside Wavelets (#738) * Update mri.py * Update mri.py * Added Complex support * Added support for complex data * reformat * Undo MRI change * More fixes * More fixes * yet more fixes, gosh why am i like this * Finally done * Take in comments * fix psnr and add performance checks --------- Co-authored-by: German Shâma Wache <[email protected]> Co-authored-by: Jérémy Scanvic <[email protected]> * add self-sup eval + history fixes (#777) * add self-sup eval + history fixes * changelog * avoid using losses as metrics + add explicit model inference with .eval + trainer tests * add docs warning * remove loss docstring from metric * add missing docstrings * small fix * doc fixes * doc fixes * fixes * fix adversarial * fixes * fix bugs * black * reblack * jeremy's comments * improve docs explanation * unsure + community + generate_datasets fix * early_stop int + fix doc typo * fix docs * add pragma * andrew's comments * docstring compute_metrics * change names + test * still some fixes to go * black * add train_metrics_history * fix wandb bug * black + docs fixes * black + docs fixes * fix * fix class link * re-order * Update deepinv/training/trainer.py Co-authored-by: Andrew Wang <[email protected]> * andrew's comments * andrew's comments * fix julian issue * docstring fixes --------- Co-authored-by: Andrew Wang <[email protected]> * Add 3D variants of CNN networks (#869) * dncnn * gan * . * ruff check * fix ordering test_pad * maxpool unet nd * icnn * dsccp * PDNet * fix icnn * fix dim drunet * pragma * fix dip * test dsccp 3D * icnn 3D test * more tests * test PDNet * tests GAN * . * test all modes drunet + fix bug from main * changelog * clean dummy * remove c variable * fix doc rician noise * 3d models -> denoisers * warnings * fixdim doc * userguide * syntax * Fix LsdirHR download (#866) * . * ruff + fix docs * add proposed train glob * changelog * fix docstring randompatchsampler * fix sphynx? * pragma * pragma * fix denoiser docs * `dinv.io` module and clean up utils (#768) * start develpoing io submodule + modify examples * add scipy load mat * fix doc * add references to DIP demo (idk why this hasn't been added before) * add raster * io -> io_utils, signal -> signals * checks * ISMRMD loading * add all tests * fix optional deps * add new example * fix example * changelog * fix docs * oops * fix doc * update typing * remove load_np and load_nifit in preparation for merge from vicky PR * black * test assert in doctest * add nifti vicky comment + io_utils -> io * pydicom rescale + lidc uses load_dicom * fix test * fix typing in io and add kwargs * fix sphinx typing * fix test * fix doc * vicky comments * vicky comment * fix doc * blosc2 test * PET comment * Minor README fix (#882) * fix (#885) Co-authored-by: mh-nguyen712 <[email protected]> * Minor trainer docstring fixes (#884) * io -> utils.io * improve docstring * very minor fix * Fix formatting in contributing section of README (#888) * Fix #887 (#890) * Move changelog.rst to docs (#889) * init * fix links * fix links * fix double changelog * change pytorch link * remove llm bs * black * fix docs * Update .github/workflows/release.yml Co-authored-by: Minh Hai Nguyen <[email protected]> * fix ghs * fix ghs --------- Co-authored-by: Minh Hai Nguyen <[email protected]> * Enable ruff rule N805 (#892) * enable ruff rule N805 Co-authored-by: Jérémy Scanvic <[email protected]> * ruff check --fix --unsafe-fixes Co-authored-by: Jérémy Scanvic <[email protected]> --------- Co-authored-by: Andrewwango <[email protected]> * Fix announcement text in conf.py (#901) * Update link to changelog in PR template (#903) * Fix docstring formatting in BaseSDE class (#902) * Refactor release workflow (#914) * Refactor release workflow by removing CI checks Remove CI status check step from release workflow * fixes * fixes * black * Release v0.3.6 (#917) Co-authored-by: tachella <[email protected]> * Fix license formatting in CITATION.cff (#920) * Fix license formatting in CITATION.cff * Create workflow to validate CITATION.cff Add GitHub Actions workflow to validate CITATION.cff file * Add check_citation.yml workflow file * Fix license format in CITATION.cff * Add DOI and version to CITATION.cff * Warn inconsistent upsampling parameters in SwinIR (#909) * add a warning to SwinIR * update the changelog * let test be skipped when timm is absent (#922) * Enable ruff rule E402 (#910) * enable rule E402 Co-authored-by: Jérémy Scanvic <[email protected]> * ignore E402 in the examples Co-authored-by: Jérémy Scanvic <[email protected]> * disable E402 in deepinv/__init__.py Co-authored-by: Jérémy Scanvic <[email protected]> * pull changes from original PR * fix the import location * put back the weird named import --------- Co-authored-by: Thibaut Modrzyk <[email protected]> * Add demo to connect spyrit and deepinv (#865) * Add demo to connect spyrit and deepinv * implement changes * lint * add spyrit optional dependency * fix toml file * fix attribute to /images/ * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> * Update examples/external-libraries/demo_connect_spyrit.py Co-authored-by: Jérémy Scanvic <[email protected]> --------- Co-authored-by: Thibaut Modrzyk <[email protected]> Co-authored-by: Jérémy Scanvic <[email protected]> * add youtube (#926) * Add center_crop parameter to Metric class (#913) * Initial plan * Add center_crop functionality to Metric class and update all child classes Co-authored-by: tachella <[email protected]> * fix metric docs + black + test for mixed signs * fix docstrings --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: tachella <[email protected]> Co-authored-by: Julián Tachella <[email protected]> * fix contributing doc * remove auto labeller --------- Co-authored-by: Matthieu Terris <[email protected]> Co-authored-by: Julián Tachella <[email protected]> Co-authored-by: romainvo <[email protected]> Co-authored-by: Andrew Wang <[email protected]> Co-authored-by: Vicky De Ridder <[email protected]> Co-authored-by: Jérémy Scanvic <[email protected]> Co-authored-by: Johannes Hertrich <[email protected]> Co-authored-by: Brayan Monroy <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Chaithya G R <[email protected]> Co-authored-by: German Shâma Wache <[email protected]> Co-authored-by: Jérémy Scanvic <[email protected]> Co-authored-by: Minh Hai Nguyen <[email protected]> Co-authored-by: mh-nguyen712 <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Thibaut Modrzyk <[email protected]> Co-authored-by: tbaudier <[email protected]> Co-authored-by: Julián Tachella <[email protected]> --------- Co-authored-by: Matthieu Terris <[email protected]> Co-authored-by: Julián Tachella <[email protected]> Co-authored-by: romainvo <[email protected]> Co-authored-by: Andrew Wang <[email protected]> Co-authored-by: Vicky De Ridder <[email protected]> Co-authored-by: Jérémy Scanvic <[email protected]> Co-authored-by: Johannes Hertrich <[email protected]> Co-authored-by: Brayan Monroy <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Chaithya G R <[email protected]> Co-authored-by: German Shâma Wache <[email protected]> Co-authored-by: Jérémy Scanvic <[email protected]> Co-authored-by: Minh Hai Nguyen <[email protected]> Co-authored-by: mh-nguyen712 <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Thibaut Modrzyk <[email protected]> Co-authored-by: tbaudier <[email protected]> Co-authored-by: Julián Tachella <[email protected]>


Fix #421
This restores the exact functionality from the previous LsdirHR; except for a single bugfix: previously,
mode='val'would actually capture theX2,X3,X4images too, resulting inlen(dataset)==1000, != 250.Additionally, after extracting a shard, it deletes the associated
tar.gzto save space, because LsdirHR in its entirety is relatively huge; if we do not delete the.tar.gzfiles, some users might not be able to download without running out of space, despite their system having comfortable amount of available space relative to the final dataset size (if the.tar.gzfiles are not present anymore).Note: I tested the checksum only for val, and it remained the same. I did not explicitly test the one for train due to disk space / time constraints.
Note 2: HF uses Xet and it's quite hard to disable or change the caching limits at runtime . It can still consume up to 10GB of cache without us being able to delete (safely). While possible if the user does not import huggingface hub related components, and the first import is performed inside
LsdirHR, I do not think we should complexify things (what if, for example, the user later uses HF hub related components in their own code, and all of a sudden these settings were changed by us?).Especially considering that 10GB is relatively small compared to the entire dataset, I think things are fine as they are on this front.
Note 3: While the
ImageFolderinit is now robust formode="val", formode="train"it is only initialized properly ifself.rootonly contains.pngfiles coming from theLsdirHRdataset. This is silent to the user right now.Two options:
__len__returns the expected values.x_path="shard-[0-1][0-9]/**/*.png"in the case ofmode="train". This fails only in the special case where e.g.shard-17would somehow be present, so it avoids most problems except for a (I would argue) synthetic edge case.Also fixed some syntactic errors in the docstrings of
RandomPatchSampler.Checks to be done before submitting your PR
python3 -m pytest deepinv/testsruns successfully.black .runs successfully.make htmlruns successfully (in thedocs/directory).