add ngc download bundle by yiheng-wang-nv · Pull Request #5710 · Project-MONAI/MONAI

yiheng-wang-nv · 2022-12-12T15:10:30Z

Signed-off-by: Yiheng Wang [email protected]

Description

This PR adds the support of download bundles from ngc: https://catalog.ngc.nvidia.com/models?filters=&orderBy=scoreDESC&query=monai
In addition, when "version" is not provided, it changes to download the latest version in default.

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: Yiheng Wang <[email protected]>

monai/bundle/scripts.py

Nic-Ma · 2022-12-15T23:37:28Z

Hi @yiheng-wang-nv ,

NGC bundles are public available now:
https://catalog.ngc.nvidia.com/models?filters=&orderBy=scoreDESC&query=monai

Thanks.

monai/bundle/scripts.py

Signed-off-by: Yiheng Wang <[email protected]>

monai/bundle/scripts.py

tests/test_bundle_download.py

Signed-off-by: Yiheng Wang <[email protected]>

mingxin-zheng · 2022-12-16T10:18:08Z

The latest version cannot be resolved in NGC source with spleen_ct_segmentation as the name instead of monai_spleen_ct_segmentation

BUNDLE_DOWNLOAD_SRC=ngc python -m monai.bundle download --name spleen_ct_segmentation
2022-12-16 10:13:47,797 - INFO - --- input summary of monai.bundle.scripts.download ---
2022-12-16 10:13:47,797 - INFO - > name: 'spleen_ct_segmentation'
2022-12-16 10:13:47,797 - INFO - > source: 'ngc'
2022-12-16 10:13:47,797 - INFO - > progress: True
2022-12-16 10:13:47,797 - INFO - ---


spleen_ct_segmentation
monai_spleen_ct_segmentation
monai_spleen_ct_segmentation_vNone.zip: 0.00B [00:00, ?B/s]
2022-12-16 10:13:51,118 - ERROR - Download failed from https://api.ngc.nvidia.com/v2/models/nvidia/monaitoolkit/monai_spleen_ct_segmentation/versions/None/zip to /tmp/tmpqfrv7010/monai_spleen_ct_segmentation_vNone.zip.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/workspace/monai/monai-in-dev/monai/bundle/__main__.py", line 18, in <module>
    fire.Fire()
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/conda/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/workspace/monai/monai-in-dev/monai/bundle/scripts.py", line 291, in download
    _download_from_ngc(
  File "/workspace/monai/monai-in-dev/monai/bundle/scripts.py", line 151, in _download_from_ngc
    download_url(url=url, filepath=filepath, hash_val=None, progress=progress)
  File "/workspace/monai/monai-in-dev/monai/apps/utils.py", line 203, in download_url
    _download_with_progress(url, tmp_name, progress=progress)
  File "/workspace/monai/monai-in-dev/monai/apps/utils.py", line 114, in _download_with_progress
    raise e
  File "/workspace/monai/monai-in-dev/monai/apps/utils.py", line 107, in _download_with_progress
    urlretrieve(url, filepath, reporthook=t.update_to)
  File "/opt/conda/lib/python3.8/urllib/request.py", line 247, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/opt/conda/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/conda/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/opt/conda/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/opt/conda/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/opt/conda/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/opt/conda/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404:

monai/bundle/scripts.py

Signed-off-by: Yiheng Wang <[email protected]>

yiheng-wang-nv · 2022-12-17T05:22:26Z

An error in make html: https://github.com/Project-MONAI/MONAI/actions/runs/3718516325/jobs/6306771505#step:7:584
A bit weird since I did not meet the same error in local test.

monai/bundle/scripts.py

Signed-off-by: Yiheng Wang <[email protected]>

…iheng-wang-nv/MONAI into 5679-add-ngc-source-in-bundle-download

Signed-off-by: Yiheng Wang <[email protected]>

wyli

looks good to me, @Nic-Ma could you please help confirm..

mingxin-zheng · 2022-12-17T12:29:23Z

The behavior looks good to me now

> BUNDLE_DOWNLOAD_SRC=ngc python -m monai.bundle download --name spleen_ct_segmentation --bundle_dir ./abc
2022-12-17 12:27:41,263 - INFO - --- input summary of monai.bundle.scripts.download ---
2022-12-17 12:27:41,263 - INFO - > name: 'spleen_ct_segmentation'
2022-12-17 12:27:41,263 - INFO - > bundle_dir: './abc'
2022-12-17 12:27:41,263 - INFO - > source: 'ngc'
2022-12-17 12:27:41,263 - INFO - > remove_prefix: 'monai_'
2022-12-17 12:27:41,263 - INFO - > progress: True
2022-12-17 12:27:41,263 - INFO - ---


monai_spleen_ct_segmentation_v0.3.7.zip: 34.0MB [00:19, 1.83MB/s]                                                                                                                                          
2022-12-17 12:28:03,310 - INFO - Downloaded: abc/monai_spleen_ct_segmentation_v0.3.7.zip
2022-12-17 12:28:03,310 - INFO - Expected md5 is None, skip md5 check for file abc/monai_spleen_ct_segmentation_v0.3.7.zip.
2022-12-17 12:28:03,310 - INFO - Writing into directory: abc/spleen_ct_segmentation.

> ls abc/
monai_spleen_ct_segmentation_v0.3.7.zip  spleen_ct_segmentation

wyli · 2022-12-17T13:17:18Z

/build

monai/bundle/scripts.py

add ngc download bundle

3e707e6

Signed-off-by: Yiheng Wang <[email protected]>

Nic-Ma reviewed Dec 12, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

Nic-Ma requested a review from wyli December 12, 2022 15:19

mingxin-zheng reviewed Dec 14, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

mingxin-zheng reviewed Dec 16, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

yiheng-wang-nv and others added 2 commits December 16, 2022 11:34

Merge branch 'dev' into 5679-add-ngc-source-in-bundle-download

7f92f96

add download ngc tests

497c258

Signed-off-by: Yiheng Wang <[email protected]>

yiheng-wang-nv marked this pull request as ready for review December 16, 2022 08:00

fix flake8

38987ba

Signed-off-by: Yiheng Wang <[email protected]>

yiheng-wang-nv mentioned this pull request Dec 16, 2022

Download latest version of bundle automatically #5320

Closed

yiheng-wang-nv requested review from Nic-Ma, SachidanandAlle and mingxin-zheng December 16, 2022 08:22

remove extra comment

934b14b

Signed-off-by: Yiheng Wang <[email protected]>

mingxin-zheng reviewed Dec 16, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

wyli reviewed Dec 16, 2022

View reviewed changes

tests/test_bundle_download.py Outdated Show resolved Hide resolved

add check name

a4be93a

Signed-off-by: Yiheng Wang <[email protected]>

Nic-Ma reviewed Dec 16, 2022

View reviewed changes

monai/bundle/scripts.py Show resolved Hide resolved

yiheng-wang-nv and others added 4 commits December 16, 2022 23:34

revert and remove name check

7431f26

Signed-off-by: Yiheng Wang <[email protected]>

move to ngc mmar loading

3490031

Signed-off-by: Yiheng Wang <[email protected]>

update arg

6094d44

Signed-off-by: Yiheng Wang <[email protected]>

Merge branch 'dev' into 5679-add-ngc-source-in-bundle-download

475f871

yiheng-wang-nv requested review from Nic-Ma, mingxin-zheng and wyli December 17, 2022 04:46

wyli reviewed Dec 17, 2022

View reviewed changes

monai/bundle/scripts.py Outdated Show resolved Hide resolved

fix error and uncomment workflow

7356fc8

Signed-off-by: Yiheng Wang <[email protected]>

yiheng-wang-nv added 3 commits December 17, 2022 18:16

Merge branch '5679-add-ngc-source-in-bundle-download' of github.com:y…

87a0658

…iheng-wang-nv/MONAI into 5679-add-ngc-source-in-bundle-download

rename workflow

d6427b5

Signed-off-by: Yiheng Wang <[email protected]>

uncomment schedule

6609dd7

Signed-off-by: Yiheng Wang <[email protected]>

wyli approved these changes Dec 17, 2022

View reviewed changes

mingxin-zheng approved these changes Dec 17, 2022

View reviewed changes

wyli merged commit 67d84d3 into Project-MONAI:dev Dec 17, 2022

Nic-Ma reviewed Dec 17, 2022

View reviewed changes

monai/bundle/scripts.py Show resolved Hide resolved

yiheng-wang-nv deleted the 5679-add-ngc-source-in-bundle-download branch December 19, 2022 02:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add ngc download bundle#5710

add ngc download bundle#5710
wyli merged 14 commits intoProject-MONAI:devfrom
yiheng-wang-nv:5679-add-ngc-source-in-bundle-download

yiheng-wang-nv commented Dec 12, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Nic-Ma commented Dec 15, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mingxin-zheng commented Dec 16, 2022 •

edited

Loading

Uh oh!

Uh oh!

yiheng-wang-nv commented Dec 17, 2022

Uh oh!

Uh oh!

wyli left a comment

Uh oh!

mingxin-zheng commented Dec 17, 2022 •

edited

Loading

Uh oh!

wyli commented Dec 17, 2022

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

yiheng-wang-nv commented Dec 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Uh oh!

Uh oh!

Uh oh!

Nic-Ma commented Dec 15, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mingxin-zheng commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

yiheng-wang-nv commented Dec 17, 2022

Uh oh!

Uh oh!

wyli left a comment

Choose a reason for hiding this comment

Uh oh!

mingxin-zheng commented Dec 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wyli commented Dec 17, 2022

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yiheng-wang-nv commented Dec 12, 2022 •

edited

Loading

mingxin-zheng commented Dec 16, 2022 •

edited

Loading

mingxin-zheng commented Dec 17, 2022 •

edited

Loading