Skip to content

10 Update workflow for automatic upload bundle#11

Merged
yiheng-wang-nv merged 50 commits intodevfrom
10-enhance-workflow-for-update-models
Jun 16, 2022
Merged

10 Update workflow for automatic upload bundle#11
yiheng-wang-nv merged 50 commits intodevfrom
10-enhance-workflow-for-update-models

Conversation

@yiheng-wang-nv
Copy link
Copy Markdown
Collaborator

@yiheng-wang-nv yiheng-wang-nv commented May 20, 2022

Fixes #10 .

Description

This PR is used to add the automatic uploading actions after a PR is merged. The workflow is like:
flow chart

Status

Ready

Please ensure all the checkboxes:

  • Codeformat tests passed locally by running ./runtests.sh --codeformat.

@yiheng-wang-nv yiheng-wang-nv changed the title 10 update workflow 10 Update workflow for automatic upload bundle May 20, 2022
@yiheng-wang-nv
Copy link
Copy Markdown
Collaborator Author

So far, the auto steps include:

  1. after a PR is merged, zip all bundles and check which one has been changed.
  2. for changed bundles, upload the new zip file to https://github.com/Project-MONAI/model-zoo/releases/tag/hosting_storage_v1
  3. update new version, checksum and download into models/model_info.json, and push into a new branch, the branch name is in the form of [merged PR number]-auto-update-model-info.

Then, an admin/contributor is able to create a new pull request of this created new branch (this create pr is still manual, I tried to add gh cli within update_model_info.py, but met the error like here, Hi @wyli , do you have any suggests about how to automatically make the PR? )

Another place that to be added is the uploading results check, since the upload thing may fail due to network or other issues, I will address this part later.

@wyli
Copy link
Copy Markdown
Collaborator

wyli commented May 20, 2022

looks like a permission issue you can try

name: Push to master

on:
  push:
    branches:
      - master

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    # the checkout action persists the passed credentials by default
    # subsequent git commands will pick them up automatically
    - uses: actions/checkout@v2
      with:
        token: ${{secrets.PAT}}
    - run: |
        # do something
        git push

https://stackoverflow.com/a/64078507

@yiheng-wang-nv
Copy link
Copy Markdown
Collaborator Author

Thanks @wyli , the push part is fine, and the issue comes from the auto pull request part. I used gh pr create ..., but it seems there are some prompts after the command.

@yiheng-wang-nv
Copy link
Copy Markdown
Collaborator Author

Hi @Nic-Ma
FYI: with the commit (link), admin can do:

  1. download large files.
  2. compress (exclude the large files config file) the bundle to a zip file
  3. upload the zip file into releases
    by running the following command:
python .github/update_model_info.py -b bundle_name

The next step is to put it into a workflow.

@Nic-Ma
Copy link
Copy Markdown
Collaborator

Nic-Ma commented May 31, 2022

Hi @yiheng-wang-nv ,

Great improvement, please hold on for the workflow, maybe let's put it into Blossom directly.
CC @YanxuanLiu .

Thanks.

@Nic-Ma
Copy link
Copy Markdown
Collaborator

Nic-Ma commented Jun 1, 2022

I see you commented much code in this PR, those parts are not ready to use?

Thanks.

@yiheng-wang-nv
Copy link
Copy Markdown
Collaborator Author

Hi @wyli @Nic-Ma and @ericspod , I just marked the status of this PR into "Ready", and could you please help to review it?
I attached a flow chart at the beginning, and it might be helpful to understand the processes.

During a previous commit, I used the spleen ct segmentation bundle for test: log, uploaded place, produced new branch

@wyli
Copy link
Copy Markdown
Collaborator

wyli commented Jun 14, 2022

thanks, this looks great, does this support deletion as well? just wanted to confirm that if I submit a PR to permanently remove a bundle folder, the PR won't break the pipeline. And I guess we should provide some instructions to the users/admins about how to remove a bundle folder.

@Nic-Ma
Copy link
Copy Markdown
Collaborator

Nic-Ma commented Jun 14, 2022

Hi @wyli ,

That's a good corner case, @yiheng-wang-nv please make sure a PR can delete some bundles in the source code(not in the release storage).

Thanks.

Copy link
Copy Markdown
Collaborator

@Nic-Ma Nic-Ma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @wyli ,

Could you please help review the PR? I think you are expert on the Github workflows.

Thanks in advance.

@ericspod
Copy link
Copy Markdown
Member

The version strings are part of the compressed file's names so we should mention in the specification that versions should follow semantic versioning and contain only characters valid in filenames.

@yiheng-wang-nv
Copy link
Copy Markdown
Collaborator Author

yiheng-wang-nv commented Jun 15, 2022

thanks, this looks great, does this support deletion as well? just wanted to confirm that if I submit a PR to permanently remove a bundle folder, the PR won't break the pipeline. And I guess we should provide some instructions to the users/admins about how to remove a bundle folder.

Thanks! @wyli and @Nic-Ma . If delete a whole bundle, according to the logic (utils.get_changed_bundle_list, it is used to grab all changed and exsiting bundles), the github release part will not be changed actually. I just pushed a new commit to print some information to address this case.

Copy link
Copy Markdown
Collaborator

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me, a minor point -- requirements.txt could be updated

@yiheng-wang-nv yiheng-wang-nv merged commit e166d32 into dev Jun 16, 2022
@yiheng-wang-nv yiheng-wang-nv deleted the 10-enhance-workflow-for-update-models branch June 16, 2022 03:33
@Nic-Ma
Copy link
Copy Markdown
Collaborator

Nic-Ma commented Jun 16, 2022

The version strings are part of the compressed file's names so we should mention in the specification that versions should follow semantic versioning and contain only characters valid in filenames.

Hi @ericspod ,

I created a small PR for it: Project-MONAI/MONAI#4512, could you please help review it?
And I think this PR is ready to merge now.

Thanks in advance.

yiheng-wang-nv added a commit to yiheng-wang-nv/model-zoo that referenced this pull request Jul 29, 2024
* update workflow

* remove test print and run cmd

* update git config in update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* run git config at first

* fix flake8

* add prompt enter

* remove the pr part

* set back to merge workflow

* use zip method

* enable manual trigger

* remove lfs in workflow to check quota

* uncomment upload

* fix black

* check lfs in workflow

* add check pr changed files yml

* update download and upload release

Signed-off-by: Yiheng Wang <[email protected]>

* fix format issue and comment workflows

* handle errors

* fix flake8 error

* change large file name

* test update spleen

* fix format error

* change error message and use monai 0.9.0rc2

* change requirements

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix issue

* fix flake8 error

* raise error if update wrong

* update workflow with push new commit

* update requirements and test with spleen

* remove extra lfs

* add check status for push commit

* add push error message

* fix error naming

* remove duplicate try raise

* remove update model token

* revert test part for formal usage

* update delete case and clean rmtree code

* fix codeformat issue

* update readme about how to remove a bundle

* Update requirements.txt

Co-authored-by: Wenqi Li <[email protected]>

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wenqi Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enhance workflow for automatically update models

4 participants