Skip to content

initial network consistency test#2171

Merged
rijobro merged 2 commits intoProject-MONAI:devfrom
rijobro:network_consistency
May 14, 2021
Merged

initial network consistency test#2171
rijobro merged 2 commits intoProject-MONAI:devfrom
rijobro:network_consistency

Conversation

@rijobro
Copy link
Copy Markdown
Contributor

@rijobro rijobro commented May 10, 2021 •
edited
Loading

Suggested format of testing network consistency. Uses a git repository to store the data so as not to clog up the main repository. Currently this submodule sits in my user area, but this would obviously be migrated to Project-MONAI.

The idea would be that if the extra repository is present, it'll loop over each of the *.pt files in the folder, load the network, and checking consistency of forward pass.

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@rijobro
Copy link
Copy Markdown
Contributor Author

rijobro commented May 10, 2021

@Nic-Ma @ericspod @charliebudd @wyli Just a rough suggestion of how we could track network consistency, open for discussion. It stores extra data in a git submodule. Most of this PR is currently taken up by creating a network factory, which I would split into its own PR if we were to go ahead with this.

@wyli
Copy link
Copy Markdown
Contributor

wyli commented May 11, 2021

thanks, the tests look great. I think git-submodule can be very difficult to work with, don't want to introduce it to this project, unless it's 200% necessary...

@rijobro
Copy link
Copy Markdown
Contributor Author

rijobro commented May 11, 2021

Not using a git submodule (I agree they can be tricky) means that we either abandon this idea, store the data in this repository, or store it somewhere else. What do you think?

The data for the 3d unet was ~7.5MB, so for all networks we might be looking at something like ~100MB.

@rijobro
Copy link
Copy Markdown
Contributor Author

rijobro commented May 11, 2021

@ericspod suggested (offline) simply git cloning the test repository, rather than having it as a submodule. Seems like an easy alternative @wyli

@wyli
Copy link
Copy Markdown
Contributor

wyli commented May 12, 2021

thanks, perhaps we can start a new repo under project-monai to host the test data with git-lfs?

@rijobro
Copy link
Copy Markdown
Contributor Author

rijobro commented May 12, 2021

That sounds like a good plan. I hadn't heard of git lfs before!

@rijobro rijobro force-pushed the network_consistency branch from 4940381 to e0c0bfc Compare May 12, 2021 14:29
@rijobro
Copy link
Copy Markdown
Contributor Author

rijobro commented May 13, 2021

@wyli unrelated error, the added test passes: https://github.com/Project-MONAI/MONAI/pull/2171/checks?check_run_id=2574279151#step:6:2046.

wyli
wyli approved these changes May 13, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@wyli wyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I think this is a good initial example, we could merge it if it's decoupled from the network factory idea.

@rijobro
test network consistency
fe2dde9 
Signed-off-by: Richard Brown <[email protected]>
@rijobro rijobro force-pushed the network_consistency branch from ab920a3 to fe2dde9 Compare May 14, 2021 08:56
@rijobro rijobro marked this pull request as ready for review May 14, 2021 08:56
@rijobro
depth 1
cfc88a4 
Signed-off-by: Richard Brown <[email protected]>
@rijobro rijobro enabled auto-merge (squash) May 14, 2021 09:25
@rijobro rijobro merged commit e68af74 into Project-MONAI:dev May 14, 2021
@rijobro rijobro deleted the network_consistency branch May 14, 2021 10:38
@rijobro rijobro mentioned this pull request May 17, 2021
Closed
wyli pushed a commit that referenced this pull request May 26, 2021
@rijobro @wyli
initial network consistency test (#2171)
74f0d53 
test network consistency
wyli pushed a commit that referenced this pull request May 26, 2021
@rijobro @wyli
initial network consistency test (#2171)
07a38f1 
test network consistency
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wyli wyli wyli approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rijobro @wyli