Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

avoid skipping validation stage when data read iterator exists #18289

Closed
heaseny wants to merge 5 commits into from
Closed

avoid skipping validation stage when data read iterator exists #18289

heaseny wants to merge 5 commits into from

Conversation

@heaseny
Copy link
Contributor

@heaseny heaseny commented May 12, 2020

Description

avoid skipping validation stage when data read iterator exists.
Test commands:
cd example/image-classification
python train_cifar10.py --network='resnet' --num-layers=50 --image-shape='3,28,28' --num-epochs=1

Abnormal test log in current master branch:
INFO:root:Epoch[0] Train-accuracy=0.386069
INFO:root:Epoch[0] Train-top_k_accuracy_5=0.875060
INFO:root:Epoch[0] Time cost=38.553

Normal test log which is expected:
INFO:root:Epoch[0] Batch [360-380] Speed: 1302.17 samples/sec accuracy=0.497266 top_k_accuracy_5=0.935156
INFO:root:Epoch[0] Train-accuracy=0.386069
INFO:root:Epoch[0] Train-top_k_accuracy_5=0.875060
INFO:root:Epoch[0] Time cost=39.096
INFO:root:Epoch[0] Validation-accuracy=0.448378
INFO:root:Epoch[0] Validation-top_k_accuracy_5=0.918018

Checklist

Essentials

  • Changes are completed
  • All changes have test coverage.

Changes

  • Feature: avoid skipping validation stage when validation data read iterator exists

heaseny and others added 5 commits February 19, 2020 18:29
@heaseny heaseny requested a review from szha as a code owner May 12, 2020 08:27
@mxnet-bot
Copy link

mxnet-bot commented May 12, 2020

Hey @heaseny , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [sanity, miscellaneous, unix-cpu, website, edge, centos-gpu, clang, centos-cpu, windows-cpu, windows-gpu, unix-gpu]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@heaseny
Copy link
Contributor Author

heaseny commented May 13, 2020

@mxnet-bot run ci [unix-cpu]

@mxnet-bot
Copy link

mxnet-bot commented May 13, 2020

Jenkins CI successfully triggered : [unix-cpu]

@heaseny
Copy link
Contributor Author

heaseny commented May 13, 2020 •
edited
Loading

@mxnet-bot run ci [centos-cpu,unix-cpu,unix-gpu,windows-gpu]
failed due to test_numpy_op.py:test_np_mixed_precision_binary_funcs
Related issue : #16848
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-cpu/detail/PR-18289/1/pipeline/
http://jenkins.mxnet-ci.amazon-ml.com/job/mxnet-validation/job/unix-cpu/job/PR-18289/2/display/redirect
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18289/1/pipeline/
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-18289/1/pipeline/

@mxnet-bot
Copy link

mxnet-bot commented May 14, 2020

Jenkins CI successfully triggered : [unix-gpu, windows-gpu, centos-cpu, unix-cpu]

@heaseny
Copy link
Contributor Author

heaseny commented May 15, 2020

@szha Can you help to review this pull request, and merge it if there is no problem? Thanks a lot.

@lanking520 lanking520 added the pr-awaiting-review PR is waiting for code review label Jul 26, 2020
@szha szha changed the base branch from master to v1.x August 14, 2020 05:27
@szha szha changed the base branch from v1.x to master August 14, 2020 05:27
@szha
Copy link
Member

szha commented Aug 14, 2020

apologies for the spam.

@heaseny we deprecated module in the master branch and we will continue to support it in v1.x. it would be great if you could rebase your change to the v1.x branch.

@heaseny
Copy link
Contributor Author

heaseny commented Aug 18, 2020

@szha as work changing and I am not focusing on the mxnet test and have no environment, if the change is necessary in v1.x, can you help to rebase the change, or should I close this PR?

@szha szha mentioned this pull request Sep 7, 2020
Merged
6 tasks
@szha
Copy link
Member

szha commented Sep 7, 2020

picked up the change for v1.x in the above PR.

@szha szha closed this Sep 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@szha szha Awaiting requested review from szha szha is a code owner

Assignees

No one assigned

Labels

pr-awaiting-review PR is waiting for code review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@heaseny @mxnet-bot @szha @lanking520