Reduce time of integration tests #39682
Conversation
tiborvass
force-pushed
the
jenkinsfile_reduce_time
branch
8 times, most recently
from
208776c to
5a147c7
Compare
tiborvass
force-pushed
the
jenkinsfile_reduce_time
branch
3 times, most recently
from
c8fe607 to
55d938f
Compare
tiborvass
force-pushed
the
jenkinsfile_reduce_time
branch
from
55d938f to
5e4ae80
Compare
tiborvass
force-pushed
the
jenkinsfile_reduce_time
branch
from
5e4ae80 to
8fafa8b
Compare
andrewhsu
left a comment
andrewhsu
left a comment
There was a problem hiding this comment.
LGTM w.r.t. reducing janky PR check times from 2 hrs down to 1 hr
tiborvass
force-pushed
the
jenkinsfile_reduce_time
branch
5 times, most recently
from
5488d3e to
346bb1f
Compare
andrewhsu
force-pushed
the
jenkinsfile_reduce_time
branch
from
4019239 to
da72bbc
Compare
|
With the last 4 commits I added, the total time for the PR check with This was accomplished by moving the integration-cli tests to only run on merge to master for experimental, z, and powerpc builds. The integration-cli tests are still run on PR checks in the janky job. The integration-cli tests will likely add 1 hr of time to the jobs that run against master branch, so it will be around 2 hrs total: https://ci.docker.com/public/blue/organizations/jenkins/moby/activity?branch=master |
Signed-off-by: Tibor Vass <[email protected]>
Signed-off-by: Tibor Vass <[email protected]>
…3 parallel runs Signed-off-by: Tibor Vass <[email protected]>
The powerpc-master stage will just run the integration-cli tests. The existing powerpc stage will run the unit tests and the integration tests. In this way, PR check jobs will be shorter, but all integration tests will run after PR is merged to master. Signed-off-by: Andrew Hsu <[email protected]>
The z-master stage will just run the integration-cli tests. The existing z stage will run the unit tests and the integration tests. In this way, PR check jobs will be shorter, but all integration tests will run after PR is merged to master. Signed-off-by: Andrew Hsu <[email protected]>
Signed-off-by: Andrew Hsu <[email protected]>
Signed-off-by: Andrew Hsu <[email protected]>
Signed-off-by: Andrew Hsu <[email protected]>
In case a job fails before even generating a report file. Signed-off-by: Andrew Hsu <[email protected]>
Signed-off-by: Andrew Hsu <[email protected]>
Signed-off-by: Andrew Hsu <[email protected]>
Signed-off-by: Andrew Hsu <[email protected]>
andrewhsu
force-pushed
the
jenkinsfile_reduce_time
branch
from
46cffd5 to
eb30f0a
Compare
|
I've rebased to resolve conflicts. With these final changes in, would like to merge when green so we can reap the benefits of faster PR check results. |
|
From the PR check on eb30f0a there is just a known flaky test failure: The legacy PR check on janky has the same test passing. |
|
@thaJeztah @vdemeester @cpuguy83 this PR is ready for your consideration |
| ''' | ||
| } | ||
| } | ||
| // needs to be last stage that calls make.sh for the junit report to work |
There was a problem hiding this comment.
What was the problem if it wasn't the last stage? Does it help if you upload the junit.xml as part of the stage itself? (See #39719)
There was a problem hiding this comment.
The unit test stage needs to be last because the tests will be written to bundles directory and if the other stages run after the unit test stage then the directory would be wiped out.
Want to honor the return value of the unit tests, so if the tests has a failure then it will return non-zero back to the shell and jenkins will stop processing that step and mark that step as red. The always stage is an exception catch stage that will be run whether any of the steps fail, so that would be a suitable place to consume the junit.xml file.
There was a problem hiding this comment.
Maybe. Can do in followup
There was a problem hiding this comment.
Each stage can upload its bundles/results before the next stage starts
|
LGTM |
cpuguy83
left a comment
cpuguy83
left a comment
There was a problem hiding this comment.
LGTM
It would be nice if we can consolidate the actual logic and only setup the envs for each arch.
6 participants
Reduce time of integration tests by dividing tests into 3 parallel runs
I have to open a new PR because force push on #39680 did not work.
Related to #39651