[test] Run extended tests once daily in Travis #10052
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jnewbery
changed the title
Member
|
Do you know if emails will be sent out to the maintainers on failing
builds? I think we'd want to avoid that, as the extended tests on
travis could end up being rather flaky.
utACK 88e3aa0 otherwise.
…On Wed, Mar 22, 2017 at 4:11 PM, John Newbery ***@***.***> wrote:
Travis has beta support for a new feature called cron jobs. We should hook
into this to run our extended functional test suite once per day.
This requires two changes: merging the change to .travis.yml in this PR, and
then enabling daily builds of the master branch through Travis settings.
For an example of a cron build see
https://travis-ci.org/jnewbery/bitcoin/builds/213827649. This currently
fails because:
rpcbind_tests.py requires ipv6, which Travis currently doesn't support. I
have a workaround for that on the way.
pruning.py times out. I've now updated the .travis.yml file to exclude
running pruning.
The granularity for cron jobs is very coarse - currently Travis only allows
jobs to be run daily, weekly or monthly. There have been lots of requests
for more flexible cron schedules in this thread so Travis may add this as an
enhancement at some point.
________________________________
You can view, comment on, or merge this pull request online at:
#10052
Commit Summary
Run extended tests once daily
File Changes
M .travis.yml (3)
Patch Links:
https://github.com/bitcoin/bitcoin/pull/10052.patch
https://github.com/bitcoin/bitcoin/pull/10052.diff
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Contributor
Author
|
I think email notifications for cron jobs are the same as for regular branch builds. In any case, I'm not sure whether that's a bad thing. If the test is really broken, then it's better to know immediately that something's broken it, rather than whenever someone happens to run the extended tests. And if you're worried about the extended tests being unreliable, then we should make them more robust. |
Member
|
It is fine when the failure can be reproduced locally, though there have
been cases where travis just fails out of the blue and as soon as you try
to debug, it will behave as if nothing was wrong.
As you can see in the log of the cron job in your fork, bip9-softforks
sometimes fails: https://travis-ci.org/jnewbery/bitcoin/jobs/213827655
I tried to reproduce this failure locally but it always passes for me...
…On Wed, Mar 22, 2017 at 8:13 PM, John Newbery ***@***.***> wrote:
I think email notifications for cron jobs are the same as for regular
branch builds.
In any case, I'm not sure whether that's a bad thing. If the test is
really broken, then it's better to know immediately that something's broken
it, rather than whenever someone happens to run the extended tests. And if
you're worried about the extended tests being unreliable, then we should
make them more robust.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#10052 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGGmv7HhG2DsFdvilONDhkuGpOhlEREqks5roXLIgaJpZM4MlWf4>
.
|
Contributor
Author
|
Yes - that could get annoying. I plan to improve travis logging by having it upload detailed logs from failed test cases to an S3 bucket. In the long run we should definitely try to make sure travis runs of extended test cases are reliable, but for now you could manually exclude those test cases which you think are unreliable? |
Contributor
|
Concept ACK |
maflcko
pushed a commit
to maflcko/bitcoin-core
that referenced
this pull request
Mar 24, 2017
88e3aa0 Run extended tests once daily (John Newbery) Tree-SHA512: 92478ee930b60d8d90d1bf7352ff81853198391559281ce3074748b55145202984ba4150f751d25f3256e696e3fd8f25ea02ecd57b881744736adf505c275178
Contributor
Author
|
Great! Now I need to fix bip9-softforks.py :) |
Contributor
Author
|
I believe #10072 should help with bip9-softforks. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Travis has beta support for a new feature called cron jobs. We should hook into this to run our extended functional test suite once per day.
This requires two changes: merging the change to .travis.yml in this PR, and then enabling daily builds of the master branch through Travis settings.
For an example of a cron build see https://travis-ci.org/jnewbery/bitcoin/builds/213827649. This currently fails because:
The granularity for cron jobs is very coarse - currently Travis only allows jobs to be run daily, weekly or monthly. There have been lots of requests for more flexible cron schedules in this thread so Travis may add this as an enhancement at some point.