Allow running many runs per test without flooding the logs#9158
Merged
jtattermusch merged 2 commits intogrpc:masterfrom Dec 20, 2016
Merged
Allow running many runs per test without flooding the logs#9158jtattermusch merged 2 commits intogrpc:masterfrom
jtattermusch merged 2 commits intogrpc:masterfrom
Conversation
jtattermusch
force-pushed
the
deflaking_adjustments
branch
from
c47dc2e to
68e27bf
Compare
adelez
reviewed
Dec 20, 2016
Contributor
adelez
left a comment
adelez
left a comment
There was a problem hiding this comment.
Should we do this for interop tests as well?
| help='Generates a JUnit-compatible XML report') | ||
| argp.add_argument('--report_suite_name', default='tests', type=str, | ||
| help='Test suite name to use in generated JUnit XML report') | ||
| argp.add_argument('--quiet_success', |
Contributor
There was a problem hiding this comment.
Looks like this option only makes sense when n > 1, maybe we don't need to expose it as an arg?
| return [ | ||
| self.config.job_spec(['src/objective-c/tests/run_tests.sh'], | ||
| timeout_seconds=None, | ||
| timeout_seconds=60*60, |
Contributor
There was a problem hiding this comment.
Define a constant for timeout values?
| with open('tools/run_tests/sanity/sanity_tests.yaml', 'r') as f: | ||
| return [self.config.job_spec(cmd['script'].split(), | ||
| timeout_seconds=None, environ={'TEST': 'true'}, | ||
| timeout_seconds=30*60, environ={'TEST': 'true'}, |
Contributor
Author
|
@adelez, I merged the PR before I received the comment from you. Currently I am not planning to introduce this for interop tests because it will take more work to make use of changes this PR (and that's what I am planning to work on in the nearest future). We might try to introduce a similar change for interop tests later (but on the other hand, running multiple instances of interop tests is in fact what stress tests are doing). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #9161 (fix needed to prevent tests from hanging if they timeout).
Allow invoking run_tests_matrix.py with specified number of runs per tests. To prevent the test reports from becoming too large (and to consume too much memory in a run_tests.py instance), also allow omitting passing tests from the XML report and from console output.
-- one more thing that needs to be addressed is increasing the timeout in run_tests_matrix.py accordingly (currently 4 hours should be enough for most wrapped languages, but it might not be enough for C/C++).