Combine back multiple test types into single jobs #11504
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kaxil
approved these changes
Oct 13, 2020
potiuk
force-pushed
the
smart-recombining-test-jobs
branch
from
7ddc6d7 to
16b707c
Compare
potiuk
force-pushed
the
smart-recombining-test-jobs
branch
from
0e58283 to
6c17940
Compare
ashb
approved these changes
Oct 13, 2020
potiuk
force-pushed
the
smart-recombining-test-jobs
branch
from
6c17940 to
1535530
Compare
Member
Author
|
Nope. Kubernetes is a bit more work as the cluster is setup in Action. We'd have to setup all three of them at once. Reverted the last fixup |
potiuk
force-pushed
the
smart-recombining-test-jobs
branch
from
1535530 to
6749a5d
Compare
Member
Author
|
Few more corrections to Integration tests |
potiuk
force-pushed
the
smart-recombining-test-jobs
branch
from
6749a5d to
84c4915
Compare
Seems that by splitting the tests into many small jobs has a bad effect - since we only have queue size = 180 for the whole Apache organisation, we are competing with other projects for the jobs and with the jobs being so short we got starved much more than if we had long jobs. Therefore we are re-combining the test types into single jobs per Python version/Database version and run all the tests sequentially on those machines.
potiuk
force-pushed
the
smart-recombining-test-jobs
branch
from
84c4915 to
e943652
Compare
potiuk
added a commit
to PolideaInternal/airflow
that referenced
this pull request
Oct 14, 2020
The test-type strategy matrix were not deleted entirely when combined back tests in apache#11504
potiuk
added a commit
that referenced
this pull request
Oct 14, 2020
The test-type strategy matrix were not deleted entirely when combined back tests in #11504
potiuk
added a commit
that referenced
this pull request
Nov 14, 2020
Seems that by splitting the tests into many small jobs has a bad effect - since we only have queue size = 180 for the whole Apache organisation, we are competing with other projects for the jobs and with the jobs being so short we got starved much more than if we had long jobs. Therefore we are re-combining the test types into single jobs per Python version/Database version and run all the tests sequentially on those machines. (cherry picked from commit 4297aba)
potiuk
added a commit
that referenced
this pull request
Nov 14, 2020
potiuk
added
the
type:misc/internal
potiuk
added a commit
that referenced
this pull request
Nov 16, 2020
Seems that by splitting the tests into many small jobs has a bad effect - since we only have queue size = 180 for the whole Apache organisation, we are competing with other projects for the jobs and with the jobs being so short we got starved much more than if we had long jobs. Therefore we are re-combining the test types into single jobs per Python version/Database version and run all the tests sequentially on those machines. (cherry picked from commit 4297aba)
potiuk
added a commit
that referenced
this pull request
Nov 16, 2020
potiuk
added a commit
that referenced
this pull request
Nov 16, 2020
Seems that by splitting the tests into many small jobs has a bad effect - since we only have queue size = 180 for the whole Apache organisation, we are competing with other projects for the jobs and with the jobs being so short we got starved much more than if we had long jobs. Therefore we are re-combining the test types into single jobs per Python version/Database version and run all the tests sequentially on those machines. (cherry picked from commit 4297aba)
potiuk
added a commit
that referenced
this pull request
Nov 16, 2020
kaxil
pushed a commit
that referenced
this pull request
Nov 18, 2020
Seems that by splitting the tests into many small jobs has a bad effect - since we only have queue size = 180 for the whole Apache organisation, we are competing with other projects for the jobs and with the jobs being so short we got starved much more than if we had long jobs. Therefore we are re-combining the test types into single jobs per Python version/Database version and run all the tests sequentially on those machines. (cherry picked from commit 4297aba)
kaxil
pushed a commit
that referenced
this pull request
Nov 18, 2020
cfei18
pushed a commit
to cfei18/incubator-airflow
that referenced
this pull request
Mar 5, 2021
Seems that by splitting the tests into many small jobs has a bad effect - since we only have queue size = 180 for the whole Apache organisation, we are competing with other projects for the jobs and with the jobs being so short we got starved much more than if we had long jobs. Therefore we are re-combining the test types into single jobs per Python version/Database version and run all the tests sequentially on those machines. (cherry picked from commit 4297aba)
cfei18
pushed a commit
to cfei18/incubator-airflow
that referenced
this pull request
Mar 5, 2021
The test-type strategy matrix were not deleted entirely when combined back tests in apache#11504 (cherry picked from commit bcf0557)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Seems that by splitting the tests into many small jobs has a bad
effect - since we only have queue size = 180 for the whole Apache
organisation, we are competing with other projects for the jobs
and with the jobs being so short we got starved much more than if
we had long jobs. Therefore we are re-combining the test types into
single jobs per Python version/Database version and run all the
tests sequentially on those machines.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.