Only rebuilds base python image when upgrading to newer deps #14783
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
Hey @kaxil, all -> with recent problems with Python version, I think I gave up on the idea of updating to latest version automatically in CI. It's nice in theory but has a number of problems (like the recent test failiures with CVE). With this change we will be much more stable - always the same base python version will be used with few exceptions - the scheduled nightly build will use the "latest available" version - so after we stabilize it, it might flag any problems like the one we had with the CVE (same mechanism as with "upgrade-to-newer-depenedencies". And we will be able to refresh the latest version in alll our registries easily manually (command to do it is added to docs). |
potiuk
force-pushed
the
make-sure-ci-tests-use-cached-python-version
branch
3 times, most recently
from
4ab94c1 to
1a98f3e
Compare
The base python image is only updated when manually triggered and in case of checking for upgraded dependencies in master build. While automated upgrade to latest Python image is good for security, it can cause a number of problems when run automatically in the CI: * cache invalidation - thus longer builds * sudden test failures This happened in the past already quite a number of times so it is time to switch to a bit different mode. Python images will only be automatically upgraded in those cases: 1) When Master CI build is run in scheduled nightly build - to check that tests still pass for latest version of the image 2) When manually refreshed with --force-pull-base-python-image 3) When DockerHub official images (from tags) are built. The procedure to refresh the images manually in our CI has been added to the documentation.
potiuk
force-pushed
the
make-sure-ci-tests-use-cached-python-version
branch
from
1a98f3e to
27e152a
Compare
mik-laj
approved these changes
Mar 14, 2021
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
github-actions
bot
added
the
full tests needed
potiuk
added a commit
that referenced
this pull request
Mar 23, 2021
The base python image is only updated when manually triggered and in case of checking for upgraded dependencies in master build. While automated upgrade to latest Python image is good for security, it can cause a number of problems when run automatically in the CI: * cache invalidation - thus longer builds * sudden test failures This happened in the past already quite a number of times so it is time to switch to a bit different mode. Python images will only be automatically upgraded in those cases: 1) When Master CI build is run in scheduled nightly build - to check that tests still pass for latest version of the image 2) When manually refreshed with --force-pull-base-python-image 3) When DockerHub official images (from tags) are built. The procedure to refresh the images manually in our CI has been added to the documentation. (cherry picked from commit 4762396)
ashb
pushed a commit
that referenced
this pull request
Apr 15, 2021
The base python image is only updated when manually triggered and in case of checking for upgraded dependencies in master build. While automated upgrade to latest Python image is good for security, it can cause a number of problems when run automatically in the CI: * cache invalidation - thus longer builds * sudden test failures This happened in the past already quite a number of times so it is time to switch to a bit different mode. Python images will only be automatically upgraded in those cases: 1) When Master CI build is run in scheduled nightly build - to check that tests still pass for latest version of the image 2) When manually refreshed with --force-pull-base-python-image 3) When DockerHub official images (from tags) are built. The procedure to refresh the images manually in our CI has been added to the documentation. (cherry picked from commit 4762396)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The base python image is only updated when manually triggered and
in case of checking for upgraded dependencies in master build.
While automated upgrade to latest Python image is good for
security, it can cause a number of problems when run automatically
in the CI:
This happened in the past already quite a number of times so it
is time to switch to a bit different mode. Python images will only
be automatically upgraded in those cases:
When Master CI build is run in scheduled nightly build - to check
that tests still pass for latest version of the image
When manually refreshed with --force-pull-base-python-image
When DockerHub official images (from tags) are built.
The procedure to refresh the images manually in our CI has been
added to the documentation.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.