Replace chicken-egg providers with automated use of unreleased packages#49799
Replace chicken-egg providers with automated use of unreleased packages#49799potiuk merged 2 commits intoapache:mainfrom
Conversation
38e7cfe to
be1aa05
Compare
|
BTW. @kaxil @eladkal, @gopidesupavan @ashb @amoghrajesh and others potentially interested - some more explaation why previous rules were so complex and brittle, and why after removing ".dev0" prefix from our sources suddenly it became so much easier. -> this is what .dev0 removal allowed for as simplification (it's not the last cleanup - there is one more cleanup in flooring the suffix and provider preparation) What we can do here is we can rely on This is becuase in CI we will build apache airflow==3.1.0 and common-compat==1.6.1 and we will use those packages for building PROD image and generarting Previously we had to do some wicked magic because we produced
This also applied to All this is an unfortunate consequence of https://packaging.python.org/en/latest/specifications/version-specifiers/#summary-of-permitted-suffixes-and-relative-ordering which says that "3.1.0.dev0" < " 3.1.0.rc0" < "3.1.0.rc1" < "3.1.0". If we had other rules and "3.1.0,dev0" was considered as satisfying the ">=3.1.0" - that would be a different story. But it's not, unfortunately. For some reason (probably not realising the consequences) made the decision that strict ordering is better and that So what we had to do previously (this part is still to be removed in a separate PR) we had to artifficially modify those >= requirements - when we prepared `apache-airflow==3.1.0.dev0 we had to make sure that all provider packages it referred to also had >= x.y.z.dev0 (and the other way round). And it got even weirder when we had to build stuff for rc packages for PROD images. If we stick to non-dev0 non-rc suffix anywhere in CI, things are suddenly way simpler:
I think that handles all the edge cases and combinations of released, RC, non-released providers and dependencies between those we had encountered so far. |
|
I still have to figure out why docker-compose test fails, but i think it's because of some of the latest base-url/docker-compose changes that are not properly detected as failing in the current CI because the PROD image had slightly different ly built packages - but I will take a look at that tomorrow. |
|
Ah no... docker-compose one also fails on main |
|
Some more checks:
All seem to be working as expected. |
jscheffl
left a comment
There was a problem hiding this comment.
This was a longer read. I fully get the point and can confirm. Good cleanup in my view! Thanks for this! (I assume the prod image generation is a minor risk... still it is. And I also feel that the handling of pip is like it is, would be great being able to change it....)
The docker compose error for me seems to be un-related, wondering a bit this is not happening on main. Because the error that I can re-produce locally with the failing api-server is:
ERROR: Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/starlette/routing.py", line 692, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/app.py", line 62, in lifespan
await stack.enter_async_context(
File "/usr/local/lib/python3.9/contextlib.py", line 575, in enter_async_context
result = await _cm_type.__aenter__(cm)
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/home/airflow/.local/lib/python3.9/site-packages/svcs/fastapi.py", line 66, in __call__
async with self.registry, cm(app, self.registry) as state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/execution_api/app.py", line 94, in lifespan
registry.register_value(JWTValidator, _jwt_validator(), ping=JWTValidator.status)
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/execution_api/app.py", line 65, in _jwt_validator
**get_sig_validation_args(make_secret_key_if_needed=False),
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/auth/tokens.py", line 581, in get_sig_validation_args
return {"secret_key": get_signing_key("api_auth", "jwt_secret", make_secret_key_if_needed)}
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/auth/tokens.py", line 538, in get_signing_key
raise ValueError(f"The value {section}/{key} must be set!")
ValueError: The value api_auth/jwt_secret must be set!
ERROR: Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/starlette/routing.py", line 692, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/app.py", line 62, in lifespan
await stack.enter_async_context(
File "/usr/local/lib/python3.9/contextlib.py", line 575, in enter_async_context
result = await _cm_type.__aenter__(cm)
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
Please make sure to build the frontend in static/ directory and restart the server
Please make sure to build the frontend in static/ directory and restart the server
File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
async with original_context(app) as maybe_original_state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/home/airflow/.local/lib/python3.9/site-packages/svcs/fastapi.py", line 66, in __call__
async with self.registry, cm(app, self.registry) as state:
File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
return await self.gen.__anext__()
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/execution_api/app.py", line 94, in lifespan
registry.register_value(JWTValidator, _jwt_validator(), ping=JWTValidator.status)
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/execution_api/app.py", line 65, in _jwt_validator
**get_sig_validation_args(make_secret_key_if_needed=False),
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/auth/tokens.py", line 581, in get_sig_validation_args
return {"secret_key": get_signing_key("api_auth", "jwt_secret", make_secret_key_if_needed)}
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/auth/tokens.py", line 538, in get_signing_key
raise ValueError(f"The value {section}/{key} must be set!")
ValueError: The value api_auth/jwt_secret must be set!
ERROR: Application startup failed. Exiting.
I am working on a fix (and way improved docker-compose-test diagnostics). There were FEW things wrong in the current docker-compose. |
712ce27 to
0a4b96a
Compare
When we got rid of the .dev0 suffix, it is now possible to entirely rely on building the packages locally using exsting mechanisms, that check if packages have been already released - for CI builds, and can rely on the fact that we need at least pre-release version of packages if we are building pre-release version of airflow. It works as follows: * for CI builds (generate constraints and PROD image builds) - we are alwasys attempt to build ALL provider packages, but without --skip-tag-check - which means that if provider has been already released and it's version did not change in main, we are not going to build it locally and we will use it from PyPI. However if provider version is updated and the provider has not yet been released (checked by tag) - it will be build locally from sources and it will be used for constraint generation. * for release PROD images build, on the other hand we NEVER build packages locally - we always rely on PyPI released packages, however if we are building pre-release version of airflow, we automatically add --pre flag that looks for pre-release packages in PyPI - this way pre-release version of airflow can be built with pre-release version of providers. We are still attempting to use constraints for that, however first - so unless there are no limits in apache airflow that prevent it from using released versions of providers, the constraint versions will be used - only if it fails, PROD images will fall back to non-constraint installation that will allow to use freely pre-release versions of packages from PyPI. This means for example that if we cherry-pick a change from main that increases minimum version of provider for apache-airflow to one that does not even have a pre-release version, building of rc version image for airflow will fail (which is a good thing). Lack of --pre flag for "release" version of Airlfow also means that if airlfow has a min version of provider that has no "released" version yet (only rc) - it will also fail (which is also a good thing)
0a4b96a to
12e83de
Compare
|
This is really cool :) nice to see the automated usage. |
Indeed :) |
…es (apache#49799) * Replace chicken-egg providers with automated use of unreleased packages When we got rid of the .dev0 suffix, it is now possible to entirely rely on building the packages locally using exsting mechanisms, that check if packages have been already released - for CI builds, and can rely on the fact that we need at least pre-release version of packages if we are building pre-release version of airflow. It works as follows: * for CI builds (generate constraints and PROD image builds) - we are alwasys attempt to build ALL provider packages, but without --skip-tag-check - which means that if provider has been already released and it's version did not change in main, we are not going to build it locally and we will use it from PyPI. However if provider version is updated and the provider has not yet been released (checked by tag) - it will be build locally from sources and it will be used for constraint generation. * for release PROD images build, on the other hand we NEVER build packages locally - we always rely on PyPI released packages, however if we are building pre-release version of airflow, we automatically add --pre flag that looks for pre-release packages in PyPI - this way pre-release version of airflow can be built with pre-release version of providers. We are still attempting to use constraints for that, however first - so unless there are no limits in apache airflow that prevent it from using released versions of providers, the constraint versions will be used - only if it fails, PROD images will fall back to non-constraint installation that will allow to use freely pre-release versions of packages from PyPI. This means for example that if we cherry-pick a change from main that increases minimum version of provider for apache-airflow to one that does not even have a pre-release version, building of rc version image for airflow will fail (which is a good thing). Lack of --pre flag for "release" version of Airlfow also means that if airlfow has a min version of provider that has no "released" version yet (only rc) - it will also fail (which is also a good thing) * Update scripts/in_container/run_generate_constraints.py
…eased packages (apache#49799) * Replace chicken-egg providers with automated use of unreleased packages When we got rid of the .dev0 suffix, it is now possible to entirely rely on building the packages locally using exsting mechanisms, that check if packages have been already released - for CI builds, and can rely on the fact that we need at least pre-release version of packages if we are building pre-release version of airflow. It works as follows: * for CI builds (generate constraints and PROD image builds) - we are alwasys attempt to build ALL provider packages, but without --skip-tag-check - which means that if provider has been already released and it's version did not change in main, we are not going to build it locally and we will use it from PyPI. However if provider version is updated and the provider has not yet been released (checked by tag) - it will be build locally from sources and it will be used for constraint generation. * for release PROD images build, on the other hand we NEVER build packages locally - we always rely on PyPI released packages, however if we are building pre-release version of airflow, we automatically add --pre flag that looks for pre-release packages in PyPI - this way pre-release version of airflow can be built with pre-release version of providers. We are still attempting to use constraints for that, however first - so unless there are no limits in apache airflow that prevent it from using released versions of providers, the constraint versions will be used - only if it fails, PROD images will fall back to non-constraint installation that will allow to use freely pre-release versions of packages from PyPI. This means for example that if we cherry-pick a change from main that increases minimum version of provider for apache-airflow to one that does not even have a pre-release version, building of rc version image for airflow will fail (which is a good thing). Lack of --pre flag for "release" version of Airlfow also means that if airlfow has a min version of provider that has no "released" version yet (only rc) - it will also fail (which is also a good thing) * Update scripts/in_container/run_generate_constraints.py (cherry picked from commit c6a2b8e) Co-authored-by: Jarek Potiuk <[email protected]>
…eased packages (apache#49799) * Replace chicken-egg providers with automated use of unreleased packages When we got rid of the .dev0 suffix, it is now possible to entirely rely on building the packages locally using exsting mechanisms, that check if packages have been already released - for CI builds, and can rely on the fact that we need at least pre-release version of packages if we are building pre-release version of airflow. It works as follows: * for CI builds (generate constraints and PROD image builds) - we are alwasys attempt to build ALL provider packages, but without --skip-tag-check - which means that if provider has been already released and it's version did not change in main, we are not going to build it locally and we will use it from PyPI. However if provider version is updated and the provider has not yet been released (checked by tag) - it will be build locally from sources and it will be used for constraint generation. * for release PROD images build, on the other hand we NEVER build packages locally - we always rely on PyPI released packages, however if we are building pre-release version of airflow, we automatically add --pre flag that looks for pre-release packages in PyPI - this way pre-release version of airflow can be built with pre-release version of providers. We are still attempting to use constraints for that, however first - so unless there are no limits in apache airflow that prevent it from using released versions of providers, the constraint versions will be used - only if it fails, PROD images will fall back to non-constraint installation that will allow to use freely pre-release versions of packages from PyPI. This means for example that if we cherry-pick a change from main that increases minimum version of provider for apache-airflow to one that does not even have a pre-release version, building of rc version image for airflow will fail (which is a good thing). Lack of --pre flag for "release" version of Airlfow also means that if airlfow has a min version of provider that has no "released" version yet (only rc) - it will also fail (which is also a good thing) * Update scripts/in_container/run_generate_constraints.py (cherry picked from commit c6a2b8e) Co-authored-by: Jarek Potiuk <[email protected]>
…eased packages (apache#49799) * Replace chicken-egg providers with automated use of unreleased packages When we got rid of the .dev0 suffix, it is now possible to entirely rely on building the packages locally using exsting mechanisms, that check if packages have been already released - for CI builds, and can rely on the fact that we need at least pre-release version of packages if we are building pre-release version of airflow. It works as follows: * for CI builds (generate constraints and PROD image builds) - we are alwasys attempt to build ALL provider packages, but without --skip-tag-check - which means that if provider has been already released and it's version did not change in main, we are not going to build it locally and we will use it from PyPI. However if provider version is updated and the provider has not yet been released (checked by tag) - it will be build locally from sources and it will be used for constraint generation. * for release PROD images build, on the other hand we NEVER build packages locally - we always rely on PyPI released packages, however if we are building pre-release version of airflow, we automatically add --pre flag that looks for pre-release packages in PyPI - this way pre-release version of airflow can be built with pre-release version of providers. We are still attempting to use constraints for that, however first - so unless there are no limits in apache airflow that prevent it from using released versions of providers, the constraint versions will be used - only if it fails, PROD images will fall back to non-constraint installation that will allow to use freely pre-release versions of packages from PyPI. This means for example that if we cherry-pick a change from main that increases minimum version of provider for apache-airflow to one that does not even have a pre-release version, building of rc version image for airflow will fail (which is a good thing). Lack of --pre flag for "release" version of Airlfow also means that if airlfow has a min version of provider that has no "released" version yet (only rc) - it will also fail (which is also a good thing) * Update scripts/in_container/run_generate_constraints.py (cherry picked from commit c6a2b8e) Co-authored-by: Jarek Potiuk <[email protected]>
…eased packages (apache#49799) * Replace chicken-egg providers with automated use of unreleased packages When we got rid of the .dev0 suffix, it is now possible to entirely rely on building the packages locally using exsting mechanisms, that check if packages have been already released - for CI builds, and can rely on the fact that we need at least pre-release version of packages if we are building pre-release version of airflow. It works as follows: * for CI builds (generate constraints and PROD image builds) - we are alwasys attempt to build ALL provider packages, but without --skip-tag-check - which means that if provider has been already released and it's version did not change in main, we are not going to build it locally and we will use it from PyPI. However if provider version is updated and the provider has not yet been released (checked by tag) - it will be build locally from sources and it will be used for constraint generation. * for release PROD images build, on the other hand we NEVER build packages locally - we always rely on PyPI released packages, however if we are building pre-release version of airflow, we automatically add --pre flag that looks for pre-release packages in PyPI - this way pre-release version of airflow can be built with pre-release version of providers. We are still attempting to use constraints for that, however first - so unless there are no limits in apache airflow that prevent it from using released versions of providers, the constraint versions will be used - only if it fails, PROD images will fall back to non-constraint installation that will allow to use freely pre-release versions of packages from PyPI. This means for example that if we cherry-pick a change from main that increases minimum version of provider for apache-airflow to one that does not even have a pre-release version, building of rc version image for airflow will fail (which is a good thing). Lack of --pre flag for "release" version of Airlfow also means that if airlfow has a min version of provider that has no "released" version yet (only rc) - it will also fail (which is also a good thing) * Update scripts/in_container/run_generate_constraints.py (cherry picked from commit c6a2b8e) Co-authored-by: Jarek Potiuk <[email protected]>
…eased packages (apache#49799) * Replace chicken-egg providers with automated use of unreleased packages When we got rid of the .dev0 suffix, it is now possible to entirely rely on building the packages locally using exsting mechanisms, that check if packages have been already released - for CI builds, and can rely on the fact that we need at least pre-release version of packages if we are building pre-release version of airflow. It works as follows: * for CI builds (generate constraints and PROD image builds) - we are alwasys attempt to build ALL provider packages, but without --skip-tag-check - which means that if provider has been already released and it's version did not change in main, we are not going to build it locally and we will use it from PyPI. However if provider version is updated and the provider has not yet been released (checked by tag) - it will be build locally from sources and it will be used for constraint generation. * for release PROD images build, on the other hand we NEVER build packages locally - we always rely on PyPI released packages, however if we are building pre-release version of airflow, we automatically add --pre flag that looks for pre-release packages in PyPI - this way pre-release version of airflow can be built with pre-release version of providers. We are still attempting to use constraints for that, however first - so unless there are no limits in apache airflow that prevent it from using released versions of providers, the constraint versions will be used - only if it fails, PROD images will fall back to non-constraint installation that will allow to use freely pre-release versions of packages from PyPI. This means for example that if we cherry-pick a change from main that increases minimum version of provider for apache-airflow to one that does not even have a pre-release version, building of rc version image for airflow will fail (which is a good thing). Lack of --pre flag for "release" version of Airlfow also means that if airlfow has a min version of provider that has no "released" version yet (only rc) - it will also fail (which is also a good thing) * Update scripts/in_container/run_generate_constraints.py (cherry picked from commit c6a2b8e) Co-authored-by: Jarek Potiuk <[email protected]>
…eased packages (#49799) (#50109) * Replace chicken-egg providers with automated use of unreleased packages When we got rid of the .dev0 suffix, it is now possible to entirely rely on building the packages locally using exsting mechanisms, that check if packages have been already released - for CI builds, and can rely on the fact that we need at least pre-release version of packages if we are building pre-release version of airflow. It works as follows: * for CI builds (generate constraints and PROD image builds) - we are alwasys attempt to build ALL provider packages, but without --skip-tag-check - which means that if provider has been already released and it's version did not change in main, we are not going to build it locally and we will use it from PyPI. However if provider version is updated and the provider has not yet been released (checked by tag) - it will be build locally from sources and it will be used for constraint generation. * for release PROD images build, on the other hand we NEVER build packages locally - we always rely on PyPI released packages, however if we are building pre-release version of airflow, we automatically add --pre flag that looks for pre-release packages in PyPI - this way pre-release version of airflow can be built with pre-release version of providers. We are still attempting to use constraints for that, however first - so unless there are no limits in apache airflow that prevent it from using released versions of providers, the constraint versions will be used - only if it fails, PROD images will fall back to non-constraint installation that will allow to use freely pre-release versions of packages from PyPI. This means for example that if we cherry-pick a change from main that increases minimum version of provider for apache-airflow to one that does not even have a pre-release version, building of rc version image for airflow will fail (which is a good thing). Lack of --pre flag for "release" version of Airlfow also means that if airlfow has a min version of provider that has no "released" version yet (only rc) - it will also fail (which is also a good thing) * Update scripts/in_container/run_generate_constraints.py (cherry picked from commit c6a2b8e)
When we removed "chicken-egg-providers" from the build in apache#49799 we should be able now to build PROD image in v3*test branch without special case of only using PyPI providers. This should work because in v3-*test branch we automatically detect (by presence of relesed tag) whether we should build provider from sources or use it from PyPI. In normal circumstances, we would need use providers fro PyPI, but sometimes (For example for 3.0.1) we need to use some providers from sources before they get released (for example common.messaging 1.0.1 that is the lowest possible common.messaging for Airflow 3.0.1. This PR removes that special case and switches to the same method of building PROD image in main and in release branch.
When we removed "chicken-egg-providers" from the build in apache#49799 we should be able now to build PROD image in v3*test branch without special case of only using PyPI providers. This should work because in v3-*test branch we automatically detect (by presence of relesed tag) whether we should build provider from sources or use it from PyPI. In normal circumstances, we would need use providers fro PyPI, but sometimes (For example for 3.0.1) we need to use some providers from sources before they get released (for example common.messaging 1.0.1 that is the lowest possible common.messaging for Airflow 3.0.1. This PR removes that special case and switches to the same method of building PROD image in main and in release branch.
When we removed "chicken-egg-providers" from the build in apache#49799 we should be able now to build PROD image in v3*test branch without special case of only using PyPI providers. This should work because in v3-*test branch we automatically detect (by presence of relesed tag) whether we should build provider from sources or use it from PyPI. In normal circumstances, we would need use providers fro PyPI, but sometimes (For example for 3.0.1) we need to use some providers from sources before they get released (for example common.messaging 1.0.1 that is the lowest possible common.messaging for Airflow 3.0.1. This PR removes that special case and switches to the same method of building PROD image in main and in release branch.
When we removed "chicken-egg-providers" from the build in #49799 we should be able now to build PROD image in v3*test branch without special case of only using PyPI providers. This should work because in v3-*test branch we automatically detect (by presence of relesed tag) whether we should build provider from sources or use it from PyPI. In normal circumstances, we would need use providers fro PyPI, but sometimes (For example for 3.0.1) we need to use some providers from sources before they get released (for example common.messaging 1.0.1 that is the lowest possible common.messaging for Airflow 3.0.1. This PR removes that special case and switches to the same method of building PROD image in main and in release branch.
…ranch (apache#50115) When we removed "chicken-egg-providers" from the build in apache#49799 we should be able now to build PROD image in v3*test branch without special case of only using PyPI providers. This should work because in v3-*test branch we automatically detect (by presence of relesed tag) whether we should build provider from sources or use it from PyPI. In normal circumstances, we would need use providers fro PyPI, but sometimes (For example for 3.0.1) we need to use some providers from sources before they get released (for example common.messaging 1.0.1 that is the lowest possible common.messaging for Airflow 3.0.1. This PR removes that special case and switches to the same method of building PROD image in main and in release branch. (cherry picked from commit 9deb2a7) Co-authored-by: Jarek Potiuk <[email protected]>
…ranch (#50115) (#50118) When we removed "chicken-egg-providers" from the build in #49799 we should be able now to build PROD image in v3*test branch without special case of only using PyPI providers. This should work because in v3-*test branch we automatically detect (by presence of relesed tag) whether we should build provider from sources or use it from PyPI. In normal circumstances, we would need use providers fro PyPI, but sometimes (For example for 3.0.1) we need to use some providers from sources before they get released (for example common.messaging 1.0.1 that is the lowest possible common.messaging for Airflow 3.0.1. This PR removes that special case and switches to the same method of building PROD image in main and in release branch. (cherry picked from commit 9deb2a7)





When we got rid of the .dev0 suffix, it is now possible to entirely
rely on building the packages locally using exsting mechanisms, that
check if packages have been already released - for CI builds, and can
rely on the fact that we need at least pre-release version of packages
if we are building pre-release version of airflow.
It works as follows:
for CI builds (generate constraints and PROD image builds) - we are
alwasys attempt to build ALL provider packages, but without
--skip-tag-check - which means that if provider has been already
released and it's version did not change in main, we are not going
to build it locally and we will use it from PyPI. However if provider
version is updated and the provider has not yet been released (checked
by tag) - it will be build locally from sources and it will be used
for constraint generation.
for release PROD images build, on the other hand we NEVER build
packages locally - we always rely on PyPI released packages, however
if we are building pre-release version of airflow, we automatically
add --pre flag that looks for pre-release packages in PyPI - this way
pre-release version of airflow can be built with pre-release version
of providers. We are still attempting to use constraints for that,
however first - so unless there are no limits in apache airflow
that prevent it from using released versions of providers, the
constraint versions will be used - only if it fails, PROD images
will fall back to non-constraint installation that will allow to
use freely pre-release versions of packages from PyPI. This means
for example that if we cherry-pick a change from main that increases
minimum version of provider for apache-airflow to one that does not
even have a pre-release version, building of rc version image for
airflow will fail (which is a good thing). Lack of --pre flag
for "release" version of Airlfow also means that if airlfow has
a min version of provider that has no "released" version yet (only
rc) - it will also fail (which is also a good thing)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.