Skip to content

Replace chicken-egg providers with automated use of unreleased packages#49799

Merged
potiuk merged 2 commits intoapache:mainfrom
potiuk:remove-manual-chicken-egg-providers
Apr 26, 2025
Merged

Replace chicken-egg providers with automated use of unreleased packages#49799
potiuk merged 2 commits intoapache:mainfrom
potiuk:remove-manual-chicken-egg-providers

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Apr 25, 2025

When we got rid of the .dev0 suffix, it is now possible to entirely
rely on building the packages locally using exsting mechanisms, that
check if packages have been already released - for CI builds, and can
rely on the fact that we need at least pre-release version of packages
if we are building pre-release version of airflow.

It works as follows:

  • for CI builds (generate constraints and PROD image builds) - we are
    alwasys attempt to build ALL provider packages, but without
    --skip-tag-check - which means that if provider has been already
    released and it's version did not change in main, we are not going
    to build it locally and we will use it from PyPI. However if provider
    version is updated and the provider has not yet been released (checked
    by tag) - it will be build locally from sources and it will be used
    for constraint generation.

  • for release PROD images build, on the other hand we NEVER build
    packages locally - we always rely on PyPI released packages, however
    if we are building pre-release version of airflow, we automatically
    add --pre flag that looks for pre-release packages in PyPI - this way
    pre-release version of airflow can be built with pre-release version
    of providers. We are still attempting to use constraints for that,
    however first - so unless there are no limits in apache airflow
    that prevent it from using released versions of providers, the
    constraint versions will be used - only if it fails, PROD images
    will fall back to non-constraint installation that will allow to
    use freely pre-release versions of packages from PyPI. This means
    for example that if we cherry-pick a change from main that increases
    minimum version of provider for apache-airflow to one that does not
    even have a pre-release version, building of rc version image for
    airflow will fail (which is a good thing). Lack of --pre flag
    for "release" version of Airlfow also means that if airlfow has
    a min version of provider that has no "released" version yet (only
    rc) - it will also fail (which is also a good thing)


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@potiuk
Copy link
Member Author

potiuk commented Apr 25, 2025

BTW. @kaxil @eladkal, @gopidesupavan @ashb @amoghrajesh and others potentially interested - some more explaation why previous rules were so complex and brittle, and why after removing ".dev0" prefix from our sources suddenly it became so much easier.

-> this is what .dev0 removal allowed for as simplification (it's not the last cleanup - there is one more cleanup in flooring the suffix and provider preparation)

What we can do here is we can rely on >=x,y.z dependendencies between Airflow and providers, because both airlfow and providers do not have the suffixes any more when we prepare then on CI. So if somoene says apache-airflow==3.1.0 (in development) depends on (say) comon-compat>=1.6.1 and common.compat==1.6.1 depending on apache-airflow>=3.1.0 the CI will continue to work nicely,

This is becuase in CI we will build apache airflow==3.1.0 and common-compat==1.6.1 and we will use those packages for building PROD image and generarting PyPI constraints. And both packages installed in CI will satisfy each-others requirements.

Previously we had to do some wicked magic because we produced apache-airflow==3.1.0.dev0 and common-compat==1.6.1dev0 and we had to do the magic because:

  • apache-airflow >= 3.1.0 was NOT satisfied by apache-airflow==3.1.0.dev0
  • comon-compat >= 1.6.1 was NOT satisifed by comon-compat==1.6.1.dev0

This also applied to workspace dependencies in sources. When we had "apache-ariflow==3.1.0.dev0" - we could never work on a provider package that had "apache-airflow>=3.1.0" - because "3.1.0.dev0" does not satisfy ">=3.1.0". If not that limitation, we could simply strip-down '.dev0" always in CI and keep it in the sources, however with workspace the same dependencies that we have in requirements are used to determine in workspace can be installed together, we would not be able to work at the same time on provider / airflow that depend on each other's current main version if we kept the .dev0 suffix.

All this is an unfortunate consequence of https://packaging.python.org/en/latest/specifications/version-specifiers/#summary-of-permitted-suffixes-and-relative-ordering which says that "3.1.0.dev0" < " 3.1.0.rc0" < "3.1.0.rc1" < "3.1.0". If we had other rules and "3.1.0,dev0" was considered as satisfying the ">=3.1.0" - that would be a different story. But it's not, unfortunately. For some reason (probably not realising the consequences) made the decision that strict ordering is better and that ,dev0 always goes before final version - which is logical, but also does not allow to have such pre-release cross-dependencies. There are a number of discussion about this choice (in this context) in pip: for example pypa/pip#12049 where maintainers confirmed that pip behaves according to the specification (hard to argue with it because it does) but still being open to control that behaviour - but that would be a difficult one to define nicely. Also uv follows the spec in workspaces and it would be hard to change the spec and adjust thing in all the tooling if we would like to change it.

So what we had to do previously (this part is still to be removed in a separate PR) we had to artifficially modify those >= requirements - when we prepared `apache-airflow==3.1.0.dev0 we had to make sure that all provider packages it referred to also had >= x.y.z.dev0 (and the other way round). And it got even weirder when we had to build stuff for rc packages for PROD images.

If we stick to non-dev0 non-rc suffix anywhere in CI, things are suddenly way simpler:

  • workspaces work out of the box. We can have in main "apache-airflow==3.1.0" depending on "common.compat>=1.6.1" that depends on "apache-airlfow>=3.1.0" and uv will nicely install such workspace (this is the main reason why removal of .dev0 suffix hard-coded in our sources was needed - cc: @kaxil )

  • if a provider package is not released yet (either not RC-d or RC in PyPI) we will always build such provider package in main build with "final" version - wich means we will not have to modify airflow's dependency to have >= x.y.z.rc0 or in any other way - we can just build both packages in CI without any suffixes and they will install together nicely (we do it for constraint generation and PROD image building).

  • simlarly, even if Airflow is not released yet but we have (say) 3.1.0 in main - we will always build the "final" version in our CI and even if we release a new provider that depends on >=3.1.0 - such provider will nicely build and install in Ci - without having to modify the requirements at all.

  • when we release Dockerhub RC PROD airflow image - we will first have to at least release an RC image. And here we will still have modify the requirements in Airlfow and providers to have >=X.Y.Zrc1 - and there is really no other way, but this is already way simpler, knowing that we have no other suffixes to strip. And in this case we will not use locally built providers at all - when we release Dockerhub RC PROD image we ALWAYS will have to install things from PyPI. We will use "--pre" flag so we will install pre-release providers that are already released in PyPi - but we will never build the providers locally in CI for those image (cc: @kaxil - that will avoid the issue we had with the broken standard provided in RC5 of 3.0.0) - this happened because we installed provider from locally modified sources in main that were not released even in RC yet.

  • when we release FINAL PROD airflow image - we will not use the --pre flag, and airflow will not haverc added to >= - which means that FINAL PROD airflow image will always require "final" providers released in PyPI.

I think that handles all the edge cases and combinations of released, RC, non-released providers and dependencies between those we had encountered so far.

@potiuk
Copy link
Member Author

potiuk commented Apr 25, 2025

I still have to figure out why docker-compose test fails, but i think it's because of some of the latest base-url/docker-compose changes that are not properly detected as failing in the current CI because the PROD image had slightly different ly built packages - but I will take a look at that tomorrow.

@potiuk
Copy link
Member Author

potiuk commented Apr 25, 2025

Ah no... docker-compose one also fails on main

@potiuk
Copy link
Member Author

potiuk commented Apr 26, 2025

Some more checks:

  1. when generating PyPI constraints we automatically build the non-released-yet providers (either being voted or not yet even RC-d)

image

  1. they are nicely reflected in PYPI constraints:

image

  1. When building PROD image for testing in CI. all "prod" prod packages are built from sources - and the non-released yetproviders have the "future" versions:

image

  1. This all works despite common.compat 1.6.1 not released yet

image

  1. And with several packages having common.compat >= 1.6.1 in their dependencies (even if they are not even released yet as RCs and their version is not yet bumped by the release manager - because we build those packages from main sources in CI)

image

All seem to be working as expected.

@potiuk potiuk requested a review from amoghrajesh April 26, 2025 06:07
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a longer read. I fully get the point and can confirm. Good cleanup in my view! Thanks for this! (I assume the prod image generation is a minor risk... still it is. And I also feel that the handling of pip is like it is, would be great being able to change it....)

The docker compose error for me seems to be un-related, wondering a bit this is not happening on main. Because the error that I can re-produce locally with the failing api-server is:

ERROR:    Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/starlette/routing.py", line 692, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/opt/airflow/airflow-core/src/airflow/api_fastapi/app.py", line 62, in lifespan
    await stack.enter_async_context(
  File "/usr/local/lib/python3.9/contextlib.py", line 575, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/home/airflow/.local/lib/python3.9/site-packages/svcs/fastapi.py", line 66, in __call__
    async with self.registry, cm(app, self.registry) as state:
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/opt/airflow/airflow-core/src/airflow/api_fastapi/execution_api/app.py", line 94, in lifespan
    registry.register_value(JWTValidator, _jwt_validator(), ping=JWTValidator.status)
  File "/opt/airflow/airflow-core/src/airflow/api_fastapi/execution_api/app.py", line 65, in _jwt_validator
    **get_sig_validation_args(make_secret_key_if_needed=False),
  File "/opt/airflow/airflow-core/src/airflow/api_fastapi/auth/tokens.py", line 581, in get_sig_validation_args
    return {"secret_key": get_signing_key("api_auth", "jwt_secret", make_secret_key_if_needed)}
  File "/opt/airflow/airflow-core/src/airflow/api_fastapi/auth/tokens.py", line 538, in get_signing_key
    raise ValueError(f"The value {section}/{key} must be set!")
ValueError: The value api_auth/jwt_secret must be set!

ERROR:    Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/starlette/routing.py", line 692, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/opt/airflow/airflow-core/src/airflow/api_fastapi/app.py", line 62, in lifespan
    await stack.enter_async_context(
  File "/usr/local/lib/python3.9/contextlib.py", line 575, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
Please make sure to build the frontend in static/ directory and restart the server
Please make sure to build the frontend in static/ directory and restart the server
  File "/home/airflow/.local/lib/python3.9/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/home/airflow/.local/lib/python3.9/site-packages/svcs/fastapi.py", line 66, in __call__
    async with self.registry, cm(app, self.registry) as state:
  File "/usr/local/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/opt/airflow/airflow-core/src/airflow/api_fastapi/execution_api/app.py", line 94, in lifespan
    registry.register_value(JWTValidator, _jwt_validator(), ping=JWTValidator.status)
  File "/opt/airflow/airflow-core/src/airflow/api_fastapi/execution_api/app.py", line 65, in _jwt_validator
    **get_sig_validation_args(make_secret_key_if_needed=False),
  File "/opt/airflow/airflow-core/src/airflow/api_fastapi/auth/tokens.py", line 581, in get_sig_validation_args
    return {"secret_key": get_signing_key("api_auth", "jwt_secret", make_secret_key_if_needed)}
  File "/opt/airflow/airflow-core/src/airflow/api_fastapi/auth/tokens.py", line 538, in get_signing_key
    raise ValueError(f"The value {section}/{key} must be set!")
ValueError: The value api_auth/jwt_secret must be set!

ERROR:    Application startup failed. Exiting.

@potiuk
Copy link
Member Author

potiuk commented Apr 26, 2025

The docker compose error for me seems to be un-related, wondering a bit this is not happening on main. Because the error that I can re-produce locally with the failing api-server is:

I am working on a fix (and way improved docker-compose-test diagnostics). There were FEW things wrong in the current docker-compose.

@potiuk potiuk force-pushed the remove-manual-chicken-egg-providers branch from 712ce27 to 0a4b96a Compare April 26, 2025 11:53
@potiuk potiuk enabled auto-merge (squash) April 26, 2025 11:53
potiuk added 2 commits April 26, 2025 14:51
When we got rid of the .dev0 suffix, it is now possible to entirely
rely on building the packages locally using exsting mechanisms, that
check if packages have been already released - for CI builds, and can
rely on the fact that we need at least pre-release version of packages
if we are building pre-release version of airflow.

It works as follows:

* for CI builds (generate constraints and PROD image builds) - we are
  alwasys attempt to build ALL provider packages, but without
  --skip-tag-check - which means that if provider has been already
  released and it's version did not change in main, we are not going
  to build it locally and we will use it from PyPI. However if provider
  version is updated and the provider has not yet been released (checked
  by tag) - it will be build locally from sources and it will be used
  for constraint generation.

* for release PROD images build, on the other hand we NEVER build
  packages locally - we always rely on PyPI released packages, however
  if we are building pre-release version of airflow, we automatically
  add --pre flag that looks for pre-release packages in PyPI - this way
  pre-release version of airflow can be built with pre-release version
  of providers. We are still attempting to use constraints for that,
  however first - so unless there are no limits in apache airflow
  that prevent it from using released versions of providers, the
  constraint versions will be used - only if it fails, PROD images
  will fall back to non-constraint installation that will allow to
  use freely pre-release versions of packages from PyPI. This means
  for example that if we cherry-pick a change from main that increases
  minimum version of provider for apache-airflow to one that does not
  even have a pre-release version, building of rc version image for
  airflow will fail (which is a good thing). Lack of --pre flag
  for "release" version of Airlfow also means that if airlfow has
  a min version of provider that has no "released" version yet (only
  rc) - it will also fail (which is also a good thing)
@potiuk potiuk force-pushed the remove-manual-chicken-egg-providers branch from 0a4b96a to 12e83de Compare April 26, 2025 12:51
@potiuk potiuk merged commit c6a2b8e into apache:main Apr 26, 2025
95 of 96 checks passed
@potiuk potiuk deleted the remove-manual-chicken-egg-providers branch April 26, 2025 13:23
@gopidesupavan
Copy link
Member

This is really cool :) nice to see the automated usage.

@potiuk
Copy link
Member Author

potiuk commented Apr 26, 2025

This is really cool :) nice to see the automated usage.

Indeed :)

jroachgolf84 pushed a commit to jroachgolf84/airflow that referenced this pull request Apr 30, 2025
…es (apache#49799)

* Replace chicken-egg providers with automated use of unreleased packages

When we got rid of the .dev0 suffix, it is now possible to entirely
rely on building the packages locally using exsting mechanisms, that
check if packages have been already released - for CI builds, and can
rely on the fact that we need at least pre-release version of packages
if we are building pre-release version of airflow.

It works as follows:

* for CI builds (generate constraints and PROD image builds) - we are
  alwasys attempt to build ALL provider packages, but without
  --skip-tag-check - which means that if provider has been already
  released and it's version did not change in main, we are not going
  to build it locally and we will use it from PyPI. However if provider
  version is updated and the provider has not yet been released (checked
  by tag) - it will be build locally from sources and it will be used
  for constraint generation.

* for release PROD images build, on the other hand we NEVER build
  packages locally - we always rely on PyPI released packages, however
  if we are building pre-release version of airflow, we automatically
  add --pre flag that looks for pre-release packages in PyPI - this way
  pre-release version of airflow can be built with pre-release version
  of providers. We are still attempting to use constraints for that,
  however first - so unless there are no limits in apache airflow
  that prevent it from using released versions of providers, the
  constraint versions will be used - only if it fails, PROD images
  will fall back to non-constraint installation that will allow to
  use freely pre-release versions of packages from PyPI. This means
  for example that if we cherry-pick a change from main that increases
  minimum version of provider for apache-airflow to one that does not
  even have a pre-release version, building of rc version image for
  airflow will fail (which is a good thing). Lack of --pre flag
  for "release" version of Airlfow also means that if airlfow has
  a min version of provider that has no "released" version yet (only
  rc) - it will also fail (which is also a good thing)

* Update scripts/in_container/run_generate_constraints.py
@potiuk potiuk added this to the Airflow 3.0.1 milestone May 2, 2025
potiuk added a commit to potiuk/airflow that referenced this pull request May 2, 2025
…eased packages (apache#49799)

* Replace chicken-egg providers with automated use of unreleased packages

When we got rid of the .dev0 suffix, it is now possible to entirely
rely on building the packages locally using exsting mechanisms, that
check if packages have been already released - for CI builds, and can
rely on the fact that we need at least pre-release version of packages
if we are building pre-release version of airflow.

It works as follows:

* for CI builds (generate constraints and PROD image builds) - we are
  alwasys attempt to build ALL provider packages, but without
  --skip-tag-check - which means that if provider has been already
  released and it's version did not change in main, we are not going
  to build it locally and we will use it from PyPI. However if provider
  version is updated and the provider has not yet been released (checked
  by tag) - it will be build locally from sources and it will be used
  for constraint generation.

* for release PROD images build, on the other hand we NEVER build
  packages locally - we always rely on PyPI released packages, however
  if we are building pre-release version of airflow, we automatically
  add --pre flag that looks for pre-release packages in PyPI - this way
  pre-release version of airflow can be built with pre-release version
  of providers. We are still attempting to use constraints for that,
  however first - so unless there are no limits in apache airflow
  that prevent it from using released versions of providers, the
  constraint versions will be used - only if it fails, PROD images
  will fall back to non-constraint installation that will allow to
  use freely pre-release versions of packages from PyPI. This means
  for example that if we cherry-pick a change from main that increases
  minimum version of provider for apache-airflow to one that does not
  even have a pre-release version, building of rc version image for
  airflow will fail (which is a good thing). Lack of --pre flag
  for "release" version of Airlfow also means that if airlfow has
  a min version of provider that has no "released" version yet (only
  rc) - it will also fail (which is also a good thing)

* Update scripts/in_container/run_generate_constraints.py
(cherry picked from commit c6a2b8e)

Co-authored-by: Jarek Potiuk <[email protected]>
potiuk added a commit to potiuk/airflow that referenced this pull request May 2, 2025
…eased packages (apache#49799)

* Replace chicken-egg providers with automated use of unreleased packages

When we got rid of the .dev0 suffix, it is now possible to entirely
rely on building the packages locally using exsting mechanisms, that
check if packages have been already released - for CI builds, and can
rely on the fact that we need at least pre-release version of packages
if we are building pre-release version of airflow.

It works as follows:

* for CI builds (generate constraints and PROD image builds) - we are
  alwasys attempt to build ALL provider packages, but without
  --skip-tag-check - which means that if provider has been already
  released and it's version did not change in main, we are not going
  to build it locally and we will use it from PyPI. However if provider
  version is updated and the provider has not yet been released (checked
  by tag) - it will be build locally from sources and it will be used
  for constraint generation.

* for release PROD images build, on the other hand we NEVER build
  packages locally - we always rely on PyPI released packages, however
  if we are building pre-release version of airflow, we automatically
  add --pre flag that looks for pre-release packages in PyPI - this way
  pre-release version of airflow can be built with pre-release version
  of providers. We are still attempting to use constraints for that,
  however first - so unless there are no limits in apache airflow
  that prevent it from using released versions of providers, the
  constraint versions will be used - only if it fails, PROD images
  will fall back to non-constraint installation that will allow to
  use freely pre-release versions of packages from PyPI. This means
  for example that if we cherry-pick a change from main that increases
  minimum version of provider for apache-airflow to one that does not
  even have a pre-release version, building of rc version image for
  airflow will fail (which is a good thing). Lack of --pre flag
  for "release" version of Airlfow also means that if airlfow has
  a min version of provider that has no "released" version yet (only
  rc) - it will also fail (which is also a good thing)

* Update scripts/in_container/run_generate_constraints.py
(cherry picked from commit c6a2b8e)

Co-authored-by: Jarek Potiuk <[email protected]>
potiuk added a commit to potiuk/airflow that referenced this pull request May 2, 2025
…eased packages (apache#49799)

* Replace chicken-egg providers with automated use of unreleased packages

When we got rid of the .dev0 suffix, it is now possible to entirely
rely on building the packages locally using exsting mechanisms, that
check if packages have been already released - for CI builds, and can
rely on the fact that we need at least pre-release version of packages
if we are building pre-release version of airflow.

It works as follows:

* for CI builds (generate constraints and PROD image builds) - we are
  alwasys attempt to build ALL provider packages, but without
  --skip-tag-check - which means that if provider has been already
  released and it's version did not change in main, we are not going
  to build it locally and we will use it from PyPI. However if provider
  version is updated and the provider has not yet been released (checked
  by tag) - it will be build locally from sources and it will be used
  for constraint generation.

* for release PROD images build, on the other hand we NEVER build
  packages locally - we always rely on PyPI released packages, however
  if we are building pre-release version of airflow, we automatically
  add --pre flag that looks for pre-release packages in PyPI - this way
  pre-release version of airflow can be built with pre-release version
  of providers. We are still attempting to use constraints for that,
  however first - so unless there are no limits in apache airflow
  that prevent it from using released versions of providers, the
  constraint versions will be used - only if it fails, PROD images
  will fall back to non-constraint installation that will allow to
  use freely pre-release versions of packages from PyPI. This means
  for example that if we cherry-pick a change from main that increases
  minimum version of provider for apache-airflow to one that does not
  even have a pre-release version, building of rc version image for
  airflow will fail (which is a good thing). Lack of --pre flag
  for "release" version of Airlfow also means that if airlfow has
  a min version of provider that has no "released" version yet (only
  rc) - it will also fail (which is also a good thing)

* Update scripts/in_container/run_generate_constraints.py
(cherry picked from commit c6a2b8e)

Co-authored-by: Jarek Potiuk <[email protected]>
potiuk added a commit to potiuk/airflow that referenced this pull request May 2, 2025
…eased packages (apache#49799)

* Replace chicken-egg providers with automated use of unreleased packages

When we got rid of the .dev0 suffix, it is now possible to entirely
rely on building the packages locally using exsting mechanisms, that
check if packages have been already released - for CI builds, and can
rely on the fact that we need at least pre-release version of packages
if we are building pre-release version of airflow.

It works as follows:

* for CI builds (generate constraints and PROD image builds) - we are
  alwasys attempt to build ALL provider packages, but without
  --skip-tag-check - which means that if provider has been already
  released and it's version did not change in main, we are not going
  to build it locally and we will use it from PyPI. However if provider
  version is updated and the provider has not yet been released (checked
  by tag) - it will be build locally from sources and it will be used
  for constraint generation.

* for release PROD images build, on the other hand we NEVER build
  packages locally - we always rely on PyPI released packages, however
  if we are building pre-release version of airflow, we automatically
  add --pre flag that looks for pre-release packages in PyPI - this way
  pre-release version of airflow can be built with pre-release version
  of providers. We are still attempting to use constraints for that,
  however first - so unless there are no limits in apache airflow
  that prevent it from using released versions of providers, the
  constraint versions will be used - only if it fails, PROD images
  will fall back to non-constraint installation that will allow to
  use freely pre-release versions of packages from PyPI. This means
  for example that if we cherry-pick a change from main that increases
  minimum version of provider for apache-airflow to one that does not
  even have a pre-release version, building of rc version image for
  airflow will fail (which is a good thing). Lack of --pre flag
  for "release" version of Airlfow also means that if airlfow has
  a min version of provider that has no "released" version yet (only
  rc) - it will also fail (which is also a good thing)

* Update scripts/in_container/run_generate_constraints.py
(cherry picked from commit c6a2b8e)

Co-authored-by: Jarek Potiuk <[email protected]>
potiuk added a commit to potiuk/airflow that referenced this pull request May 2, 2025
…eased packages (apache#49799)

* Replace chicken-egg providers with automated use of unreleased packages

When we got rid of the .dev0 suffix, it is now possible to entirely
rely on building the packages locally using exsting mechanisms, that
check if packages have been already released - for CI builds, and can
rely on the fact that we need at least pre-release version of packages
if we are building pre-release version of airflow.

It works as follows:

* for CI builds (generate constraints and PROD image builds) - we are
  alwasys attempt to build ALL provider packages, but without
  --skip-tag-check - which means that if provider has been already
  released and it's version did not change in main, we are not going
  to build it locally and we will use it from PyPI. However if provider
  version is updated and the provider has not yet been released (checked
  by tag) - it will be build locally from sources and it will be used
  for constraint generation.

* for release PROD images build, on the other hand we NEVER build
  packages locally - we always rely on PyPI released packages, however
  if we are building pre-release version of airflow, we automatically
  add --pre flag that looks for pre-release packages in PyPI - this way
  pre-release version of airflow can be built with pre-release version
  of providers. We are still attempting to use constraints for that,
  however first - so unless there are no limits in apache airflow
  that prevent it from using released versions of providers, the
  constraint versions will be used - only if it fails, PROD images
  will fall back to non-constraint installation that will allow to
  use freely pre-release versions of packages from PyPI. This means
  for example that if we cherry-pick a change from main that increases
  minimum version of provider for apache-airflow to one that does not
  even have a pre-release version, building of rc version image for
  airflow will fail (which is a good thing). Lack of --pre flag
  for "release" version of Airlfow also means that if airlfow has
  a min version of provider that has no "released" version yet (only
  rc) - it will also fail (which is also a good thing)

* Update scripts/in_container/run_generate_constraints.py
(cherry picked from commit c6a2b8e)

Co-authored-by: Jarek Potiuk <[email protected]>
potiuk added a commit that referenced this pull request May 2, 2025
…eased packages (#49799) (#50109)

* Replace chicken-egg providers with automated use of unreleased packages

When we got rid of the .dev0 suffix, it is now possible to entirely
rely on building the packages locally using exsting mechanisms, that
check if packages have been already released - for CI builds, and can
rely on the fact that we need at least pre-release version of packages
if we are building pre-release version of airflow.

It works as follows:

* for CI builds (generate constraints and PROD image builds) - we are
  alwasys attempt to build ALL provider packages, but without
  --skip-tag-check - which means that if provider has been already
  released and it's version did not change in main, we are not going
  to build it locally and we will use it from PyPI. However if provider
  version is updated and the provider has not yet been released (checked
  by tag) - it will be build locally from sources and it will be used
  for constraint generation.

* for release PROD images build, on the other hand we NEVER build
  packages locally - we always rely on PyPI released packages, however
  if we are building pre-release version of airflow, we automatically
  add --pre flag that looks for pre-release packages in PyPI - this way
  pre-release version of airflow can be built with pre-release version
  of providers. We are still attempting to use constraints for that,
  however first - so unless there are no limits in apache airflow
  that prevent it from using released versions of providers, the
  constraint versions will be used - only if it fails, PROD images
  will fall back to non-constraint installation that will allow to
  use freely pre-release versions of packages from PyPI. This means
  for example that if we cherry-pick a change from main that increases
  minimum version of provider for apache-airflow to one that does not
  even have a pre-release version, building of rc version image for
  airflow will fail (which is a good thing). Lack of --pre flag
  for "release" version of Airlfow also means that if airlfow has
  a min version of provider that has no "released" version yet (only
  rc) - it will also fail (which is also a good thing)

* Update scripts/in_container/run_generate_constraints.py
(cherry picked from commit c6a2b8e)
potiuk added a commit to potiuk/airflow that referenced this pull request May 2, 2025
When we removed "chicken-egg-providers" from the build in apache#49799 we
should be able now to build PROD image in v3*test branch without special
case of only using PyPI providers. This should work because in
v3-*test branch we automatically detect (by presence of relesed
tag) whether we should build provider from sources or use it from
PyPI. In normal circumstances, we would need use providers fro
PyPI, but sometimes (For example for 3.0.1) we need to use some
providers from sources before they get released (for example
common.messaging 1.0.1 that is the lowest possible common.messaging
for Airflow 3.0.1.

This PR removes that special case and switches to the same method
of building PROD image in main and in release branch.
potiuk added a commit to potiuk/airflow that referenced this pull request May 2, 2025
When we removed "chicken-egg-providers" from the build in apache#49799 we
should be able now to build PROD image in v3*test branch without special
case of only using PyPI providers. This should work because in
v3-*test branch we automatically detect (by presence of relesed
tag) whether we should build provider from sources or use it from
PyPI. In normal circumstances, we would need use providers fro
PyPI, but sometimes (For example for 3.0.1) we need to use some
providers from sources before they get released (for example
common.messaging 1.0.1 that is the lowest possible common.messaging
for Airflow 3.0.1.

This PR removes that special case and switches to the same method
of building PROD image in main and in release branch.
potiuk added a commit to potiuk/airflow that referenced this pull request May 2, 2025
When we removed "chicken-egg-providers" from the build in apache#49799 we
should be able now to build PROD image in v3*test branch without special
case of only using PyPI providers. This should work because in
v3-*test branch we automatically detect (by presence of relesed
tag) whether we should build provider from sources or use it from
PyPI. In normal circumstances, we would need use providers fro
PyPI, but sometimes (For example for 3.0.1) we need to use some
providers from sources before they get released (for example
common.messaging 1.0.1 that is the lowest possible common.messaging
for Airflow 3.0.1.

This PR removes that special case and switches to the same method
of building PROD image in main and in release branch.
potiuk added a commit that referenced this pull request May 2, 2025
When we removed "chicken-egg-providers" from the build in #49799 we
should be able now to build PROD image in v3*test branch without special
case of only using PyPI providers. This should work because in
v3-*test branch we automatically detect (by presence of relesed
tag) whether we should build provider from sources or use it from
PyPI. In normal circumstances, we would need use providers fro
PyPI, but sometimes (For example for 3.0.1) we need to use some
providers from sources before they get released (for example
common.messaging 1.0.1 that is the lowest possible common.messaging
for Airflow 3.0.1.

This PR removes that special case and switches to the same method
of building PROD image in main and in release branch.
potiuk added a commit to potiuk/airflow that referenced this pull request May 2, 2025
…ranch (apache#50115)

When we removed "chicken-egg-providers" from the build in apache#49799 we
should be able now to build PROD image in v3*test branch without special
case of only using PyPI providers. This should work because in
v3-*test branch we automatically detect (by presence of relesed
tag) whether we should build provider from sources or use it from
PyPI. In normal circumstances, we would need use providers fro
PyPI, but sometimes (For example for 3.0.1) we need to use some
providers from sources before they get released (for example
common.messaging 1.0.1 that is the lowest possible common.messaging
for Airflow 3.0.1.

This PR removes that special case and switches to the same method
of building PROD image in main and in release branch.
(cherry picked from commit 9deb2a7)

Co-authored-by: Jarek Potiuk <[email protected]>
potiuk added a commit that referenced this pull request May 2, 2025
…ranch (#50115) (#50118)

When we removed "chicken-egg-providers" from the build in #49799 we
should be able now to build PROD image in v3*test branch without special
case of only using PyPI providers. This should work because in
v3-*test branch we automatically detect (by presence of relesed
tag) whether we should build provider from sources or use it from
PyPI. In normal circumstances, we would need use providers fro
PyPI, but sometimes (For example for 3.0.1) we need to use some
providers from sources before they get released (for example
common.messaging 1.0.1 that is the lowest possible common.messaging
for Airflow 3.0.1.

This PR removes that special case and switches to the same method
of building PROD image in main and in release branch.
(cherry picked from commit 9deb2a7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants