Repository navigation
Lower cross-provider dependencies in provider lowest-deps tests - #73129
namanjain24-sudo wants to merge 2 commits into
Conversation
|
Small ping on this one. It has been ready since Monday and the only red check is the Build info label gate rather than anything in the code. The gate fires on the two provider Two ways forward, whichever is easier to review: set the I rechecked it on current main today. The branch still merges cleanly and the 22 tests in |
ad5a769 to
f45ffb8
Compare
The lowest-deps provider job syncs with --resolution lowest-direct, but other providers are workspace members, so they were always installed from the checkout and their declared lower bounds were never tested. After the sync, pick the lowest final release on PyPI that every installed distribution accepts for each provider the tested provider depends on, install those pins and run uv pip check. Dependencies marked "# use next version", without a lower bound, or with only a release candidate available keep the workspace version. Running it over all providers exposed two lower bounds that do not work with Airflow 3, so bump them: apache.flink needs cncf-kubernetes>=9.0.0 (older releases import airflow.compat) and airflow-core needs common-sql>=1.30.2 (older releases import airflow.utils.module_loading, removed in 3.2.0).
…ovider needs it unreleased lower_cross_provider_dependencies.py picks a PyPI release for a cross-provider dependency by intersecting every installed provider's declared lower bound, but it only checked the "# use next version" marker on the pyproject.toml of the provider being tested. A provider pulled in incidentally (e.g. through another provider's extra) can carry that marker on its own dependency line to say it needs content that has not been released yet, and that marker was invisible to the intersection. The result was a real regression: testing "google" (which pulls in "fab" through its "fab" extra) could pin common-compat to a released version that predates a symbol fab's current source imports unconditionally, breaking test collection with an ImportError. combined_specifier now also reads each requirer's own pyproject.toml and excludes it from the intersection when it carries that marker, keeping the dependency at its workspace version instead.
f45ffb8 to
06ff439
Compare
potiuk
added
the
closed because of open PR limit
|
Hello @namanjain24-sudo - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 7 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
The lowest-deps provider tests sync with
--resolution lowest-direct, but every other provider is a workspace member, so cross-provider dependencies always came from the checkout. A provider declaringcommon-compat>=1.8.0was really tested against the currentcommon-compat.uv can't be told to skip that for a few packages here:
--no-sources-packagefails in this workspace ("references a workspace in tool.uv.sources but is not a workspace member"), and dropping providers from the workspace breaks the lock while some providers pin versions that only exist as release candidates. So the lowering now happens right after the sync:scripts/in_container/lower_cross_provider_dependencies.pyreads the tested provider's[project].dependencies. For each provider dependency with a lower bound it picks the lowest final (non-yanked) release that every installed distribution accepts, looking up releases on pypi.org, then repeats with the pinned releases' own requirements until the pins stop changing. Lines with# use next version, dependencies without a lower bound, and ones where only an rc satisfies the bounds keep the workspace version. It logs why each dependency was pinned or kept.entrypoint_ci.shinstalls those pins and runsuv pip check. They are installed with their dependencies on purpose: an oldcncf-kubernetesneeds an oldkubernetesclient, and--no-depswould leave a broken environment.Running this over all providers found two lower bounds that don't work, fixed here so the job stays green:
apache.flinkrequiredcncf-kubernetes>=5.1.0, but 36 of the 39 releases from 5.1.0 to 8.4.2, including every one from 7.4.0 on, importairflow.compat, which Airflow 3 no longer has. flink's tests fail to import on those I tried (5.1.0, 7.4.0, 8.0.0–8.1.1, 8.3.0–8.4.2). 9.0.0 is the first release without that import, and the tests pass on 9.0.0, 10.0.0, 10.5.0, 10.6.0 and 10.12.2. Bumped to>=9.0.0.airflow-corerequirescommon-sql>=1.28.1, but common-sql releases before 1.30.2 importairflow.utils.module_loading, removed in 3.2.0 by Extract shared "module_loading" distribution #59139. Bumped to>=1.30.2, the same way the pre-installed minimums were bumped in Bump version of providers #56208. This also coversapache.arrowandslack, whose own bounds are lower.The two bumps above change
>=bounds on provider dependencies, which the selective check reserves for release managers, so CI stops at "Build info" until a maintainer who agrees with them sets theallow provider dependency bumplabel. If you'd rather have them in a separate PR, I'm happy to split them out; without them the lowest-deps jobs forapache.flinkandapache.arrowwould fail.What I ran:
uv pip check, tests; failing ones compared with a run without lowering)uv pip checkclean for every provider that synced.apache.arrowwas the only failure caused by the lowering, fixed by the core bump above. Every other test that failed with the lowering also failed in a run without it. Providers that could not sync or run their tests on my Mac (native libraries, an SSH tunnel test) are in the Breeze row.--force-lowest-dependenciesuv pip checkcleanscripts/tests/in_container/test_lower_cross_provider_dependencies.pycloses: #60888
Was generative AI tooling used to co-author this PR?
Generated-by: a Gen-AI coding assistant, following the guidelines. I reviewed the change and ran the checks above.