Add Python 3.13 support for Airflow. by potiuk · Pull Request #46891 · apache/airflow

potiuk · 2025-02-19T10:55:53Z

Added Python 3.13 support across the codebase, including:

CI/CD workflow updates to handle Python 3.13.
Dockerfile and environment variable adjustments for compatibility.
Documentation updates to list Python 3.13 as supported.
Dependency and code changes for compatibility with new/changed
libraries (e.g., numpy 2.x, greenlet, pendulum).
Test and plugin manager adjustments for Python 3.13-specific behaviors.
Removal or skipping of tests and features not compatible with Python 3.13.
Improved error handling and logging for unsupported plugins/providers
on Python 3.13.
Updated all provider README.rst files to:
- List Python 3.13 as a supported version.
- Adjust dependency requirements for Python 3.13 (e.g., pandas,
  pyarrow, ray, etc.).
- Add or update notes about excluded or conditionally supported
  dependencies for Python 3.13.
- Document any known incompatibilities or workarounds for specific
  providers.
Updated all provider pyproject.toml files and automation to:
- Add Python 3.13 to the supported classifiers.
- Adjust requires-python and dependency constraints for Python 3.13.
- Exclude Python 3.13 for providers that are not yet compatible (e.g.,
  those depending on Flask AppBuilder).
- Add conditional dependencies for new versions of libraries required
  by Python 3.13.
Updated Breeze and related scripts to:
Build and test with Python 3.13.
Update documentation and release scripts to include Python 3.13.
Ensure all dev tooling and test environments work with Python 3.13.
Removed remaining FAB related code from core and removed tests for
older FAB provider versions (<1.3.0)
some of the tests in core still depend on FAB - but we make them
optional - both for parsing (we need to skip some imports) and
execution - because they test a legacy behaviour of embedding
Flask plugins that is still supported in core.
k8s tests now run also with SimpleAuthManager and SimpleAuthManager
is used automatically when runnin tests in Python 3.13
migration tests had to be excluded fo Python 3.13 because we cannot
migrate down on Python 3.13 without FAB tables created. The tests
will be brought back when FAB is supported or when we release 3.1
and start working on 3.2, we should be able to migrate down to 3.1
Weaviate tests have been updated to handle case where weaviate-client
is > 2.10.0 - because it started to use httpx instead of requests,
and Python 3.13 actually unblocks using older version of weaviate
(it was previously blocked indirectly via Flask dependencies and
grpcio).
Added doc change in our rules on what is the "default" Python version
for airflow images - handling the case where we cannot use "newest"
Python version as default where not all "regular" image providers are
supported in the latest version
Connection is now flushed in configure_git_connection_for_dag_bundle
just before clear_db_connections - in order to handle Python 3.13
case where FAB app is not initialized by other fixtures - performing
implicit flush along the way
Cleaning db connections was added to test_dag_run and test_dags test
in fast_api as the test implicitly rely on having connections
cleared (no test git connections added) - this will make the tests
side-effect free.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

docker-stack-docs/index.rst

airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_plugins.py

airflow-core/tests/unit/api_fastapi/conftest.py

amoghrajesh

Some qns / nits on commit 2

.pre-commit-config.yaml

airflow-core/pyproject.toml

dev/breeze/tests/test_selective_checks.py

providers/amazon/pyproject.toml

providers/apache/beam/pyproject.toml

scripts/ci/pre_commit/update_airflow_pyproject_toml.py

airflow-core/pyproject.toml

airflow-core/src/airflow/api_fastapi/core_api/app.py

airflow-core/tests/unit/always/test_example_dags.py

airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py

airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dags.py

airflow-core/tests/unit/cli/commands/test_api_server_command.py

airflow-core/tests/unit/serialization/test_dag_serialization.py

task-sdk/pyproject.toml

aritra24

This is great Jarek! Had a minor nit, but other than that all looks good from a high level.

Dockerfile.ci

potiuk · 2025-07-17T00:08:35Z

Pushed changes trying out the comments in places I am not sure.

gopidesupavan

Thanks @potiuk for all the work, may iterations not an easy task .

gopidesupavan · 2025-07-17T10:38:13Z

#protm

Added Python 3.13 support across the codebase, including: * CI/CD workflow updates to handle Python 3.13. * Dockerfile and environment variable adjustments for compatibility. * Documentation updates to list Python 3.13 as supported. * Dependency and code changes for compatibility with new/changed libraries (e.g., numpy 2.x, greenlet, pendulum). * Test and plugin manager adjustments for Python 3.13-specific behaviors. * Removal or skipping of tests and features not compatible with Python 3.13. * Improved error handling and logging for unsupported plugins/providers on Python 3.13. * Updated all provider README.rst files to: * List Python 3.13 as a supported version. * Adjust dependency requirements for Python 3.13 (e.g., pandas, pyarrow, ray, etc.). * Add or update notes about excluded or conditionally supported dependencies for Python 3.13. * Document any known incompatibilities or workarounds for specific providers. * Updated all provider pyproject.toml files and automation to: * Add Python 3.13 to the supported classifiers. * Adjust requires-python and dependency constraints for Python 3.13. * Exclude Python 3.13 for providers that are not yet compatible (e.g., those depending on Flask AppBuilder). * Add conditional dependencies for new versions of libraries required by Python 3.13. * Updated Breeze and related scripts to: * Build and test with Python 3.13. * Update documentation and release scripts to include Python 3.13. * Ensure all dev tooling and test environments work with Python 3.13. * Removed remaining FAB related code from core and removed tests for older FAB provider versions (<1.3.0) * some of the tests in core still depend on FAB - but we make them optional - both for parsing (we need to skip some imports) and execution - because they test a legacy behaviour of embedding Flask plugins that is still supported in core. * k8s tests now run also with SimpleAuthManager and SimpleAuthManager is used automatically when runnin tests in Python 3.13 * migration tests had to be excluded fo Python 3.13 because we cannot migrate down on Python 3.13 without FAB tables created. The tests will be brought back when FAB is supported or when we release 3.1 and start working on 3.2, we should be able to migrate down to 3.1 * Weaviate tests have been updated to handle case where weaviate-client is > 2.10.0 - because it started to use httpx instead of requests, and Python 3.13 actually unblocks using older version of weaviate (it was previously blocked indirectly via Flask dependencies and grpcio). * Added doc change in our rules on what is the "default" Python version for airflow images - handling the case where we cannot use "newest" Python version as default where not all "regular" image providers are supported in the latest version * Connection is now flushed in `configure_git_connection_for_dag_bundle` just before `clear_db_connections` - in order to handle Python 3.13 case where FAB app is not initialized by other fixtures - performing implicit flush along the way * Cleaning db connections was added to test_dag_run and test_dags test in fast_api as the test implicitly rely on having connections cleared (no test git connections added) - this will make the tests side-effect free. * Conditionally skipping some serialization tests involving yandex and ray on Python 3.13 until they support Python 3.13

potiuk · 2025-07-17T14:53:57Z

Aaand merged :)

aritra24 · 2025-07-17T15:00:56Z

Long effort!! Great job!

shahar1 · 2025-07-17T17:15:30Z

#protm for sure, great job Jarek! :)

eladkal · 2025-07-17T18:13:08Z

🎉 🎉🎉

jscheffl · 2025-07-17T19:23:34Z

Wohoo! Party!

kyungjunleeme · 2025-07-18T13:35:01Z

Cool!

This change came from apache#46891, but that was a much bigger change to enable Python 3.13 which we don't want to backport

This change came from #46891, but that was a much bigger change to enable Python 3.13 which we don't want to backport (cherry picked from commit 488744c)

boring-cyborg bot added the area:dev-tools label Feb 19, 2025

potiuk force-pushed the add-python-3-13-support branch from 9888b0f to f4f2031 Compare February 19, 2025 17:16

potiuk added the legacy api Whether legacy API changes should be allowed in PR label Feb 19, 2025

potiuk force-pushed the add-python-3-13-support branch 3 times, most recently from 9c6e93c to b40bf29 Compare February 19, 2025 17:44

potiuk mentioned this pull request Feb 19, 2025

Workspace incorectly resolves dependencies when they are conditionally excluded astral-sh/uv#11629

Open

potiuk force-pushed the add-python-3-13-support branch from b40bf29 to 21a34cb Compare February 19, 2025 18:22

Lee-W self-requested a review February 20, 2025 07:04

potiuk mentioned this pull request Feb 20, 2025

Enables python 3.13 python-pendulum/pendulum#871

Merged

potiuk force-pushed the add-python-3-13-support branch 2 times, most recently from 004dd04 to ec6553d Compare February 20, 2025 15:41

potiuk mentioned this pull request Feb 23, 2025

Python 3.13 google/re2#516

Closed

potiuk force-pushed the add-python-3-13-support branch 4 times, most recently from 9d382f4 to 25c5d7a Compare March 1, 2025 16:21

potiuk force-pushed the add-python-3-13-support branch 3 times, most recently from 50fc7ee to be6b595 Compare March 9, 2025 21:52

potiuk force-pushed the add-python-3-13-support branch 3 times, most recently from 4cf2cc3 to b9c8b2b Compare April 13, 2025 12:26

potiuk added full tests needed We need to run full set of tests for this PR to merge all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs labels Apr 13, 2025

potiuk force-pushed the add-python-3-13-support branch 2 times, most recently from 5e1cd92 to c85447f Compare April 13, 2025 13:32

potiuk force-pushed the add-python-3-13-support branch 3 times, most recently from e39c0f9 to 2d88b64 Compare April 27, 2025 21:07

potiuk mentioned this pull request Jul 15, 2025

Added a hook to check whether provider exclusions for a specific python version match across relevant files #53355

Closed

jscheffl reviewed Jul 15, 2025

View reviewed changes

docker-stack-docs/index.rst Outdated Show resolved Hide resolved

vincbeck reviewed Jul 15, 2025

View reviewed changes

airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_plugins.py Outdated Show resolved Hide resolved

airflow-core/tests/unit/api_fastapi/conftest.py Outdated Show resolved Hide resolved

amoghrajesh reviewed Jul 16, 2025

View reviewed changes

aritra24 approved these changes Jul 16, 2025

View reviewed changes

Dockerfile.ci Outdated Show resolved Hide resolved

potiuk force-pushed the add-python-3-13-support branch from d34f626 to 7ef1f90 Compare July 17, 2025 00:08

aritra24 approved these changes Jul 17, 2025

View reviewed changes

potiuk force-pushed the add-python-3-13-support branch from 7ef1f90 to bd6bc76 Compare July 17, 2025 07:52

potiuk added 3 commits July 17, 2025 11:38

Breeze changes for Python 3.13

765d934

Pyproject.toml provider changes for Python 3.13

cb9a9f0

README.rst changes for providers for Python 3.13

f0c9d23

gopidesupavan approved these changes Jul 17, 2025

View reviewed changes

potiuk force-pushed the add-python-3-13-support branch from bd6bc76 to 0e4765f Compare July 17, 2025 11:42

potiuk mentioned this pull request Jul 17, 2025

[Bug] DAGs Fail Periodically Every 2–3 Days in Airflow 3.0.0 (Restart Fixes It) #51512

Closed

2 tasks

potiuk merged commit db3bf8f into apache:main Jul 17, 2025
190 checks passed

potiuk deleted the add-python-3-13-support branch July 17, 2025 14:53

astrojuanlu mentioned this pull request Jul 17, 2025

Python 3.13 support kedro-org/kedro-plugins#940

Open

4 tasks

eladkal mentioned this pull request Jul 29, 2025

Status of testing Providers that were prepared on July 29, 2025 #53873

Closed

ashb added a commit to astronomer/airflow that referenced this pull request Aug 2, 2025

Fixup constraint generation script

488744c

This change came from apache#46891, but that was a much bigger change to enable Python 3.13 which we don't want to backport

potiuk pushed a commit that referenced this pull request Aug 3, 2025

Fixup constraint generation script

70b83be

This change came from #46891, but that was a much bigger change to enable Python 3.13 which we don't want to backport (cherry picked from commit 488744c)

Conversation

potiuk commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amoghrajesh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aritra24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

potiuk commented Jul 17, 2025

Uh oh!

gopidesupavan left a comment

Choose a reason for hiding this comment

Uh oh!

gopidesupavan commented Jul 17, 2025

Uh oh!

Uh oh!

potiuk commented Jul 17, 2025

Uh oh!

aritra24 commented Jul 17, 2025

Uh oh!

shahar1 commented Jul 17, 2025

Uh oh!

eladkal commented Jul 17, 2025

Uh oh!

jscheffl commented Jul 17, 2025

Uh oh!

kyungjunleeme commented Jul 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

potiuk commented Feb 19, 2025 •

edited

Loading