Skip to content

Commit ce44b62

Browse files
potiukpotiuk
authored
Add Python 3.9 support (#15515)
This includes several things: * added per-provider support for python version. Each provider can now declare python versions it does not support * excluded ldap core extra from Python 3.9. * skip relevant tests in Python 3.9
1 parent 5117aaa commit ce44b62

36 files changed

+167
-62
lines changed

‎BREEZE.rst‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1264,7 +1264,7 @@ This is the current syntax for `./breeze <./breeze>`_:
12641264
12651265
One of:
12661266
1267-
3.6 3.7 3.8
1267+
3.6 3.7 3.8 3.9
12681268
12691269
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
12701270
Uses different version of Airflow when building PROD image.
@@ -1494,7 +1494,7 @@ This is the current syntax for `./breeze <./breeze>`_:
14941494
14951495
One of:
14961496
1497-
3.6 3.7 3.8
1497+
3.6 3.7 3.8 3.9
14981498
14991499
-I, --production-image
15001500
Use production image for entering the environment and builds (not for tests).
@@ -1561,7 +1561,7 @@ This is the current syntax for `./breeze <./breeze>`_:
15611561
15621562
One of:
15631563
1564-
3.6 3.7 3.8
1564+
3.6 3.7 3.8 3.9
15651565
15661566
-v, --verbose
15671567
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
@@ -1684,7 +1684,7 @@ This is the current syntax for `./breeze <./breeze>`_:
16841684
16851685
One of:
16861686
1687-
3.6 3.7 3.8
1687+
3.6 3.7 3.8 3.9
16881688
16891689
16901690
####################################################################################################
@@ -1879,7 +1879,7 @@ This is the current syntax for `./breeze <./breeze>`_:
18791879
18801880
One of:
18811881
1882-
3.6 3.7 3.8
1882+
3.6 3.7 3.8 3.9
18831883
18841884
-b, --backend BACKEND
18851885
Backend to use for tests - it determines which database is used.
@@ -1943,7 +1943,7 @@ This is the current syntax for `./breeze <./breeze>`_:
19431943
19441944
One of:
19451945
1946-
3.6 3.7 3.8
1946+
3.6 3.7 3.8 3.9
19471947
19481948
-F, --force-build-images
19491949
Forces building of the local docker images. The images are rebuilt
@@ -2351,7 +2351,7 @@ This is the current syntax for `./breeze <./breeze>`_:
23512351
23522352
One of:
23532353
2354-
3.6 3.7 3.8
2354+
3.6 3.7 3.8 3.9
23552355
23562356
****************************************************************************************************
23572357
Choose backend to run for Airflow

‎CI.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ Container Registry used as cache
5757
For the CI builds of our we are using Container Registry to store results of the "Build Image" workflow
5858
and pass it to the "CI Build" workflow.
5959

60-
Currently in main version of Airflow we run tests in 3 different versions of Python (3.6, 3.7, 3.8)
60+
Currently in main version of Airflow we run tests in 4 different versions of Python (3.6, 3.7, 3.8, 3.9)
6161
which means that we have to build 6 images (3 CI ones and 3 PROD ones). Yet we run around 12 jobs
6262
with each of the CI images. That is a lot of time to just build the environment to run. Therefore
6363
we are utilising ``pull_request_target`` feature of GitHub Actions.
@@ -779,7 +779,7 @@ The image names follow the patterns:
779779
+--------------+----------------------------+--------------------------------+--------------------------------------------------------------------------------------------+
780780
781781
* <BRANCH> might be either "main" or "v1-10-test" or "v2-*-test"
782-
* <X.Y> - Python version (Major + Minor). For "main" and "v2-*-test" should be in ["3.6", "3.7", "3.8"].
782+
* <X.Y> - Python version (Major + Minor). For "main" and "v2-*-test" should be in ["3.6", "3.7", "3.8", "3.9"].
783783
* <COMMIT_SHA> - for images that get merged to "main", "v2-*-test" of "v1-10-test", or built as part of a
784784
pull request the images are tagged with the (full length) commit SHA of that particular branch. For pull
785785
requests the SHA used is the tip of the pull request branch.

‎CONTRIBUTING.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -873,7 +873,7 @@ This can be done by running this (it utilizes parallel preparation of the constr
873873

874874
.. code-block:: bash
875875
876-
export CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING="3.6 3.7 3.8"
876+
export CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING="3.6 3.7 3.8 3.9"
877877
for python_version in $(echo "${CURRENT_PYTHON_MAJOR_MINOR_VERSIONS_AS_STRING}")
878878
do
879879
./breeze build-image --upgrade-to-newer-dependencies --python ${python_version} --build-cache-local

‎CONTRIBUTORS_QUICK_START.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ Pyenv and setting up virtual-env
135135
libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
136136
xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
137137
138-
$ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \
138+
$ sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python3.9-dev python-dev openssl \
139139
sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql
140140
141141
2. Install pyenv

‎Dockerfile‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ ARG AIRFLOW_GID="50000"
4444

4545
ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
4646

47-
ARG AIRFLOW_PIP_VERSION=21.1.1
47+
ARG AIRFLOW_PIP_VERSION=21.1.2
4848

4949
# By default PIP has progress bar but you can disable it.
5050
ARG PIP_PROGRESS_BAR="on"
@@ -232,11 +232,10 @@ ARG INSTALL_FROM_DOCKER_CONTEXT_FILES=""
232232
ARG INSTALL_FROM_PYPI="true"
233233
# Those are additional constraints that are needed for some extras but we do not want to
234234
# Force them on the main Airflow package.
235-
# * chardet<4 and certifi<2021.0.0 required to keep snowflake happy
236-
# * urllib3 - required to keep boto3 happy
235+
# * certifi<2021.0.0 required to keep snowflake happy
237236
# * pyjwt<2.0.0: flask-jwt-extended requires it
238237
# * dill<0.3.3 required by apache-beam
239-
ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0"
238+
ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0"
240239
ARG CONTINUE_ON_PIP_CHECK_FAILURE="false"
241240

242241

‎Dockerfile.ci‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -219,7 +219,7 @@ ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
219219
# By default in the image, we are installing all providers when installing from sources
220220
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
221221
ARG INSTALL_FROM_PYPI="true"
222-
ARG AIRFLOW_PIP_VERSION=21.1.1
222+
ARG AIRFLOW_PIP_VERSION=21.1.2
223223
# Setup PIP
224224
# By default PIP install run without cache to make image smaller
225225
ARG PIP_NO_CACHE_DIR="true"
@@ -263,13 +263,11 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
263263

264264
# Those are additional constraints that are needed for some extras but we do not want to
265265
# force them on the main Airflow package. Those limitations are:
266-
# * chardet<4 and certifi<2021.0.0: required by snowflake provider
266+
# * certifi<2021.0.0: required by snowflake provider
267267
# * lazy-object-proxy<1.5.0: required by astroid
268-
# * pyOpenSSL: required by snowflake provider https://github.com/snowflakedb/snowflake-connector-python/blob/v2.3.6/setup.py#L201
269-
# * urllib3<1.26: Required to keep boto3 happy
270268
# * pyjwt<2.0.0: flask-jwt-extended requires it
271269
# * dill<0.3.3 required by apache-beam
272-
ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 lazy-object-proxy<1.5.0 pyOpenSSL<20.0.0 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0"
270+
ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="lazy-object-proxy<1.5.0 pyjwt<2.0.0 dill<0.3.3 certifi<2021.0.0"
273271
ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
274272
ENV EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS} \
275273
UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES}

‎IMAGES.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ where:
8181
built from branches so they change over time. The ``2.*.*`` labels are built from git tags
8282
and they are "fixed" once built.
8383
* ``PYTHON_MAJOR_MINOR_VERSION`` - version of Python used to build the image. Examples: ``3.6``, ``3.7``,
84-
``3.8``
84+
``3.8``, ``3.9``
8585
* The ``-ci`` suffix is added for CI images
8686
* The ``-manifest`` is added for manifest images (see below for explanation of manifest images)
8787

‎LOCAL_VIRTUALENV.rst‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,8 @@ Required Software Packages
5151
Use system-level package managers like yum, apt-get for Linux, or
5252
Homebrew for macOS to install required software packages:
5353

54-
* Python (One of: 3.6, 3.7, 3.8)
55-
* MySQL
54+
* Python (One of: 3.6, 3.7, 3.8, 3.9)
55+
* MySQL 5.7+
5656
* libxml
5757

5858
Refer to the `Dockerfile.ci <Dockerfile.ci>`__ for a comprehensive list
@@ -102,7 +102,7 @@ Creating a Local virtualenv
102102

103103
To use your IDE for Airflow development and testing, you need to configure a virtual
104104
environment. Ideally you should set up virtualenv for all Python versions that Airflow
105-
supports (3.6, 3.7, 3.8).
105+
supports (3.6, 3.7, 3.8, 3.9).
106106

107107
To create and initialize the local virtualenv:
108108

‎PULL_REQUEST_WORKFLOW.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ We approached the problem by:
5858
3) Even more optimisation came from limiting the scope of tests to only "default" matrix parameters. So far
5959
in Airflow we always run all tests for all matrix combinations. The primary matrix components are:
6060

61-
* Python versions (currently 3.6, 3.7, 3.8)
61+
* Python versions (currently 3.6, 3.7, 3.8, 3.9)
6262
* Backend types (currently MySQL/Postgres)
6363
* Backed version (currently MySQL 5.7, MySQL 8, Postgres 9.6, Postgres 13
6464

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -127,9 +127,9 @@ We **highly** recommend upgrading to the latest Airflow major release at the ear
127127

128128
Apache Airflow is tested with:
129129

130-
| | Main version (dev) | Stable version (2.0.2) |
130+
| | Main version (dev) | Stable version (2.1.0) |
131131
| -------------------- | ------------------------- | ------------------------ |
132-
| Python | 3.6, 3.7, 3.8 | 3.6, 3.7, 3.8 |
132+
| Python | 3.6, 3.7, 3.8, 3.9 | 3.6, 3.7, 3.8 |
133133
| Kubernetes | 1.20, 1.19, 1.18 | 1.20, 1.19, 1.18 |
134134
| PostgreSQL | 9.6, 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 |
135135
| MySQL | 5.7, 8 | 5.7, 8 |

0 commit comments

Comments
 (0)