Skip to content

Commit fb87800

Browse files
potiukpotiuk
authored
Cleaner output of docker image building scripts (#20679)
1 parent 39d8f93 commit fb87800

19 files changed

+84
-65
lines changed

‎Dockerfile‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ ARG AIRFLOW_UID="50000"
4343

4444
ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
4545

46-
ARG AIRFLOW_PIP_VERSION=21.2.4
46+
ARG AIRFLOW_PIP_VERSION=21.3.1
4747
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
4848

4949
# By default PIP has progress bar but you can disable it.
@@ -186,9 +186,6 @@ ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \
186186
PATH=${PATH}:/root/.local/bin \
187187
AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
188188
PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR} \
189-
# Install Airflow with "--user" flag, so that we can copy the whole .local folder to the final image
190-
# from the build image and always in non-editable mode
191-
AIRFLOW_INSTALL_USER_FLAG="--user" \
192189
AIRFLOW_INSTALL_EDITABLE_FLAG="" \
193190
UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES}
194191

‎Dockerfile.ci‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -223,7 +223,7 @@ ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
223223
# By default in the image, we are installing all providers when installing from sources
224224
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
225225
ARG INSTALL_FROM_PYPI="true"
226-
ARG AIRFLOW_PIP_VERSION=21.2.4
226+
ARG AIRFLOW_PIP_VERSION=21.3.1
227227
# Setup PIP
228228
# By default PIP install run without cache to make image smaller
229229
ARG PIP_NO_CACHE_DIR="true"
@@ -258,7 +258,6 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
258258
INSTALL_MYSQL_CLIENT="true" \
259259
INSTALL_MSSQL_CLIENT="true" \
260260
AIRFLOW_INSTALLATION_METHOD="." \
261-
AIRFLOW_INSTALL_USER_FLAG="" \
262261
AIRFLOW_INSTALL_EDITABLE_FLAG="--editable" \
263262
AIRFLOW_VERSION_SPECIFICATION="" \
264263
PIP_NO_CACHE_DIR=${PIP_NO_CACHE_DIR} \

‎IMAGES.rst‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -41,14 +41,13 @@ to run Kubernetes tests. See below for the list of arguments that should be prov
4141
production image from the local sources.
4242

4343
The image is primarily optimised for size of the final image, but also for speed of rebuilds - the
44-
'airflow-build-image' segment uses the same technique as the CI jobs for pre-installing PIP dependencies.
44+
'airflow-build-image' segment uses the same technique as the CI jobs for pre-installing dependencies.
4545
It first pre-installs them from the right GitHub branch and only after that final airflow installation is
46-
done from either local sources or remote location (PIP or GitHub repository).
46+
done from either local sources or remote location (PyPI or GitHub repository).
4747

4848
You can read more details about building, extending and customizing the PROD image in the
4949
`Latest documentation <https://airflow.apache.org/docs/docker-stack/index.html>`_
5050

51-
5251
CI image
5352
--------
5453

@@ -492,7 +491,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
492491
| ``ADDITIONAL_RUNTIME_APT_ENV`` | | Additional env variables defined |
493492
| | | when installing runtime deps |
494493
+------------------------------------------+------------------------------------------+------------------------------------------+
495-
| ``AIRFLOW_PIP_VERSION`` | ``21.2.4`` | PIP version used. |
494+
| ``AIRFLOW_PIP_VERSION`` | ``21.3.1`` | PIP version used. |
496495
+------------------------------------------+------------------------------------------+------------------------------------------+
497496
| ``PIP_PROGRESS_BAR`` | ``on`` | Progress bar for PIP installation |
498497
+------------------------------------------+------------------------------------------+------------------------------------------+

‎dev/README_RELEASE_PROVIDER_PACKAGES.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -611,9 +611,9 @@ provider packages. This is especially helpful when you want to test integrations
611611
additional tools. Below is an example Dockerfile, which installs providers for Google/
612612

613613
```dockerfile
614-
FROM apache/airflow:2.0.0
614+
FROM apache/airflow:2.2.3
615615

616-
RUN pip install --upgrade --user apache-airflow-providers-google==2.0.0.rc1
616+
RUN pip install --user apache-airflow-providers-google==2.2.2.rc1
617617

618618
USER ${AIRFLOW_UID}
619619
```

‎docs/docker-stack/build-arg-ref.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ Those are the most common arguments that you use when you want to build a custom
4545
+------------------------------------------+------------------------------------------+---------------------------------------------+
4646
| ``AIRFLOW_USER_HOME_DIR`` | ``/home/airflow`` | Home directory of the Airflow user. |
4747
+------------------------------------------+------------------------------------------+---------------------------------------------+
48-
| ``AIRFLOW_PIP_VERSION`` | ``21.2.4`` | PIP version used. |
48+
| ``AIRFLOW_PIP_VERSION`` | ``21.3.1`` | PIP version used. |
4949
+------------------------------------------+------------------------------------------+---------------------------------------------+
5050
| ``PIP_PROGRESS_BAR`` | ``on`` | Progress bar for PIP installation |
5151
+------------------------------------------+------------------------------------------+---------------------------------------------+

‎scripts/ci/kubernetes/ci_run_kubernetes_tests.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ function create_virtualenv() {
8282

8383
. "${virtualenv_path}/bin/activate"
8484

85-
pip install --upgrade "pip==${AIRFLOW_PIP_VERSION}" "wheel==${WHEEL_VERSION}"
85+
pip install "pip==${AIRFLOW_PIP_VERSION}" "wheel==${WHEEL_VERSION}"
8686

8787
local constraints=(
8888
--constraint

‎scripts/ci/libraries/_initialization.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -421,7 +421,7 @@ function initialization::initialize_image_build_variables() {
421421

422422
export INSTALLED_EXTRAS="async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,imap,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv"
423423

424-
AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION:="21.2.4"}
424+
AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION:="21.3.1"}
425425
export AIRFLOW_PIP_VERSION
426426

427427
# We also pin version of wheel used to get consistent builds

‎scripts/docker/common.sh‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,11 @@
1717
# under the License.
1818
set -euo pipefail
1919

20-
test -v INSTALL_MYSQL_CLIENT
21-
test -v INSTALL_MSSQL_CLIENT
22-
test -v AIRFLOW_INSTALL_USER_FLAG
23-
test -v AIRFLOW_REPO
24-
test -v AIRFLOW_BRANCH
25-
test -v AIRFLOW_PIP_VERSION
26-
27-
set -x
20+
: "${INSTALL_MYSQL_CLIENT:?Should be true or false}"
21+
: "${INSTALL_MSSQL_CLIENT:?Should be true or false}"
22+
: "${AIRFLOW_REPO:?Should be set}"
23+
: "${AIRFLOW_BRANCH:?Should be set}"
24+
: "${AIRFLOW_PIP_VERSION:?Should be set}"
2825

2926
function common::get_airflow_version_specification() {
3027
if [[ -z ${AIRFLOW_VERSION_SPECIFICATION}
@@ -42,7 +39,6 @@ function common::override_pip_version_if_needed() {
4239
fi
4340
}
4441

45-
4642
function common::get_constraints_location() {
4743
# auto-detect Airflow-constraint reference and location
4844
if [[ -z "${AIRFLOW_CONSTRAINTS_REFERENCE}" ]]; then
@@ -60,3 +56,9 @@ function common::get_constraints_location() {
6056
AIRFLOW_CONSTRAINTS_LOCATION="${constraints_base}/${AIRFLOW_CONSTRAINTS}-${python_version}.txt"
6157
fi
6258
}
59+
60+
function common::show_pip_version_and_location() {
61+
echo "PATH=${PATH}"
62+
echo "pip on path: $(which pip)"
63+
echo "Using pip: $(pip --version)"
64+
}

‎scripts/docker/compile_www_assets.sh‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@
1717
# under the License.
1818
# shellcheck disable=SC2086
1919
set -euo pipefail
20-
set -x
2120

2221
# Installs additional dependencies passed as Argument to the Docker build command
2322
function compile_www_assets() {
@@ -35,8 +34,27 @@ function compile_www_assets() {
3534
www_dir="$(python -m site --user-site)/airflow/www"
3635
fi
3736
pushd ${www_dir} || exit 1
38-
yarn install --frozen-lockfile --no-cache
39-
yarn run prod
37+
set +e
38+
yarn install --frozen-lockfile --no-cache 2>/tmp/out-yarn-install.txt
39+
local res=$?
40+
if [[ ${res} != 0 ]]; then
41+
>&2 echo
42+
>&2 echo "Error when running yarn install:"
43+
>&2 echo
44+
>&2 cat /tmp/out-yarn-install.txt && rm -f /tmp/out-yarn-install.txt
45+
exit 1
46+
fi
47+
yarn run prod 2>/tmp/out-yarn-run.txt
48+
res=$?
49+
if [[ ${res} != 0 ]]; then
50+
>&2 echo
51+
>&2 echo "Error when running yarn install:"
52+
>&2 echo
53+
>&2 cat /tmp/out-yarn-run.txt && rm -f /tmp/out-yarn-run.txt
54+
exit 1
55+
fi
56+
rm -f /tmp/out-yarn-run.txt
57+
set -e
4058
find package.json yarn.lock static/css static/js -type f | sort | xargs md5sum > "${md5sum_file}"
4159
rm -rf "${www_dir}/node_modules"
4260
rm -vf "${www_dir}"/{package.json,yarn.lock,.eslintignore,.eslintrc,.stylelintignore,.stylelintrc,compile_assets.sh,webpack.config.js}

‎scripts/docker/install_additional_dependencies.sh‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,10 @@
1818
# shellcheck disable=SC2086
1919
set -euo pipefail
2020

21-
test -v UPGRADE_TO_NEWER_DEPENDENCIES
22-
test -v ADDITIONAL_PYTHON_DEPS
23-
test -v EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS
24-
test -v AIRFLOW_INSTALL_USER_FLAG
25-
test -v AIRFLOW_PIP_VERSION
21+
: "${UPGRADE_TO_NEWER_DEPENDENCIES:?Should be true or false}"
22+
: "${ADDITIONAL_PYTHON_DEPS:?Should be set}"
23+
: "${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS:?Should be set}"
24+
: "${AIRFLOW_PIP_VERSION:?Should be set}"
2625

2726
# shellcheck source=scripts/docker/common.sh
2827
. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
@@ -36,26 +35,26 @@ function install_additional_dependencies() {
3635
echo
3736
echo Installing additional dependencies while upgrading to newer dependencies
3837
echo
39-
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade --upgrade-strategy eager \
38+
pip install --upgrade --upgrade-strategy eager \
4039
${ADDITIONAL_PYTHON_DEPS} ${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
4140
# make sure correct PIP version is used
42-
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade "pip==${AIRFLOW_PIP_VERSION}"
41+
pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}"
4342
pip check
4443
else
4544
echo
4645
echo Installing additional dependencies upgrading only if needed
4746
echo
48-
pip install ${AIRFLOW_INSTALL_USER_FLAG} \
49-
--upgrade --upgrade-strategy only-if-needed \
47+
pip install --upgrade --upgrade-strategy only-if-needed \
5048
${ADDITIONAL_PYTHON_DEPS}
5149
# make sure correct PIP version is used
52-
pip install ${AIRFLOW_INSTALL_USER_FLAG} --upgrade "pip==${AIRFLOW_PIP_VERSION}"
50+
pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}"
5351
pip check
5452
fi
5553
}
5654

5755
common::get_airflow_version_specification
5856
common::override_pip_version_if_needed
5957
common::get_constraints_location
58+
common::show_pip_version_and_location
6059

6160
install_additional_dependencies

0 commit comments

Comments
 (0)