[AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3] #4938

potiuk · 2019-03-18T22:23:35Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira

Description

Here are some details about my PR, including screenshots of any UI changes:

This is 3rd stage of parent https://issues.apache.org/jira/browse/AIRFLOW-3718 task. It implements executing tests in CI environment.

Note that if you are reviewer, until the first part is merged, you should only take a look at the last commit rather than at all commits - currently there is no way to make a PR to a commit that is being reviewed yet.

Tests

This change introduces running all Airflow tests using Multi-staging image created in previous steps.

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

No documentation needed for that change. Design is documented in https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-10+Multi-layered+and+multi-stage+official+Airflow+image

Follow up documentation will be added in Simplified Development Workflow change: #4932

Code Quality

Passes flake8

potiuk · 2019-03-19T06:56:03Z

Note removed.

.travis.yml

run_tests.sh

scripts/ci/docker-compose-interactive.yml

scripts/ci/docker-compose.yml

setup.py

potiuk · 2019-03-25T15:24:07Z

@ashb -> thanks for the thorough review. I will apply all the comments shortly

potiuk · 2019-03-29T14:24:57Z

@Fokko @ashb -> I pushed the latest version with pretty much all comments applied (except setup.py).

There is one change that might require longer discussion/argumentation.

I had to change the ci image to use root rather than "airflow" user - so the Dockerfile is now configurable (main airflow image uses "airflow" user and "CI" image user root user.

This was needed in order to replicate CI environment on Linux. I tested it now extensively on Linux and the main problem was that when you tried to map local source folder to the container, you could not easily map it as "airflow" user and the folders are not accesible to airflow user.

This is a known Docker problem (there is an open issue for that moby/moby#2259 ). I looked at possible workaround, but it turns out, until this issue is solved, it's easiest if airflow is run as root (then everything is root-owned inside the container).

This problem does not occur on Mac because on Mac local volumes are mounted to virtual machine using user-space synchronising filesystem via virtual machine (which makes it much slower on one hand, but it allows to carry the user mapping with the bind volume). But if someone develops on Linux that could be a huge blocker for convenient development environment.

I hope this is something that you will be able to live with :)

potiuk

@Fokko @ashb

Please take a look and see if you have any further "big" comments to it

Further progress: This commit is quite close to have the tox-less Travis CI tests. I am very close to have all the tests pass:

mysql is working (it was a permission problem - I solved it in one of the previous commits where I am fixing now only group write permissions for Docker context)
There are only three tests + minikube that are currently failing (there are few fixes I had to do beacause of change to the root user for CI tests):

tests.transplant_class..C.test_doctests
tests.www.api.experimental.test_kerberos_endpoints.ApiKerberosTests.test_trigger_dag
tests.test_models.DagTest.test_following_previous_schedule_daily_dag_CEST_to_CET
minikube tests

I simplified docker-compose-backend settings (now only variables specific for backend are specified there - docker compose launches all backends now. This was necessary because some of the Mysql tests use running Postgres (Postgres Hoook etc.).
I have added a nice summary of only failed tests at the end of the test run - this way we can see the exact list of failed tests immediately rather than searching through entire log

You can see it here for example: https://api.travis-ci.org/v3/job/513873530/log.txt

I have one outstanding question about python setup.py test behaviour (separate comment thread).

.travis.yml

Fokko · 2019-04-02T09:01:54Z

Thanks @potiuk

I'm a bit busy, but why do we need MySQL in the container itself? We can use the official MySQL container using docker-compose, or am I missing something?
Love the summary of the failing tests

ashb · 2019-04-03T09:28:10Z

This was necessary because some of the Mysql tests use running Postgres (Postgres Hoook etc.).

I thought that we skipped the operators for foreign operators:

airflow/tests/operators/test_operators.py

Lines 56 to 58 in e72ff0a

    
               @unittest.skipUnless('mysql' in configuration.conf.get('core', 'sql_alchemy_conn'), 
        
                                    "This is a MySQL test") 
        
               def test_mysql_operator_test(self):

Maybe we don't do that universally.

potiuk · 2019-04-03T10:28:51Z

@Fokko -> i understand being busy. I am quite busy myself now so I move it in a slower pace than I would like to (the remainig tests and kubernetes parts).

Answering the mysql part -> we are only installing client libraries for mysql not the server ones. Our python mysql library(https://pypi.org/project/mysqlclient/) requires mysql-client and libmysqlclient-dev native libraries to be installed. Since we are using debian-stretch as a base (this is what official python images use as a base), default mysql libraries are actually mariadb ones not the "oracle mysql" ones. They are supposed to be compatible (mariadb is fork of mysql).

I figured that Airflow's target is the original "MySQL" rather than "MariaDB" so I think using "official" libs from Oracle might be better choice for CI build - that's why we have separate step to install Oracle libs (following official instructions). It's a bit long because you need to install Oracle Keys and set-up repos, but it boils down to:

apt-get install --no-install-recommends -y mysql-client libmysqlclient-dev

Note that Ubuntu has Oracle MySQL installed by default rather than MariaDB, so I guess most of airflow users will also have the MySQL client rather than MariaDB.

feluelle · 2019-07-18T07:46:23Z

WOW great job @potiuk !

turbaszek · 2019-07-18T08:12:50Z

Awesome! 🚀

potiuk · 2019-07-18T08:55:30Z

281 and counting ... Not that I am saying anything ;)

feluelle · 2019-07-18T10:38:29Z

^ This one doesn't count 🙄😄

mik-laj · 2019-07-18T11:05:17Z

This is the first full-merged AIP. One click for a man, but a big step for Airflow :-D 🍩 🍮 🍦 🍨 🍧 🍰 🍪 🍫 🍬 🍭

(cherry picked from commit 2d086d7)

…apache#4938)" This reverts commit 2d086d7.

…#4938) (cherry picked from commit 2d086d7)

…#4938)

…#4938) (cherry picked from commit 2d086d7)

potiuk changed the title ~~Multi Stage Image - Travis CI tests~~ [AIRFLOW-4117] Multi Stage Image - Travis CI tests Mar 18, 2019

potiuk mentioned this pull request Mar 18, 2019

[AIRFLOW-3718] [WIP/SPLIT] Multi-layered version of the docker image #4543

Closed

potiuk changed the title ~~[AIRFLOW-4117] Multi Stage Image - Travis CI tests~~ [AIRFLOW-4117] Multi Stage Image - Travis CI tests [Step 3/3] Mar 18, 2019

potiuk changed the title ~~[AIRFLOW-4117] Multi Stage Image - Travis CI tests [Step 3/3]~~ [AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3] Mar 18, 2019

potiuk force-pushed the ms-travis-ci-tests branch 2 times, most recently from 3da8bd9 to 0e9f1b7 Compare March 19, 2019 01:03

potiuk mentioned this pull request Mar 19, 2019

[AIRFLOW-3611] Breeze #4932

Merged

potiuk force-pushed the ms-travis-ci-tests branch 2 times, most recently from dd3f349 to 07cd943 Compare March 24, 2019 20:34

ashb reviewed Mar 25, 2019

View reviewed changes

potiuk force-pushed the ms-travis-ci-tests branch 2 times, most recently from 4b76f3b to 63b4a19 Compare March 29, 2019 10:28

potiuk closed this Mar 29, 2019

potiuk reopened this Mar 29, 2019

potiuk force-pushed the ms-travis-ci-tests branch from 63b4a19 to 9a94899 Compare March 29, 2019 15:23

potiuk mentioned this pull request Mar 29, 2019

[AIRFLOW-XXX] 1-setup-env.sh should only run in docker #5003

Merged

6 tasks

potiuk force-pushed the ms-travis-ci-tests branch from 9a94899 to 168c02f Compare March 29, 2019 16:30

potiuk mentioned this pull request Mar 30, 2019

[AIRFLOW-4061] Remove incubator in CI process #4896

Closed

2 tasks

potiuk force-pushed the ms-travis-ci-tests branch 4 times, most recently from 9aca2c1 to 8833ad7 Compare April 1, 2019 06:06

potiuk commented Apr 1, 2019

View reviewed changes

Fokko reviewed Apr 2, 2019

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

potiuk added a commit that referenced this pull request Jul 19, 2019

[AIRFLOW-4117] Travis CI uses multi-stage images to run tests (#4938)

778ecd4

potiuk added a commit that referenced this pull request Jul 20, 2019

[AIRFLOW-4117] Travis CI uses multi-stage images to run tests (#4938)

d840f9e

(cherry picked from commit 2d086d7)

potiuk added a commit to potiuk/airflow that referenced this pull request Jul 23, 2019

Revert "[AIRFLOW-4117] Travis CI uses multi-stage images to run tests (…

0855b7b

…apache#4938)" This reverts commit 2d086d7.

ashb pushed a commit to ashb/airflow that referenced this pull request Jul 23, 2019

[AIRFLOW-4117] Travis CI uses multi-stage images to run tests (apache…

0893cdc

…#4938) (cherry picked from commit 2d086d7)

wmorris75 pushed a commit to modmed-external/incubator-airflow that referenced this pull request Jul 29, 2019

[AIRFLOW-4117] Travis CI uses multi-stage images to run tests (apache…

2a0ef7b

…#4938)

dharamsk pushed a commit to postmates/airflow that referenced this pull request Aug 8, 2019

[AIRFLOW-4117] Travis CI uses multi-stage images to run tests (apache…

b7ae68d

…#4938) (cherry picked from commit 2d086d7)

mik-laj mentioned this pull request Aug 15, 2019

[AIRFLOW-XXX] Remove "Setup Test Environment using MySQL" article #5833

Merged

6 tasks

ashb mentioned this pull request Oct 9, 2020

Fully support running more than one scheduler concurrently #10956

Merged

8 tasks

[AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3] #4938

[AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3] #4938

Uh oh!

Conversation

potiuk commented Mar 18, 2019 • edited by dimberman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira

Description

Tests

Commits

Documentation

Code Quality

Uh oh!

potiuk commented Mar 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

potiuk commented Mar 25, 2019

Uh oh!

potiuk commented Mar 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fokko commented Apr 2, 2019

Uh oh!

ashb commented Apr 3, 2019

Uh oh!

potiuk commented Apr 3, 2019

Uh oh!

feluelle commented Jul 18, 2019

Uh oh!

turbaszek commented Jul 18, 2019

Uh oh!

potiuk commented Jul 18, 2019

Uh oh!

feluelle commented Jul 18, 2019

Uh oh!

mik-laj commented Jul 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

potiuk commented Mar 18, 2019 •

edited by dimberman

Loading

potiuk commented Mar 19, 2019 •

edited

Loading

potiuk commented Mar 29, 2019 •

edited

Loading

mik-laj commented Jul 18, 2019 •

edited

Loading