Run tests against multiple airflow versions #370
Conversation
Codecov Report
@@ Coverage Diff @@
## main #370 +/- ##
==========================================
- Coverage 93.04% 90.44% -2.60%
==========================================
Files 57 57
Lines 1969 1968 -1
Branches 302 302
==========================================
- Hits 1832 1780 -52
- Misses 100 144 +44
- Partials 37 44 +7
Continue to review full report at Codecov.
|
.github/workflows/ci.yaml
Outdated
| - run: sqlite3 /tmp/sqlite_default.db "VACUUM;" | ||
| - run: pip3 install nox | ||
| - run: nox -s test -- --splits 12 --group ${{ matrix.group }} --cov=src --cov-report=xml --cov-branch | ||
| - run: nox -s unit_test -- --splits 6 --group ${{ matrix.group }} --cov=src --cov-report=xml --cov-branch |
There was a problem hiding this comment.
Did this change increase the time it takes for the CI to run tests? If yes, in how much?
|
|
||
|
|
||
| @nox.session(python=["3.7", "3.8", "3.9"]) | ||
| @nox.parametrize("airflow", ["2.2.5", "2.3"]) |
noxfile.py
Outdated
|
|
||
| @nox.session(python=["3.7", "3.8", "3.9"]) | ||
| def test(session: nox.Session) -> None: | ||
| def unit_test(session: nox.Session) -> None: |
There was a problem hiding this comment.
I like the split between unit and integration tests!
|
@kaxil, once the tests are passing, let's merge this! It is probably worth double-checking that coverage is working as expected after the tests are fixed. |
kaxil
force-pushed
the
run-tests-multiple-airflow-versions
branch
from
a7df2b7 to
ff88560
Compare
|
@kaxil, how is this going? Do we have any risks that Astro may stop working with previous versions of Airflow if this is not merged sooner than later? |
Uses https://docs.pytest.org/en/latest/how-to/usage.html#profiling-test-execution-duration to get a list of slowest 30 tests for which the durations is over 1.0s long. This is a small change but will help identify if a test takes too long. Found a need to do that while working on #367 and #370
No risk, until we release a new version and update any libraries in |
kaxil
force-pushed
the
run-tests-multiple-airflow-versions
branch
from
ff88560 to
3da49fd
Compare
Uses https://docs.pytest.org/en/latest/how-to/usage.html#profiling-test-execution-duration to get a list of slowest 30 tests for which the durations is over 1.0s long. This is a small change but will help identify if a test takes too long. Found a need to do that while working on #367 and #370
kaxil
changed the title
kaxil
force-pushed
the
run-tests-multiple-airflow-versions
branch
from
3da49fd to
83457be
Compare
Codecov Report
@@ Coverage Diff @@
## main #370 +/- ##
==========================================
+ Coverage 90.45% 90.50% +0.05%
==========================================
Files 53 53
Lines 1792 1791 -1
Branches 280 279 -1
==========================================
Hits 1621 1621
Misses 131 131
+ Partials 40 39 -1
Continue to review full report at Codecov.
|
kaxil
force-pushed
the
run-tests-multiple-airflow-versions
branch
2 times, most recently
from
72a3882 to
55bef50
Compare
This commit allows separately running unit tests and integration tests. Also allows running integration tests with Airflow 2.2.5 and 2.3
These are some tests but there are lot more Revert "Mark certain tests as integration tests" This reverts commit f98f185.
kaxil
force-pushed
the
run-tests-multiple-airflow-versions
branch
from
55bef50 to
c68e9b2
Compare
|
We want to make sure that Astro SDK continues working with Airflow 2.2.5 and Airflow 2.3 - so this commit adds this capability in nox. A problem I foresee is some tests failing intermittently as some of our "integration tests" don't have unique ids for tables, dataset which means there might be times when an integration test is running in 2.2.5 which deletes a resource that is being used by same test running for 2.3. One "quick" solution is to run them sequentially but that will increase total test time. A better solution will be to fix the tests but that will take time. |
The CI is failing intermittently as I initially suspected in #370 Example: https://github.com/astronomer/astro-sdk/runs/6601723455?check_suite_focus=true This PR runs 2.2.5 tests after 2.3 tests are complete and also only runs a subset of the tests (Example DAGs and one of the integration tests) for 2.2.5 which should be enough for 2.2.5
The CI is failing intermittently as I initially suspected in #370 Example: https://github.com/astronomer/astro-sdk/runs/6601723455?check_suite_focus=true This PR runs 2.2.5 tests after 2.3 tests are complete and also only runs a subset of the tests (Example DAGs and one of the integration tests) for 2.2.5 which should be enough for 2.2.5
The CI is failing intermittently as I initially suspected in #370 Example: https://github.com/astronomer/astro-sdk/runs/6601723455?check_suite_focus=true This PR runs 2.2.5 tests after 2.3 tests are complete and also only runs a subset of the tests (Example DAGs and one of the integration tests) for 2.2.5 which should be enough for 2.2.5
The CI is failing intermittently as I initially suspected in #370 Example: https://github.com/astronomer/astro-sdk/runs/6601723455?check_suite_focus=true This PR runs 2.2.5 tests after 2.3 tests are complete and also only runs a subset of the tests (Example DAGs and one of the integration tests) for 2.2.5 which should be enough for 2.2.5
We want to make sure that Astro SDK continues working with Airflow 2.2.5 and Airflow 2.3 - so this commit adds this capability in nox.
A problem I foresee is some tests failing intermittently as some of our "integration tests" don't have unique ids for tables, dataset which means there might be times when an integration test is running in 2.2.5 which deletes a resource that is being used by same test running for 2.3.
One "quick" solution is to run them sequentially but that will increase total test time. A better solution will be to fix the tests but that will take time.