Skip to content

TST: test pollution in scipy-only tests #17745

New issue
New issue
Closed
#17758
#17747
Closed
TST: test pollution in scipy-only tests#17745
#17758
#17747
Labels
Bugtesting

Description

@neutrinoceros
neutrinoceros
opened on Feb 8, 2025

As I mentioned in #17732 (review), I'm currently seeing 12 tests consistently fail when running double tests locally1

❯ pytest --color=no --keep-duplicates --pyargs astropy astropy > double_test.log 2>&1

...
=========================== short test summary info ============================
FAILED astropy/modeling/tests/test_input.py::test_call_keyword_args_1[model0]
FAILED astropy/modeling/tests/test_input.py::test_call_keyword_args_1[model1]
FAILED astropy/modeling/tests/test_input.py::test_call_keyword_args_1[model2]
FAILED astropy/modeling/tests/test_input.py::test_call_keyword_args_1[model3]
FAILED astropy/modeling/tests/test_input.py::test_call_keyword_args_1[model4]
FAILED astropy/modeling/tests/test_input.py::test_call_keyword_args_2[model0]
FAILED astropy/modeling/tests/test_input.py::test_call_keyword_args_2[model1]
FAILED astropy/modeling/tests/test_input.py::test_call_keyword_args_2[model2]
FAILED astropy/units/tests/test_quantity_ufuncs.py::TestUfuncHelpers::test_scipy_registered
FAILED astropy/units/tests/test_quantity_ufuncs.py::test_scipy_registration
FAILED astropy/visualization/wcsaxes/tests/test_wcsapi.py::test_coord_type_from_ctype
FAILED astropy/visualization/wcsaxes/tests/test_wcsapi.py::test_custom_coord_type_1d_2d_wcs_overwrite
==== 12 failed, 57587 passed, 752 skipped, 446 xfailed in 302.79s (0:05:02) ====

I checked that this reproduced even from a fresh clone + fresh env (resolved via uv pip install -e [test_all]), so I'm confident it's not a side effect of my own copy being dirty. For reference it fails the same way with numpy 1.26 or numpy 2.
I also tried running the same invocation with v6.1.x2 (branch) and saw no failures there.
However v7.0.0 (tag) shows3 the same 12 failures.

I intend to try out https://pypi.org/project/detect-test-pollution/ to narrow it down and will report back when I know more about this issue.

Footnotes

  1. this was already the case with the previous invocation,python -c 'import sys; from astropy import test; test(); sys.exit(test())', so TST: Do not use astropy.test() in double run #17732 is not the culprit. ↩

  2. specifically with Python 3.12 + numpy<2, otherwise some of units completeness test don't pass, which is basically noise in this context ↩

  3. specifically with Python 3.13 + numpy<2.2, for the same reason as 2. ↩

Activity

  1. neutrinoceros
    added
    Needs-clarification
    testing
    on Feb 8, 2025

  2. neutrinoceros commented on Feb 8, 2025

    @neutrinoceros
    neutrinoceros
    on Feb 8, 2025
    ContributorAuthor

    Phew, detect-test-pollution actually returned something in under 7min, which is much faster than I anticipated:

    ❯ detect-test-pollution --failing-test astropy/units/tests/test_quantity_ufuncs.py::test_scipy_registration --tests ./astropy
discovering all tests...
-> discovered 29905 tests!
ensuring test passes by itself...
-> OK!
ensuring test fails with test group...
-> OK!
running step 1:
- 29904 tests remaining (about 15 steps)
running step 2:
- 14952 tests remaining (about 14 steps)
running step 3:
- 7476 tests remaining (about 13 steps)
running step 4:
- 3738 tests remaining (about 12 steps)
running step 5:
- 1869 tests remaining (about 11 steps)
running step 6:
- 934 tests remaining (about 10 steps)
running step 7:
- 467 tests remaining (about 9 steps)
running step 8:
- 233 tests remaining (about 8 steps)
running step 9:
- 116 tests remaining (about 7 steps)
running step 10:
- 58 tests remaining (about 6 steps)
running step 11:
- 29 tests remaining (about 5 steps)
running step 12:
- 14 tests remaining (about 4 steps)
running step 13:
- 7 tests remaining (about 3 steps)
running step 14:
- 3 tests remaining (about 2 steps)
running step 15:
- 2 tests remaining (about 1 steps)
double checking we found it...
-> the polluting test is: astropy/units/tests/test_quantity_ufuncs.py::TestScipySpecialUfuncs::test_erf_scalar[erf]

    I'll see about that after lunch ! In the mean time, I'll keep my computer busy by running a git bisection between 6.1.x and 7.0.0 to collect more clues.

  3. neutrinoceros commented on Feb 8, 2025

    @neutrinoceros
    neutrinoceros
    on Feb 8, 2025
    ContributorAuthor

    Nevermind, I actually can reproduce on 6.1.x as well. According to the logs, my first attempt didn't include every optional deps like scipy or matplotlib so maybe that's why. Since each iteration takes several minutes, and I don't actually know of a "good" commit, let's give up on bisection for now.
    What's extremely puzzling still is that double tests has been running fine on CI all this time, though it may be a platform specific thing (I'm on macOS)

    Other results

    ❯ detect-test-pollution --failing-test "astropy/modeling/tests/test_input.py::test_call_keyword_args_1[model0]" --tests ./astropy
discovering all tests...
-> discovered 29905 tests!
ensuring test passes by itself...
-> OK!
ensuring test fails with test group...
-> expected failure -- but it passed?

    ❯ detect-test-pollution --failing-test astropy/visualization/wcsaxes/tests/test_wcsapi.py::test_coord_type_from_ctype --tests ./astropy
discovering all tests...
-> discovered 29905 tests!
ensuring test passes by itself...
-> OK!
ensuring test fails with test group...
-> OK!
running step 1:
- 29904 tests remaining (about 15 steps)

...

- 2 tests remaining (about 1 steps)
double checking we found it...
-> the polluting test is: astropy/visualization/wcsaxes/tests/test_wcsapi.py::test_custom_coord_type_1d_2d_wcs_overwrite
  4. neutrinoceros
    closed this as completedin #17747on Feb 10, 2025

  5. neutrinoceros commented on Feb 10, 2025

    @neutrinoceros
    neutrinoceros
    on Feb 10, 2025
    ContributorAuthor

    Oops, this was accidentally linked to #17747 where @mhvk said it does "not fix #17745"

  6. neutrinoceros
    reopened this on Feb 10, 2025

  7. neutrinoceros commented on Feb 10, 2025

    @neutrinoceros
    neutrinoceros
    on Feb 10, 2025
    ContributorAuthor

    regarding tests about scipy-registration:
    while it's pretty clear why these tests create pollution, why they fail on a second run (it is pretty evident in

    astropy/astropy/units/tests/test_quantity_ufuncs.py

    Line 1565 in 38faeb3

    def test_scipy_registration():
    ), as well as why we don't see these failures on CI (scipy is not included in double test jobs), what's not clear to me at the moment is how to address it... these tests are so intrinsically coupled to a global state that it would probably require mocking various parts of it until tests don't actually check the code much at all.
    ref #7636

    @mhvk, any ideas ?

  8. neutrinoceros
    added
    Bug
    and removed
    Needs-clarification
    on Feb 10, 2025

  9. neutrinoceros commented on Feb 10, 2025

    @neutrinoceros
    neutrinoceros
    on Feb 10, 2025
    ContributorAuthor

    Actually I have an idea for a minimal patch, stay tuned.

    update: this is #17750

  10. neutrinoceros commented on Feb 10, 2025

    @neutrinoceros
    neutrinoceros
    on Feb 10, 2025
    ContributorAuthor

    All reported problems should be addressed within

    None of the above will close the present issue mind you. Once they are all in, I'll open one last PR to add scipy to double tests jobs, to avoid regressions there.

  11. neutrinoceros
    changed the title [-]TST: test pollution seen locally in double test runs[/-] [+]TST: test pollution in scipy-only tests[/+] on Feb 10, 2025
  12. neutrinoceros
    closed this as completedin #17758on Nov 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Bugtesting

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions