Skip to content

Some tests run much slower with coverage testing on Python 3 compared to Python 2 #4826

Closed

Description

@astrofrog

If I run just the astropy.units tests with:

python setup.py test -P units --coverage

On Python 2 it runs in:

============= 2739 passed, 15 skipped, 3 xfailed in 115.31 seconds =============

while on Python 3 it runs in:

============= 1658 passed, 3 xfailed in 335.95 seconds =============

Note that Python 3 runs fewer tests (because it doesn't do the tests with unicode_literals) and even then it's 3 times slower. I'm not sure if the bottleneck is in our functionality, or the coverage testing itself.

Activity

  1. astrofrog commented on May 2, 2016

    @astrofrog
    MemberAuthor

    Once this is solved, we can enable coverage testing in .travis.yml for Python 3.x

  2. pllim commented on May 2, 2016

    @pllim
    Member

    Just to re-iterate something I said in #4822 -- Is it slow in all Python 3.x or just 3.4?

  3. pllim commented on May 2, 2016

    @pllim
    Member

    @astrofrog said slow in both 3.4 and 3.5.

  4. astrofrog commented on May 2, 2016

    @astrofrog
    MemberAuthor

    Yep - the above timings were done on Python 3.5

  5. astrofrog commented on May 2, 2016

    @astrofrog
    MemberAuthor

    I think the next thing to look at would be to try and output the times for all the tests and see if there's a specific kind of test that is slow.

  6. astrofrog commented on May 2, 2016

    @astrofrog
    MemberAuthor

    On my laptop, the tests take about 15 minutes to run in Python 3.5 with coverage enabled:

    =============== 1 failed, 7589 passed, 92 skipped, 37 xfailed in 931.81 seconds ===============
    
  7. pllim commented on May 2, 2016

    @pllim
    Member

    Just want to be clear -- So, when coverage is not enabled, the testing times are comparable between Python 2 and 3?

  8. astrofrog commented on May 2, 2016

    @astrofrog
    MemberAuthor

    With no coverage:

    Python 2

    10101 passed, 426 skipped, 45 xfailed in 281.13 seconds
    

    Python 3

    1 failed, 7589 passed, 92 skipped, 37 xfailed in 206.43 seconds
    

    With coverage:

    Python 2

    10101 passed, 426 skipped, 45 xfailed in 374.28 seconds
    

    Python 3

    1 failed, 7589 passed, 92 skipped, 37 xfailed in 931.81 seconds
    
  9. pllim commented on May 2, 2016

    @pllim
    Member

    I wonder if this is a pytest problem rather than an Astropy problem. If only there is a convenient way to also run coverage in both Python 2 and 3 for something that is not related to Astropy and see if it has the same timing difference. Hmm...

  10. astrofrog commented on May 2, 2016

    @astrofrog
    MemberAuthor

    @pllim - yes, I think it would good to do that test. I might try with glue.

  11. astrofrog commented on May 2, 2016

    @astrofrog
    MemberAuthor

    I don't think it's a pytest issue. Results for glue:

    Without coverage:

    Python 2

    2 failed, 1666 passed, 43 skipped, 1 xfailed in 82.43 seconds
    

    Python 3

    1707 passed, 4 skipped, 1 xfailed in 104.01 seconds
    

    With coverage:

    Python 2

    2 failed, 1666 passed, 43 skipped, 1 xfailed in 98.55 seconds
    

    Python 3

    1707 passed, 4 skipped, 1 xfailed in 98.75 seconds
    
  12. pllim commented on May 2, 2016

    @pllim
    Member

    😱 @astrofrog , thanks for confirming!

  13. 40 remaining items

  14. pllim commented on Sep 6, 2016

    @pllim
    Member

    @mhvk , I don't either, since @MSeifert04 claimed that his PR is not supposed to fix this particular problem... Maybe Travis CI timing will confirm for sure.

  15. removed
    Close?Tell stale bot that this issue/PR is stale
    on Sep 6, 2016
  16. mhvk commented on Sep 6, 2016

    @mhvk
    Contributor

    Let me move the discussion back here: @astrofrog - do you get much longer times on your own computer? I ask since both @pllim and I seemed to get short ones, at least with the simple python3 setup.py test --coverage -P units line you had at the very top of this file (while travis obviously still is very slow).

  17. pllim commented on Sep 7, 2016

    @pllim
    Member

    I am running a series of coverage tests, once per module, and then altogether, and recording the run times on both Python versions. Hopefully it will be finished before Thanksgiving.

  18. mhvk commented on Sep 7, 2016

    @mhvk
    Contributor

    If you're thinking of Canadian thanksgiving, we only have to wait about a month...

  19. MSeifert04 commented on Sep 7, 2016

    @MSeifert04
    Contributor

    @pllim and @mhvk What versions of coverage are you using locally? As I noted in #5308 (comment) it may just be something that was fixed in a newer version of coverage.

  20. pllim commented on Sep 7, 2016

    @pllim
    Member

    The latest I could get via conda update on both Python versions. I'll check when I'm back at work tomorrow.

  21. mhvk commented on Sep 8, 2016

    @mhvk
    Contributor

    I'm using what Debian testing provides, which is 4.2 (i.e., the newest according to the comment you linked to). Since we seem to install apt, it may be possible to get it via that? It would seem a good possibility!

  22. pllim commented on Sep 8, 2016

    @pllim
    Member

    See potential fix in astropy/ci-helpers#132

  23. pllim commented on Sep 8, 2016

    @pllim
    Member

    Attached is my report from running coverage tests on each of the sub-package, and then the whole thing in the end. My conclusion is that the slowness in Python 3 is no longer an issue (at least with coverage 4.2). I agree with @MSeifert04 and @mhvk that the culprit might be the older coverage version, not Astropy itself. The one failure you see in the report is reported separately in #5316.

    zcov_result.txt

  24. MSeifert04 commented on Sep 8, 2016

    @MSeifert04
    Contributor

    Would be nice if it's indeed the coverage version.

  25. bsipocz commented on Oct 10, 2016

    @bsipocz
    Member

    Travis runs 2 mins faster for me for astropy when testing for @pllim's PR in ci-helpers, so hopefully some of that gain is real. https://travis-ci.org/bsipocz/astropy/builds/166497095

  26. mhvk commented on Oct 12, 2016

    @mhvk
    Contributor

    I think this is mostly solved, so closing!

  27. holvianssi commented on Jul 28, 2022

    @holvianssi

    As this happens to be first hit on Google, and I just spent a lot of time chasing why coverage is slow, I'm going to leave a hint here for the future:

    Check you have C extensions installed in your coverage package!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions