Repository navigation
Closed
cleanlab/cleanlab
#262Labels
Description
Describe the bug
A clear and concise description of the bug.
We've been having issues with our CI in GitHub Actions for the last few hours, and think it might be because of Coverage 6.3 - it's the one thing that's changed, and freezing it at 6.2 seems to allow runs to complete successfully.
To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:
- What version of Python are you using? 3.7 seems to be a bit buggy with this. We also run 3.8 - 3.10 in CI and no issues seen with that.
- What version of coverage.py shows the problem? The output of
coverage debug sysis helpful. 6.2.3 - What versions of what packages do you have installed? The output of
pip freezeis helpful. See https://gist.github.com/tunetheweb/4d288ea4467ba74a66b3a0e2e8d5e4ea - What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix. This is tricky. We run a lot of commands in CI, but checking out https://github.com/sqlfluff/sqlfluff/ and running
tox -e py37 -- -n 2 testshould reproduce it. Having problems setting up a 3.7 environment but will try to get a better test case. We do use a multithreaded process and noticed some changes to that. - What commands did you run?
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.
Will try to get a better repo.
Reacted by vEpiphyte, Jordan Heemskerk, alonsotagle-voxie, Guillaume Lemaitre, Olivier Grisel, Tom Young, Thomas Grainger, ni-todo-spot, Harmen Stoppels, Massimiliano Culpo and 14 more
Activity
Pin coverage at 6.2 for the moment coveragepy/coveragepy#1310
vEpiphyte commentedon Jan 25, 2022
We've experienced a similar issue with CI timeouts in CircleCI when using Python 3.8.
This occurs when using pytest as our test runner with coverage.
The hang occurs after tests pass and the coverage is reported to the console; it's almost like the python process fails to exit.
Pip freeze output
This happened to our master commit here
vertexproject/synapse@1b2d327
This was fixed in https://github.com/vertexproject/synapse/pull/2534/files#diff-c9c796259f3852b51b531b79cbf07820145088d4f108fb006ab44b15f0b15934R8
We did find the following line in a CI log for one of the failing tests which may or may not be related - it occurred prior to the pytest warnings summary and the console coverage output.
tunetheweb commentedon Jan 25, 2022
Interesting that you saw this on 3.8. We experienced some intermittent hanging in other Python versions but repeated hanging (every time) in 3.7, so wasn’t sure if the intermittent issues were related.
I also forgot to include my error message. It happened near the end of the test, but not quite at the end as you experienced (not sqlfluff is our project, but the last line might be useful):
nedbat commentedon Jan 26, 2022
I'm not clear what problems you are reporting. @tunetheweb the original issue description doesn't mention what the actual problem is. @vEpiphyte says they've seen similar problems, and mentions hanging. Then @tunetheweb shows a module-not-measured warning. This sounds like two different issues.
Anything you can do to give me a reproducible scenario would be great.
tunetheweb commentedon Jan 26, 2022
Yes I appreciate this is a very poor issue to raise. Sorry about that, but thought I'd give you a heads up as saw it right after the release, and it stopped happening right after I pinned to the old version.
Will try to give a more meaningful reproducible use case if I can narrow it down.
The issue is jobs hanging (I cancelled them after an hour, when they normally take 5 mins. One job I left running and it took 3 hours and counting). That was for Python 3.7.
I also saw a lot of slow down in jobs (5 min jobs were more often than not taking 20mins when they did complete). That was for Python 3.8+.
Initially I thought GitHub Actions were on the blink but at soon as I pinned the old version of coverage it all worked. Unpinning it again breaks it again. Nothing else changes between the runs.
But as to narrowing it down why, at the moment I don't have more info to help explain that. But will keep digging...
Think the error message is a red-herring. It was the last thing printed before it froze. But do see that at the end of the run elsewhere (though interestingly I only see it at the end of good runs, whereas that appeared after our CI job was only at 90% and then hung).
Sorry, again for such a poor report. Can close it if you want until I can get you more info, as appreciate it's difficult to do anything with what I've written here but as I say this was more a heads up, and a vague hope someone would have an idea what it might be 😞
Pin coverage==6.2 to fix tests
Pin coverage==6.2 to fix tests (#2634)
nedbat commentedon Jan 26, 2022
@tunetheweb @vEpiphyte @sir-sigurd Can you try installing this branch of coverage.py to see if it fixes the problem?
tunetheweb commentedon Jan 26, 2022
Doesn't look any better with that to be honest.
BTW happy to move this discussion to #1312 as that issue would appear to be a duplicate of this one, but with much better detail!
tunetheweb commentedon Jan 26, 2022
Yeah definitely seeing same issue with that unfortunately :-(
But it is very reproducible so, while I haven't been able to help with a minimal reproducible test case, I'm very happy to test any other test branches you have. Just takes about 10mins to run.
sir-sigurd commentedon Jan 26, 2022
Nope, it doesn't help.
Test a coverage fix proposed from coveragepy/coveragepy#1310 coverage…
vEpiphyte commentedon Jan 26, 2022
I did reproduce the error locally with a set of our unit tests. Recording this big blob for posterity - I haven't had the time to whittle it down yet but will do shortly.
This produces the same hang locally using Python 3.8.10 on linux (ubuntu 21.04) and coverage 6.3.0.
I then installed ned's patch with the following
And got the same hang. Sending sigint (ctrl+c) to the process yielded the following traceback
90 remaining items
asottile commentedon May 18, 2022
can confirm it works with my minimal reproduction above 🎉 (and again fails with
[run] sigterm = trueas expected!)tunetheweb commentedon May 18, 2022
Can confirm I can see no issues with my pipeline with this branch either.
Great to get this resolve so can unpin the version number on the next release. Thanks for your efforts!
Upgrade to coverage>=6.4
nedbat commentedon May 22, 2022
This is now released as part of coverage 6.4.
johnthagen commentedon May 23, 2022
@nedbat I can confirm that Coverage 6.4 solved our
multiprocessing.Pooldeadlocks in CI.Thanks! ❤️
Upgrade to latest version of coverage
pllim commentedon May 23, 2022
v6.4 works for me. Thanks!
Upgrade to latest version of coverage (#262)
Unpin coverage again
Unpin coverage again
py-coverage: update to 6.4.4.
py-coverage: update to 6.4.4.
py-coverage: update to 6.4.4.
Pin coverage to version 6.2 to fix CI hanging
py-coverage: update to 6.4.4.