You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CI failure: "RuntimeError: tk.h version (8.6) doesn't match libtk.a version (8.5)" #1132
Fix RuntimeError: tk.h version (8.6) doesn't match libtk.a version (8.5)
The problem was that CPPFLAGS and LDFLAGS for brewed Tk were specified to environment variable but Python (seems) prefer the CPPFLAGS in arguments specified to './configure', mean "-I#{MacOS.sdk_path}/usr/include" version is used.
The confusing thing is that that fix went in in 2015, so why is it showing up again now? Also are the Github runners even using brew's python? Do they have their own python build that accidentally recreated the bug? And why only 3.7 and 3.8?
This smells like a bug in GitHub's hands, especially since it progressed from consistently only being 3.7 to consistently being 3.7 and 3.8. I think they've deployed broken python environments somehow.
The failure pattern is different: here all the the Windows/Linux tests passed, but there they all failed; however, the macOS tests show the same pattern
So I'm more convinced this is GitHub's fault.
Here's my theory: for some reason matplotlib on macOS is defaulting to Tk, whereas on other OSes it has other choices.
I think on our end we should figure out why matplotlib is defaulting to Tk. It should be in a headless environment, so it shouldn't be trying to do any GUI things at all. At most, it should be using Agg and rendering to .png.
The first usable backend in the following list is selected: MacOSX, QtAgg, GTK4Agg, Gtk3Agg, TkAgg, WxAgg, Agg. The last, Agg, is a non-interactive backend that can only write to files. It is used on Linux, if Matplotlib cannot connect to either an X display or a Wayland display.
which means it's doing something like
for backend in ['MacOS', 'QtAgg', ..., 'TkAgg', ..., 'Agg']:
try:
import(backend)
break
except InitError:
pass
else:
raise 'No Usable Backend'
i.e. it's trying each backend in turn until one works. But for some reason, trying to init TkAgg is giving it an error it's not setup to catch -- this low-level library mismatch problem -- and escaping the loop.
If so, we should be able to sidestep the problem by going direct to Agg: #1133
Activity
kousu commentedon May 17, 2022
The failures are all throwing the same exception:
I looked around for help and found an eclectic mixture of reports of this exact error, some on macOS, others in pyenv (which we don't use):
as well as this reported fix in
brew:The confusing thing is that that fix went in in 2015, so why is it showing up again now? Also are the Github runners even using brew's python? Do they have their own python build that accidentally recreated the bug? And why only 3.7 and 3.8?
kousu commentedon May 17, 2022
Chonologically, the first failure was 13 days ago:
Then
master: https://github.com/ivadomed/ivadomed/actions/runs/2340863632 - today - 3.7 & 3.8This smells like a bug in GitHub's hands, especially since it progressed from consistently only being 3.7 to consistently being 3.7 and 3.8. I think they've deployed broken python environments somehow.
kousu commentedon May 17, 2022
Interesting, I didn't know this but though tkinter is a part of cpython, but it's packaged separately by Ubuntu:
Maybe macOS is in a similar situation? And that's prone to breakage like this?
kousu commentedon May 17, 2022
A tip from some of those threads: if you do have Tk installed, then this will load and test it:
If it is GitHub's fault, we should be able to see that by making a fresh CI script that has nothing to do with ivadomed...
I've made a separate repo and stripped down ivadomed's test script to just this and it's running now: https://github.com/ivadomed/test-tk-ci/actions/runs/2341876975
Second attempt: call
python -c 'import tkinter; window=tkinter.Tk(className="matplotlib")'which is closer to what actually triggers the bug here, and seems to sidestep the need to a$DISPLAY(so it can work headless): here -> https://github.com/ivadomed/test-tk-ci/actions/runs/2341904576The failure pattern is different: here all the the Windows/Linux tests passed, but there they all failed; however, the macOS tests show the same pattern
So I'm more convinced this is GitHub's fault.
Here's my theory: for some reason matplotlib on macOS is defaulting to Tk, whereas on other OSes it has other choices.
Round 3, just with macOS, that I can report upstream: here -> https://github.com/ivadomed/test-tk-ci/actions/runs/2341951055
Interestingly, one outlier here: the python3.10 x macOS-10.15 one gave the same error the Linuxes were giving: https://github.com/ivadomed/test-tk-ci/runs/6480196513?check_suite_focus=true
but otherwise the pattern is consistent.
I've reported upstream at https://github.community/t/macos-runners-hostedtoolcaches-python-3-7-3-8-tkinter-is-broken/251343 ((but those are the community forums; hopefully their staff can direct me to the actual repo that builds the actual macOS runners to report the bug there)).
EDIT: it was reported last week at actions/setup-python#402 (and actions/runner-images#5511).
Update run_tests.yml
kousu commentedon May 17, 2022
I think on our end we should figure out why matplotlib is defaulting to Tk. It should be in a headless environment, so it shouldn't be trying to do any GUI things at all. At most, it should be using
Aggand rendering to .png.I think I understand now the failure pattern: matplotlib explains
which means it's doing something like
i.e. it's trying each backend in turn until one works. But for some reason, trying to init TkAgg is giving it an error it's not setup to catch -- this low-level library mismatch problem -- and escaping the loop.
If so, we should be able to sidestep the problem by going direct to Agg: #1133
kousu commentedon May 18, 2022
Here's a different fix than #1133: just downgrade python 3.7 itself (from '3.7' to '3.7.9'):
carlosperate/ubittool@8583620
If #1133 doesn't work we can apply that, just until actions/setup-python#402 is fixed.
Update run_tests.yml
Use headless ("Agg") matplotlib backend in CI (#1133)