Skip to content

CI failure: "RuntimeError: tk.h version (8.6) doesn't match libtk.a version (8.5)" #1132

Closed

Description

@kousu

Issue description

Current behavior

CI is failing on {macOS} x {Python 3.7, Python 3.8}. Python3.6 and 3.9 are unaffected.

Steps to reproduce

Run CI on almost any branch currently and it will fail. Example failures:

Typical pattern of failures:

Screenshot 2022-05-17 at 18-15-15 Update publish_package yml · ivadomed_ivadomed@c4c425b

Linked Issues

Activity

  1. added
    CIcategory: TravisCI, GitHub Actions, etc.
    on May 17, 2022
  2. kousu commented on May 17, 2022

    @kousu
    ContributorAuthor

    The failures are all throwing the same exception:

    ../../../hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/matplotlib/backends/_backend_tk.py:949: in new_figure_manager_given_figure
        window = tk.Tk(className="matplotlib")
    ../../../hostedtoolcache/Python/3.7.13/x64/lib/python3.7/tkinter/__init__.py:2025: in __init__
        self._loadtk()
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <tkinter.Tk object .>
    
        def _loadtk(self):
            self._tkloaded = 1
            global _default_root
            # Version sanity checks
            tk_version = self.tk.getvar('tk_version')
            if tk_version != _tkinter.TK_VERSION:
                raise RuntimeError("tk.h version (%s) doesn't match libtk.a version (%s)"
    >                              % (_tkinter.TK_VERSION, tk_version))
    E           RuntimeError: tk.h version (8.6) doesn't match libtk.a version (8.5)
    

    I looked around for help and found an eclectic mixture of reports of this exact error, some on macOS, others in pyenv (which we don't use):

    as well as this reported fix in brew:

    Fix RuntimeError: tk.h version (8.6) doesn't match libtk.a version (8.5)

    The problem was that CPPFLAGS and LDFLAGS for brewed Tk were specified to environment variable but Python (seems) prefer the CPPFLAGS in arguments specified to './configure', mean "-I#{MacOS.sdk_path}/usr/include" version is used.

    The confusing thing is that that fix went in in 2015, so why is it showing up again now? Also are the Github runners even using brew's python? Do they have their own python build that accidentally recreated the bug? And why only 3.7 and 3.8?

  3. kousu commented on May 17, 2022

    @kousu
    ContributorAuthor

    Chonologically, the first failure was 13 days ago:

    Then

    This smells like a bug in GitHub's hands, especially since it progressed from consistently only being 3.7 to consistently being 3.7 and 3.8. I think they've deployed broken python environments somehow.

  4. kousu commented on May 17, 2022

    @kousu
    ContributorAuthor

    Interesting, I didn't know this but though tkinter is a part of cpython, but it's packaged separately by Ubuntu:

    $ apt-cache search python3-tk
    python3-tk - Tkinter - Writing Tk applications with Python 3.x
    

    Maybe macOS is in a similar situation? And that's prone to breakage like this?

  5. kousu commented on May 17, 2022

    @kousu
    ContributorAuthor

    A tip from some of those threads: if you do have Tk installed, then this will load and test it:

    python -m tkinter -c 'tkinter._test()'
    

    Screenshot_20220517_185557

    If it is GitHub's fault, we should be able to see that by making a fresh CI script that has nothing to do with ivadomed...

    I've made a separate repo and stripped down ivadomed's test script to just this and it's running now: https://github.com/ivadomed/test-tk-ci/actions/runs/2341876975

    Second attempt: call python -c 'import tkinter; window=tkinter.Tk(className="matplotlib")' which is closer to what actually triggers the bug here, and seems to sidestep the need to a $DISPLAY (so it can work headless): here -> https://github.com/ivadomed/test-tk-ci/actions/runs/2341904576

    The failure pattern is different: here all the the Windows/Linux tests passed, but there they all failed; however, the macOS tests show the same pattern

    Screenshot 2022-05-17 at 19-14-14 Update tests yml · ivadomed_test-tk-ci@d867658

    So I'm more convinced this is GitHub's fault.

    Here's my theory: for some reason matplotlib on macOS is defaulting to Tk, whereas on other OSes it has other choices.

    Round 3, just with macOS, that I can report upstream: here -> https://github.com/ivadomed/test-tk-ci/actions/runs/2341951055

    Screenshot 2022-05-17 at 19-24-23 Update tests yml · ivadomed_test-tk-ci@a48ac2a

    Interestingly, one outlier here: the python3.10 x macOS-10.15 one gave the same error the Linuxes were giving: https://github.com/ivadomed/test-tk-ci/runs/6480196513?check_suite_focus=true

    _tkinter.TclError: no display name and no $DISPLAY environment variable

    but otherwise the pattern is consistent.

    I've reported upstream at https://github.community/t/macos-runners-hostedtoolcaches-python-3-7-3-8-tkinter-is-broken/251343 ((but those are the community forums; hopefully their staff can direct me to the actual repo that builds the actual macOS runners to report the bug there)).

    EDIT: it was reported last week at actions/setup-python#402 (and actions/runner-images#5511).

  6. added a commit that references this issue on May 17, 2022
    b148d36
  7. kousu commented on May 17, 2022

    @kousu
    ContributorAuthor

    I think on our end we should figure out why matplotlib is defaulting to Tk. It should be in a headless environment, so it shouldn't be trying to do any GUI things at all. At most, it should be using Agg and rendering to .png.

    I think I understand now the failure pattern: matplotlib explains

    The first usable backend in the following list is selected: MacOSX, QtAgg, GTK4Agg, Gtk3Agg, TkAgg, WxAgg, Agg. The last, Agg, is a non-interactive backend that can only write to files. It is used on Linux, if Matplotlib cannot connect to either an X display or a Wayland display.

    which means it's doing something like

    for backend in ['MacOS', 'QtAgg', ..., 'TkAgg', ..., 'Agg']:
      try:
        import(backend)
        break
      except InitError:
        pass
    else:
      raise 'No Usable Backend'
    

    i.e. it's trying each backend in turn until one works. But for some reason, trying to init TkAgg is giving it an error it's not setup to catch -- this low-level library mismatch problem -- and escaping the loop.

    If so, we should be able to sidestep the problem by going direct to Agg: #1133

  8. kousu commented on May 18, 2022

    @kousu
    ContributorAuthor

    Here's a different fix than #1133: just downgrade python 3.7 itself (from '3.7' to '3.7.9'):

    carlosperate/ubittool@8583620

    If #1133 doesn't work we can apply that, just until actions/setup-python#402 is fixed.

  9. added a commit that references this issue on May 18, 2022
    86815bc
  10. added a commit that references this issue on May 19, 2022
    1da019b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CIcategory: TravisCI, GitHub Actions, etc.priority:high

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions