Fix MacOs 14 Sonoma dlopen cache lookup #2549

takacsmark · 2023-11-15T19:00:13Z

Applied a fix to f8b237e to the find_library patch based on the original Stackoverflow hint https://stackoverflow.com/questions/63475461/unable-to-import-opengl-gl-in-python-on-macos.

Closes #2547

Updated the find_library patch for backward compatibility Closes vispy#2547

djhoese · 2023-11-15T19:30:27Z

@brisvag what napari folks are your typical mac testers? Anyone want to sanity check this PR? Otherwise the fact that CI is happy (which is running on an intel mac runner iirc) seems to be a pretty good test to me.

brisvag · 2023-11-16T12:51:23Z

@psobolewskiPhD is our resident mac bug hunter :)

psobolewskiPhD · 2023-11-16T15:00:05Z

I can test this on Ventura (13.6) to ensure no regression. I'll try to find time to update my personal laptop to Sonoma, but probably not till next weekend at the soonest.

ataffanel · 2024-03-19T14:55:18Z

I am maintaining a program that broke on Mac sonoma because of this bug and I can confirm that this PR fixes the problem on my test Mac M1 running Sonoma.

Is there anything more that needs to be checked before this can be merged?

djhoese · 2024-03-19T15:19:03Z

@ataffanel Do you have pyopengl installed in your environment? I'm wondering if this PR isn't needed if pyopengl is installed.

psobolewskiPhD · 2024-03-20T02:12:50Z

IMO pyopengl is not a vispy requirement, so this should get fixed.
I think I'm ok with this solution, but worried it may break things on older than macOS 11--only when pyopengl is not installed--because of changing the path?
Also I'm still very puzzled by the lowercase quartz as noted #2547 (comment)

I don't get why I don't see issues locally with pyopengl despite CDLL reporting None:

>>> from vispy.ext.cocoapy import quartz
>>> quartz
<CDLL 'None', handle fffffffffffffffe at 0x1053f3050>

By using a Q it loads properly:

In [1]: from vispy.ext.cocoapy import quartz

In [2]: quartz
Out[2]: <CDLL '/System/Library/Frameworks/Quartz.framework/Quartz', handle 3ad9bf508 at 0x108251d30>

I suspect this PR and the patch might just be masking an issue the with cdll loading of quartz, but my use case (napari mostly) just doesn't touch quartz I guess?

Looking more closely at the monkey patch:

vispy/vispy/ext/cocoapy.py

Lines 23 to 39 in 5b4731c

    
           try: 
        
               import OpenGL.GL   # noqa 
        
           except ImportError: 
        
               # print('Drat, patching for Big Sur') 
        
               orig_util_find_library = util.find_library 
        
               def new_util_find_library(name): 
        
                   res = orig_util_find_library(name) 
        
                   if res: 
        
                       return res 
        
                   lut = { 
        
                       'objc': 'libobjc.dylib', 
        
                       'quartz': 'Quartz.framework/Quartz' 
        
                   } 
        
                   return lut.get(name, name+'.framework/'+name) 
        
               util.find_library = new_util_find_library

We see that if the original find_library works, then it just uses it.
The lut from the monkey patch only triggers if the name isn't found
And I checked, this is the case only for lowercase quartz. When it triggers, instead of None it returns the wrong path and then leads to the issue. Now with this PR, without pyopengl, when quartz fails this PR provides the correct path -- note it uses Q already!
So it seems the simple solution is to directly use Quartz here:

vispy/vispy/ext/cocoapy.py

Line 1288 in 5b4731c

quartz = cdll.LoadLibrary(util.find_library('quartz'))

Which should also solve the issue by preventing the monkeypatch from firing.
I made a branch with this:
https://github.com/psobolewskiPhD/vispy/tree/fix_quartz_cdll_simple
it seems to work fine, but maybe there is an ancient macOS where Quartz was quartz? But it was Quartz when first available in 10.4 if I'm reading these docs right
https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/OSX_Technology_Overview/SystemFrameworks/SystemFrameworks.html
(the document itself is from 2015, so ~macos 10.11, vs Big Sur from 2020.

Frankly, it looks like the fix in ctypes for cdlls in bigsur is out in python >3.8
https://bugs.python.org/issue41100
vispy doesn't support 3.7 anymore.
So one could drop the monkey patch entirely and just fix the Q
I tried that, but get a cibuildwheel test failure.... with cdll loading -- AppKit as far as I can tell.
Looks like it's a python 3.8 issue...
#1885 (comment)
Anyhow, here's the branch
https://github.com/psobolewskiPhD/vispy/tree/fix_quartz_cdll

ataffanel · 2024-03-20T13:04:34Z

@djhoese No, I do not have pyopengl. Installing pyopengl does indeed fixes the crash even when using the current vispy release! So adding pyopengl as a dependency to Vispy might also be a solution? Though, if it can be avoided it feels that avoiding to add a new dependency would be a good thing. At least this does give me a quick way forward to make my software work on mac, I can add the dependency on my side.

@psobolewskiPhD You branch works fine on a clean Sonoma (M1) using Apple's Python 3.9 and yes, it does look much simpler! I am trying to test on older version on an old X86 mac, but I am not sure at how long back to try. I tried 10.13 and it would not even install the command line tools, 10.15 does not want to install vispy with apple's python and installing python with brew is currently recompiling everything. I have a couple of mac setup right now to I can help testing but the question is: what is the oldest version of MacOS/Python supported by Vispy?

aganders3 · 2024-06-04T19:44:43Z

it seems to work fine, but maybe there is an ancient macOS where Quartz was quartz? But it was Quartz when first available in 10.4 if I'm reading these docs right

I have a theory here. The macOS filesystem is (typically) case-preserving, but case-insensitive. New versions of macOS (11+) take commonly used libs and link them into a single "cache" (the DYLD shared cache). When Python merged support for macOS 11+ (python/cpython#22855) it added a lookup for these shared libs in the cache. If it was previously just looking on the filesystem, the case would not matter, but perhaps it does matter when looking in the cache.

aganders3 · 2024-06-14T13:56:26Z

So one could drop the monkey patch entirely and just fix the Q
I tried that, but get a cibuildwheel test failure.... with cdll loading -- AppKit as far as I can tell.
Looks like it's a python 3.8 issue...
#1885 (comment)
Anyhow, here's the branch
https://github.com/psobolewskiPhD/vispy/tree/fix_quartz_cdll

Note #2599 could drop support for Python <= 3.8, so maybe this would be the correct fix after that.

darrenchang · 2024-10-06T10:22:48Z

MacOS 15.0 (M1)
Python 3.12.6

pip install pyopengl fixes the issue for me

brisvag · 2025-01-16T09:59:15Z

Sorry for dropping the ball on this. Did we confirm that there is no regression here and only improvement? We can always clean up the monekypatch later if it's useless.

djhoese · 2025-01-16T15:51:32Z

I personally have lost track of if this is needed. I, as not a mac user, think we've waited long enough on this that backwards compatibility of this solution doesn't matter too much. I think MacOS 13 (which others said this PR works on) is the oldest version of Mac for GitHub Actions now (12 is deprecated, 14 is "latest"). Again, as not a mac user, I'm OK merging this since it seems to work for most people, but it also seems unnecessary give the deprecations in MacOS support (on GH Actions at least) and Python version.

psobolewskiPhD · 2025-01-19T19:10:39Z

I think #2549 (comment) is still correct.
So because 3.8 has been dropped in #2599
https://github.com/vispy/vispy/blame/469d337ae24083899e2d5a8f2fa8779de233a568/setup.py#L99

Then I think the way to go is:
https://github.com/psobolewskiPhD/vispy/tree/fix_quartz_cdll

I've run CI on my fork:
https://github.com/psobolewskiPhD/vispy/pull/2/checks
you can see the macOS build failure is gone now.

Edit: that said, I can't reproduce the issue on my Sonoma mac.
I found vispy.util.dpi.get_dpi uses quartz, so my test it:

from vispy.util.dpi import get_dpi
get_dpi()

(No pyopengl installed, so the monkey patch should be triggering on main)

brisvag · 2025-01-20T10:30:32Z

Then I think the way to go is:
psobolewskiPhD/vispy@fix_quartz_cdll

Do you have a PR open with this?

psobolewskiPhD · 2025-01-21T03:11:41Z

@brisvag here you go:
#2635

Alternative to: #2549 This PR drops the monkeypatch because: - The fix in ctypes for cdlls in bigsur is out in python >3.8: https://bugs.python.org/issue41100 - The remaining issue was python 3.8, which has been dropped in #2599 Also, changes the name of the library to the proper `Quartz`, as per: https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/OSX_Technology_Overview/SystemFrameworks/SystemFrameworks.html The lower case may work due with case insensitive file system, but as noted by Ashley, with caching, it may fail.

takacsmark added 2 commits November 15, 2023 19:39

MacOs 14 Sonoma dlopen cache fix

d480c9e

Updated the find_library patch for backward compatibility Closes vispy#2547

string interpolation fix

e5ff8b8

takacsmark mentioned this pull request Nov 15, 2023

Cannot find Quartz library on MacOS Sonoma #2547

Closed

djhoese changed the title ~~MacOs 14 Sonoma dlopen cache fix~~ Fix MacOs 14 Sonoma dlopen cache lookup Nov 15, 2023

djhoese added type: bug component: app labels Nov 15, 2023

djhoese self-assigned this Nov 15, 2023

knmcguire mentioned this pull request Mar 19, 2024

Client not starting on Mac sonoma bitcraze/crazyflie-clients-python#694

Closed

psobolewskiPhD mentioned this pull request Jan 21, 2025

Ensure proper quartz cdll loading on macOS (cocoapy) #2635

Merged

brisvag closed this Jan 21, 2025

Fix MacOs 14 Sonoma dlopen cache lookup #2549

Fix MacOs 14 Sonoma dlopen cache lookup #2549

Uh oh!

Conversation

takacsmark commented Nov 15, 2023

Uh oh!

djhoese commented Nov 15, 2023

Uh oh!

brisvag commented Nov 16, 2023

Uh oh!

psobolewskiPhD commented Nov 16, 2023

Uh oh!

ataffanel commented Mar 19, 2024

Uh oh!

djhoese commented Mar 19, 2024

Uh oh!

psobolewskiPhD commented Mar 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ataffanel commented Mar 20, 2024

Uh oh!

aganders3 commented Jun 4, 2024

Uh oh!

aganders3 commented Jun 14, 2024

Uh oh!

darrenchang commented Oct 6, 2024

Uh oh!

brisvag commented Jan 16, 2025

Uh oh!

djhoese commented Jan 16, 2025

Uh oh!

psobolewskiPhD commented Jan 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brisvag commented Jan 20, 2025

Uh oh!

psobolewskiPhD commented Jan 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

psobolewskiPhD commented Mar 20, 2024 •

edited

Loading

psobolewskiPhD commented Jan 19, 2025 •

edited

Loading