qt +opengl: fix macOS w/ AppleClang (mesa libs) by ax3l · Pull Request #16302 · spack/spack

ax3l · 2020-04-25T20:06:26Z

Building the py-jupyter stack on macOS with AppleClang breaks on the py-qtconsole -> py-qtconsole -> qt +opengl package build environment setup with

 ==> Error: AttributeError: Query of package 'mesa' for 'libs' failed
    prefix : /Users/runner/runners/2.169.1/work/macos-nightly/macos-nightly/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/mesa-18.3.6-uu5cyvrzka5g2xfjguuf2tngqsgkgfvf
    spec : [email protected]%[email protected]~glx+llvm+opengl~opengles+osmesa patches=55a5611ca9fcbe8324c4d68a07b338134954ff12c5b122dc78ff376f012a1414 swr=none arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] patches=89aa362716d898edd0b5c6ae4208dc1b6992887774848a09e8021afd676f7d61 arch=darwin-catalina-x86_64 ^[email protected]%[email protected]+shared arch=darwin-catalina-x86_64 ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected]~libbsd arch=darwin-catalina-x86_64 ^[email protected]%[email protected] patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=darwin-catalina-x86_64 ^[email protected]%[email protected]+lex patches=09c22e5c6fef327d3e48eb23f0d610dcd3a35ab9207f12e0f875701c677978d3 arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected]~cairo~cuda~gl+libxml2~netloc~nvml~pci+shared arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected]~python arch=darwin-catalina-x86_64 ^[email protected]%[email protected]~all_targets build_type=Release +clang~code_signing+compiler-rt~cuda cuda_arch=none ~gold+internal_unwind+libcxx+lld+lldb~mlir~omp_debug~omp_tsan patches=332fe65f78b2b4a242045ec2394eee8db631fbcbe27b0016d5e5c859e34f47af +polly~python~shared_libs~split_dwarf arch=darwin-catalina-x86_64 ^[email protected]%[email protected] patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=darwin-catalina-x86_64 ^[email protected]%[email protected]~symlinks+termlib arch=darwin-catalina-x86_64 ^[email protected]%[email protected]+systemcerts arch=darwin-catalina-x86_64 ^[email protected]%[email protected]~jit+multibyte+utf arch=darwin-catalina-x86_64 ^[email protected]%[email protected]+cpanm+shared+threads arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations patches=210df3f28cde02a8135b58cc4168e70ab91dbf9097359d05938f1e2843875e57 +pic+pyexpat+pythoncmd+readline~shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected]+column_metadata+fts~functions~rtree arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected] patches=12f6edb0c6b270b8c8dba2ce17998c580db01182d871ee32b7b6e4129bd1d23a,1732115f651cff98989cb0215d8f64da5e0f7911ebf0c13b064920f088f2ffe1 arch=darwin-catalina-x86_64 ^[email protected]%[email protected] arch=darwin-catalina-x86_64 ^[email protected]%[email protected]~python arch=darwin-catalina-x86_64 ^[email protected]%[email protected]+optimize+pic+shared arch=darwin-catalina-x86_64
    queried as : mesa
    extra parameters : []

 /Users/runner/runners/2.169.1/work/macos-nightly/macos-nightly/spack/lib/spack/spack/build_environment.py:819, in child_process:
         816            tb_string = traceback.format_exc()
         817
         818            # build up some context from the offending package so we can
   >>    819            # show that, too.
         820            package_context = get_package_context(tb)
         821
         822            build_log = None


 ==> Error: Failed to install qt due to ChildError: AttributeError: Query of package 'mesa' for 'libs' failed

This tries to add more library targets build by mesa to avoid this.

Refs.:

ax3l · 2020-04-25T20:12:29Z

var/spack/repos/builtin/packages/mesa/package.py

+            libs = find_libraries(['libGL', 'libOSMesa'],
+                                  join_path(self.prefix, dir),
                                  shared=True, recursive=False)
            if libs:


@adamjstewart @v-dobrev is it intentional to return None here instead of an empty list? This seams to be generally problematic when setting up the build environment for dependent packages, see the PR description error message.

Returning None informs the libs handler that the search failed (and reporting it as an error is the right behavior). Simply returning an empty list is not treated as an error, see

spack/lib/spack/spack/spec.py

Lines 862 to 891 in 4cd537f

# Trigger the callbacks in order, the first one producing a

# value wins

value = None

message = None

for f in callbacks_chain:

try:

value = f()

# A callback can return None to trigger an error indicating

# that the query failed.

if value is None:

msg = "Query of package '{name}' for '{attrib}' failed\n"

msg += "\tprefix : {spec.prefix}\n"

msg += "\tspec : {spec}\n"

msg += "\tqueried as : {query.name}\n"

msg += "\textra parameters : {query.extra_parameters}"

message = msg.format(

name=pkg.name, attrib=self.attribute_name,

spec=instance, query=instance.last_query)

else:

return value

break

except AttributeError:

pass

# value is 'None'

if message is not None:

# Here we can use another type of exception. If we do that, the

# unit test 'test_getitem_exceptional_paths' in the file

# lib/spack/spack/test/spec_dag.py will need to be updated to match

# the type.

raise AttributeError(message)

Note that the default handler (used if the libs property is not explicitly defined by a package) works in a similar manner:

spack/lib/spack/spack/spec.py

Lines 792 to 799 in 4cd537f

for shared in search_shared:

libs = fs.find_libraries(

name, spec.prefix, shared=shared, recursive=True)

if libs:

return libs

msg = 'Unable to recursively locate {0} libraries in {1}'

raise spack.error.NoLibrariesError(msg.format(spec.name, spec.prefix))

The raised exception is different but that is just unintentional inconsistency.

Thanks for all the details 👍
Why do we set the property at all, isn't the default implementation that searches for all shared libs sufficient (for glvis)?

The default search uses the name of the package only, so for mesa it will search for libmesa.{a,so,dylib} which is not the real name of the library, see the code just above the second link I posted above.

In glvis, we use spec['gl'].libs which is then delegated by spack to the specific provider for the virtual gl package -- the default for that being mesa.

ohhh, thanks again. TIL!

ax3l · 2020-04-25T20:24:25Z

@v-dobrev @hppritcha @chuckatkins @scottwittenburg @opadron
I saw in the git logs that you contribute significantly to mesa in spack 👍
Do you want to be listed as package maintainers to avoid edit races?

v-dobrev · 2020-04-26T00:08:04Z

@v-dobrev @hppritcha @chuckatkins @scottwittenburg
I saw in the git logs that you contribute significantly to mesa in spack 👍
Do you want to be listed as package maintainers to avoid edit races?

I've primarily edited this package to ensure it works well as a (virtual) dependency of the glvis package -- mesa is the default choice for the virtual gl package. Also, I don't use mesa really and don't know much (even anything) about the package and its installation process.

That said, I don't mind to be added as a maintainer with the understanding that my knowledge of the package itself is very limited.

Building the `py-jupyter` stack on macOS with AppleClang breaks on the `py-qtconsole` -> `py-qtconsole` -> `qt +opengl` package build environment setup with: ``` ==> Error: AttributeError: Query of package 'mesa' for 'libs' failed ... ==> Error: Failed to install qt due to ChildError: AttributeError: Query of package 'mesa' for 'libs' failed ``` This tries to add more library targets build by `mesa` to avoid this.

ax3l · 2020-04-27T00:07:09Z

I'll merge this for now since the GitHub action test confirms this fixes the build on macOS + AppleClang.

ax3l added macOS build-error update-package opengl labels Apr 25, 2020

ax3l requested review from adamjstewart, tgamblin and v-dobrev April 25, 2020 20:06

ax3l commented Apr 25, 2020

View reviewed changes

ax3l requested a review from chuckatkins April 25, 2020 20:24

ax3l added the virtual-dependencies label Apr 25, 2020

ax3l requested a review from scottwittenburg April 25, 2020 20:38

ax3l changed the title ~~qt: fix macOS w/ AppleClang (mesa libs)~~ qt +opengl: fix macOS w/ AppleClang (mesa libs) Apr 25, 2020

ax3l requested a review from opadron April 26, 2020 00:36

ax3l force-pushed the fix-qtMacOSClangMesa branch from d24469c to 69d3dc2 Compare April 26, 2020 02:13

ax3l force-pushed the fix-qtMacOSClangMesa branch from 69d3dc2 to 81954d0 Compare April 26, 2020 04:41

ax3l merged commit 4f3cc19 into spack:develop Apr 27, 2020

ax3l deleted the fix-qtMacOSClangMesa branch April 27, 2020 00:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

qt +opengl: fix macOS w/ AppleClang (mesa libs)#16302

qt +opengl: fix macOS w/ AppleClang (mesa libs)#16302
ax3l merged 1 commit intospack:developfrom
ax3l:fix-qtMacOSClangMesa

ax3l commented Apr 25, 2020 •

edited

Loading

Uh oh!

ax3l Apr 25, 2020 •

edited

Loading

Uh oh!

v-dobrev Apr 25, 2020

Uh oh!

ax3l Apr 26, 2020 •

edited

Loading

Uh oh!

v-dobrev Apr 26, 2020

Uh oh!

ax3l Apr 26, 2020

Uh oh!

ax3l commented Apr 25, 2020 •

edited

Loading

Uh oh!

v-dobrev commented Apr 26, 2020

Uh oh!

ax3l commented Apr 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	# Trigger the callbacks in order, the first one producing a
	# value wins
	value = None
	message = None
	for f in callbacks_chain:
	try:
	value = f()
	# A callback can return None to trigger an error indicating
	# that the query failed.
	if value is None:
	msg = "Query of package '{name}' for '{attrib}' failed\n"
	msg += "\tprefix : {spec.prefix}\n"
	msg += "\tspec : {spec}\n"
	msg += "\tqueried as : {query.name}\n"
	msg += "\textra parameters : {query.extra_parameters}"
	message = msg.format(
	name=pkg.name, attrib=self.attribute_name,
	spec=instance, query=instance.last_query)
	else:
	return value
	break
	except AttributeError:
	pass
	# value is 'None'
	if message is not None:
	# Here we can use another type of exception. If we do that, the
	# unit test 'test_getitem_exceptional_paths' in the file
	# lib/spack/spack/test/spec_dag.py will need to be updated to match
	# the type.
	raise AttributeError(message)

	for shared in search_shared:
	libs = fs.find_libraries(
	name, spec.prefix, shared=shared, recursive=True)
	if libs:
	return libs

	msg = 'Unable to recursively locate {0} libraries in {1}'
	raise spack.error.NoLibrariesError(msg.format(spec.name, spec.prefix))

Conversation

ax3l commented Apr 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ax3l Apr 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

v-dobrev Apr 25, 2020

Choose a reason for hiding this comment

Uh oh!

ax3l Apr 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

v-dobrev Apr 26, 2020

Choose a reason for hiding this comment

Uh oh!

ax3l Apr 26, 2020

Choose a reason for hiding this comment

Uh oh!

ax3l commented Apr 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

v-dobrev commented Apr 26, 2020

Uh oh!

ax3l commented Apr 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ax3l commented Apr 25, 2020 •

edited

Loading

ax3l Apr 25, 2020 •

edited

Loading

ax3l Apr 26, 2020 •

edited

Loading

ax3l commented Apr 25, 2020 •

edited

Loading