Skip to content

Prevent conflicts between libuuid and Apple Cocoa Framework on Mac#12166

Merged
adamjstewart merged 4 commits intospack:developfrom
sethrj:conflict-uuid-mac
Jul 28, 2019
Merged

Prevent conflicts between libuuid and Apple Cocoa Framework on Mac#12166
adamjstewart merged 4 commits intospack:developfrom
sethrj:conflict-uuid-mac

Conversation

@sethrj
Copy link
Copy Markdown
Contributor

@sethrj sethrj commented Jul 28, 2019

Building Python with +uuid currently requires the Spack-built libuuid package. However, the vanilla libuuid is incompatible with Apple's frameworks. This results in build failures in Python 2.7 (since that version of Python uses Carbon, which also includes libuuid), as well as in code that's downstream of Python (such as py-matplotlib, which includes an osx backend).

The commit message documents the error messages for system/compiler versions:

    MacOS version 10.14.5
    Xcode 10.2.1
    Apple LLVM version 10.0.1 (clang-1001.0.46.4)

but the final error message is always:

/usr/include/hfs/hfs_format.h:794:2: error: unknown type name 'uuid_string_t'; did you mean 'io_string_t'?

because uuid_string_t is defined only in Apple's uuid.h and not in Spack's.

@sethrj sethrj requested a review from adamjstewart July 28, 2019 16:56
adamjstewart
adamjstewart reviewed Jul 28, 2019
View reviewed changes
sethrj added 2 commits July 28, 2019 13:43
@sethrj
Remove libuuid dependency for python on Darwin
cd544c9 
Python 2.7 includes Carbon, which requires a proprietary typedef defined in
the system `/usr/include/uuid/uuid.h`:
```
In file included from /rnsdhpc/code/spack/var/spack/stage/python-2.7.16-chgwvgls4ezgujsknn27vlmkhwbzpqik/Python-2.7.16/Python/mactoolboxglue.c:27:
In file included from /rnsdhpc/code/spack/var/spack/stage/python-2.7.16-chgwvgls4ezgujsknn27vlmkhwbzpqik/Python-2.7.16/Include/pymactoolbox.h:10:
In file included from /System/Library/Frameworks/Carbon.framework/Headers/Carbon.h:20:
In file included from /System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:23:
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:208:
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/HFSVolumes.h:25:
/usr/include/hfs/hfs_format.h:794:2: error: unknown type name 'uuid_string_t'; did you mean 'io_string_t'?
```

Similarly, even with Python 3, having Spack's libuuid in the toolchain
causes matplotlib to fail to build downstream because its `osx` backend
links against Carbon:
```
building 'matplotlib.backends._macosx' extension
/rnsdhpc/code/spack/lib/spack/env/clang/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include -I/usr/X11/include -I/opt/X11/include -I. -I/rnsdhpc/code/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/python-3.7.2-bipx24oyxqxowqc75o5vfih7dljiaohz/include/python3.7m -c src/_macosx.m -o build/temp.macosx-10.14.5-x86_64-3.7/src/_macosx.o
In file included from src/_macosx.m:1:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Cocoa.framework/Headers/Cocoa.h:12:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:87:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSURLError.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:24:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:208:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/HFSVolumes.h:25:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/hfs/hfs_format.h:794:2: error: unknown type name 'uuid_string_t'; did you mean 'io_string_t'?
        uuid_string_t   ext_jnl_uuid;
        ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/device/device_types.h:89:33: note: 'io_string_t' declared here
typedef char                    io_string_t[512];
```

MacOS version 10.14.5
Xcode 10.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
@sethrj
Mark libuuid as conflicting with Darwin due to framework
9a68b75 
If Spack's libuuid is included by any other package that tests for or
requires the Cocoa or Carbon frameworks, the build will *fail* becuse it
appears that Apple's libraries require a special version of libuuid and
uuid.h.

```
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Cocoa.framework/Headers/Cocoa.h:12:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:87:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSURLError.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:24:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:208:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/HFSVolumes.h:25:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/hfs/hfs_format.h:794:2: error: unknown type name 'uuid_string_t'; did you mean 'io_string_t'?
        uuid_string_t   ext_jnl_uuid;
        ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/device/device_types.h:89:33: note: 'io_string_t' declared here
typedef char                    io_string_t[512];
```
@sethrj sethrj force-pushed the conflict-uuid-mac branch from 9b386e2 to 9a68b75 Compare July 28, 2019 17:43
@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented Jul 28, 2019

It looks like a few other packages also depend on libuuid. Should we make the dep conditional for these as well?

$ grep depends_on.*libuuid */package.py
cdo/package.py:    depends_on('libuuid')
czmq/package.py:    depends_on("libuuid")
davix/package.py:    depends_on('libuuid')
dia/package.py:    depends_on('libuuid')
flux-sched/package.py:    depends_on("libuuid")
gaudi/package.py:    depends_on('libuuid')
psm/package.py:    depends_on('libuuid')
python/package.py:    depends_on('libuuid', when='+uuid')
singularity/package.py:    depends_on('libuuid')
task/package.py:    depends_on('libuuid')
taskd/package.py:    depends_on('libuuid')
xeus/package.py:    depends_on('libuuid')

@sethrj
Copy link
Copy Markdown
Contributor Author

sethrj commented Jul 28, 2019

I'm hesitant to do so without checking. The Darwin UUID is rather diverged from the main UUID, which includes uuid_time functions. I don't think it's safe to assume that the other packages don't need that functionality.

I think that checking each one of those packages for mac compatibility is outside the scope of this PR... do you agree?

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented Jul 28, 2019

In that case, should we remove the conflicts from the libuuid package? As is, all of these packages will simply fail during concretization.

@sethrj
Copy link
Copy Markdown
Contributor Author

sethrj commented Jul 28, 2019

Well, I guess in my view failing during concretization is safer than letting some far-downstream dependency (the first common descendent of libuuid and the Carbon/Cocoa frameworks) fail to build with a difficult-to-decipher error message. For example, Python 3.7 (which uses libuuid) built without error against the Spack libuuid, but Matplotlib failed because of the dependency. If I hadn't explicitly been looking for that kind of error, it would have been very difficult to solve.

In a perfect world spack would have a Mac-only, not-buildable "Cocoa" package (analogous to the opengl package) that would be listed as a dependency for codes that link against the mac frameworks. Then we could narrow the conflict to ^cocoa.

In this imperfect world where such a package does not exist, we've got a couple of options and it's up to the Spack leadership/philosophizers to choose:

  • Assume downstream packages build until proven otherwise (remove the conflict from libuuid)
  • Assume downstream packages break until proven otherwise (leave the conflict directive)
  • Test downstream packages for compatibility (and assume the system and versions being tested are representative of all the other possible combinations of systems and versions)
  • Add such a cocoa package and gradually integrate it with existing packages in the hopes that we can catch build conflicts before they occur.

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented Jul 28, 2019

I vote for 1 for now. If no one has noticed that those packages are broken yet, then they may actually work fine. Innocent until proven guilty.

@sethrj
Copy link
Copy Markdown
Contributor Author

sethrj commented Jul 28, 2019

Fair enough :) removing the conflict remark. Hopefully anyone who might encounter a uuid.h error will be able to find this PR.

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented Jul 28, 2019

Can you also add a comment to the file saying why the dependency is disabled on macOS?

@adamjstewart
Copy link
Copy Markdown
Member

adamjstewart commented Jul 28, 2019

Since you're already playing around with matplotlib, you may be interested in #12164.

@adamjstewart adamjstewart merged commit 95985e0 into spack:develop Jul 28, 2019
@sethrj sethrj deleted the conflict-uuid-mac branch July 29, 2019 02:16
dev-zero pushed a commit to dev-zero/spack that referenced this pull request Aug 13, 2019
@sethrj @dev-zero
Prevent conflicts between libuuid and Apple Cocoa Framework on Mac (s…
98fba8e 
…pack#12166)

* Remove libuuid dependency for python on Darwin

Python 2.7 includes Carbon, which requires a proprietary typedef defined in
the system `/usr/include/uuid/uuid.h`:
```
In file included from /rnsdhpc/code/spack/var/spack/stage/python-2.7.16-chgwvgls4ezgujsknn27vlmkhwbzpqik/Python-2.7.16/Python/mactoolboxglue.c:27:
In file included from /rnsdhpc/code/spack/var/spack/stage/python-2.7.16-chgwvgls4ezgujsknn27vlmkhwbzpqik/Python-2.7.16/Include/pymactoolbox.h:10:
In file included from /System/Library/Frameworks/Carbon.framework/Headers/Carbon.h:20:
In file included from /System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:23:
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:208:
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/HFSVolumes.h:25:
/usr/include/hfs/hfs_format.h:794:2: error: unknown type name 'uuid_string_t'; did you mean 'io_string_t'?
```

Similarly, even with Python 3, having Spack's libuuid in the toolchain
causes matplotlib to fail to build downstream because its `osx` backend
links against Carbon:
```
building 'matplotlib.backends._macosx' extension
/rnsdhpc/code/spack/lib/spack/env/clang/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include -I/usr/X11/include -I/opt/X11/include -I. -I/rnsdhpc/code/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/python-3.7.2-bipx24oyxqxowqc75o5vfih7dljiaohz/include/python3.7m -c src/_macosx.m -o build/temp.macosx-10.14.5-x86_64-3.7/src/_macosx.o
In file included from src/_macosx.m:1:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Cocoa.framework/Headers/Cocoa.h:12:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:87:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSURLError.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:24:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:208:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/HFSVolumes.h:25:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/hfs/hfs_format.h:794:2: error: unknown type name 'uuid_string_t'; did you mean 'io_string_t'?
        uuid_string_t   ext_jnl_uuid;
        ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/device/device_types.h:89:33: note: 'io_string_t' declared here
typedef char                    io_string_t[512];
```

MacOS version 10.14.5
Xcode 10.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)

* Mark libuuid as conflicting with Darwin due to framework

If Spack's libuuid is included by any other package that tests for or
requires the Cocoa or Carbon frameworks, the build will *fail* becuse it
appears that Apple's libraries require a special version of libuuid and
uuid.h.

```
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Cocoa.framework/Headers/Cocoa.h:12:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:87:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSURLError.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:24:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:208:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/HFSVolumes.h:25:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/hfs/hfs_format.h:794:2: error: unknown type name 'uuid_string_t'; did you mean 'io_string_t'?
        uuid_string_t   ext_jnl_uuid;
        ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/device/device_types.h:89:33: note: 'io_string_t' declared here
typedef char                    io_string_t[512];
```

* Revert "Mark libuuid as conflicting with Darwin due to framework"

This reverts commit 9a68b75.

* Document libuuid exception in python package
@sethrj sethrj mentioned this pull request Aug 17, 2019
Closed
@sethrj sethrj mentioned this pull request Feb 18, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adamjstewart adamjstewart adamjstewart approved these changes

Assignees

No one assigned

Labels

conflicts macOS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sethrj @adamjstewart