Osx toolchain update #5582
Osx toolchain update #5582
Conversation
|
I was a bit foggy last night when writing the description after going in circles with SDKs and toolchains all day, so I'll try to provide a better breakdown here. XCodeWhen building in the normal OSX ecosystem (download a monolithic Xcode release and use it without asking questions), the compiler, linker, build tools, and SDKs all come together. However, they retain a relatively high amount of back/forward-compatibility with each-other. Each Xcode release typically comes with ~2 SDKs for use. That’s ignoring iOS, which we’re not interested in. The current stable release is Xcode 6.1.1. SDKsThe SDKs contain features available for the respective OS version, but they also remain backwards-compatible depending on a compiler switch that sets the minimum desired runtime environment. A new SDK is typically released just before each major OS upgrade. Example: As you can see, 10.10 features can happily be compiled and used, but care needs to be taken to ensure that they’re tested at runtime first. So the only downside in building against the latest/greatest SDK is that someone might have forgotten to add a runtime check for a new feature which would cause a crash in older environments. Using an older SDK in the same scenario would cause a build failure instead. ToolchainsEach version of XCode ships with a new toolchain, which is comprised of a compiler (clang/clang++), linker (ld64, apple’s own, not gold/bfd), and cctools (binutils in the rest of the unix world: ar, ranlib, nm, etc). For the most part, the toolchains and SDKs are unrelated. A sufficiently new toolchain can build using any SDK. StdlibPrior to 10.7, OSX shipped with only gnu libstdc++ runtime libs. The 10.7 SDK switched to clang and added the libc++ option, so devs could build using libc++ if they were willing to set a minimum supported runtime of 10.7. The 10.9 SDK made libc++ the default, however it’s still possible to use -stdlib=libstdc++ to avoid the change. Also, setting -mmacosx-version-min=10.6 will force the use of libstdc++, since it’s not available there. The main take-away here is that in order to build using libc++, the minimum runtime version will require a bump from 10.6 to 10.7. Putting it all togetherBecause we roll our own toolchain and build environments in Linux, we’re free to mix and match the above components however we desire, as long as they all play nicely together. For the work in this PR, I originally bumped to the following: That put us 100% up-to-date with apple’s current stable tools. Everything worked well. However, there’s not an official clang 3.5 binary for Precise, which is used by Gitian and Travis. There is a Trusty binary, but it requires a newer libstdc++ and can’t run. To get around that, I next tried downgrading clang to 3.4, which is compatible with Precise. Unfortunately, there’s a bug in 3.4 that breaks darwin builds, as discussed briefly with @gmaxwell last night. I’m still working to track down the exact cause. Finally, I arrived at clang 3.3, which builds and works fine, however it is not capable of building with the 10.10 sdk. It’s fine up to 10.9 though. TakeawaySo the final result here is that with these changes we're using the latest/greatest apple tools and SDKs and we're in good shape for the foreseeable future, with the exception that we're currently slightly constrained by our build environments. Before these changes, we were using much older versions of all of these tools, which limited the SDKs we could target. Those versions of cctools/ld64 also only worked when compiled as 32bit. Since ld64 requires the use of clang's libLTO, that meant that we were also stuck using a 32bit clang. |
|
While I was looking at the clang-3.4 problem yesterday, robbak on IRC happened to be wrestling with the same problem on FreeBSD. I tested a vanilla osx build from llvm.org natively on osx, and it had the same problem. I've managed to track down the problem, and have a confirmed fix. This llvm-mirror/clang@c174fe5 needs to be backported to the 3.4 branch. I'll report it to the LLVM list after the new year and see if they're interested in doing a 3.4.3 release, since the official 3.4.2 binaries are quite broken for (at a minimum) OSX and FreeBSD. |
|
Nice. Tested. This fixes the Qt issue of building on 10.9 with the 10.10 SDK. |
|
Thanks for figuring this all out @theuni! |
|
@laanwj master only. I don't foresee any problems, but it'd be far too risky for 0.10 at the last minute. |
theuni
force-pushed
the
osx-toolchain-update
branch
from
625b889 to
46f54bf
Compare
tl;dr: Update to the newer stable toolchain and SDK for OSX without giving up any backwards compatibility. We can move to clang 3.5 as a next step which allows use to use libc++ and the 10.10 sdk, but we'll need to find a build that works in gitian/travis first. Switch to a new, better maintained fork of cctools: https://github.com/tpoechtrager/cctools-port I've forked this and will be working on it some as well: https://github.com/theuni/cctools-port This brings in: cctools v862 ld64: v241.9 It also fixes 64bit builds, so there's no longer any need to use a 32bit clang. Since clang is no longer tied to an old/crusty 32bit build, clang has been upgraded to 3.3. Unfortunately, there's a bug in 3.4 that breaks builds. 3.5 works fine, but there are no binary builds compatible with precise, which is currently used for gitian and travis. We could always build our own if necessary. After updating to stable clang/linker/cctools, it's possible to use a more recent SDK. The current SDK (10.7) through the most recent 10.10 have all been built/tested successfully, both with and without 10.6 compatibility. However, 10.10 requires clang 3.5. SDKs >= 10.9 use libc++ rather than libstdc++. This is verified working as well.
|
First commit dropped, and rebased on top of #5586 |
This one no longer requires native libuuid or libcrypto
|
utACK |
3 participants
tl;dr: Update to a newer stable toolchain and SDK for OSX without giving up
any backwards compatibility. We can move to clang 3.5 as a next step which
allows use to use libc++ and the 10.10 sdk, but we'll need to find a build that
works in gitian/travis first.
Switch to a new, better maintained fork of cctools:
https://github.com/tpoechtrager/cctools-port
I've forked this and will be working on it some as well:
https://github.com/theuni/cctools-port
This brings in:
cctools v862
ld64: v241.9
It also fixes 64bit builds, so there's no longer any need to use a 32bit clang.
Since clang is no longer tied to an old/crusty 32bit build, clang has been
upgraded to 3.3. Unfortunately, there's a bug in 3.4 that breaks builds. 3.5
works fine, but there are no binary builds compatible with precise, which is
currently used for gitian and travis. We could always build our own if
necessary.
After updating to stable clang/linker/cctools, it's possible to use a more
recent SDK. The current SDK (10.7) through the most recent 10.10 have all been
built/tested successfully, both with and without 10.6 compatibility. However,
10.10 requires clang 3.5.
SDKs >= 10.9 use libc++ rather than libstdc++. This is verified working as well.
Sorry for the thought-dump, there are lots of pieces here. Despite that, this should be a pretty smooth upgrade. The messier parts (bumping the minimum runtime requirement, switching build SDKs, switching to libc++, rolling our own clang, etc) are left for later. All of those things have been tested and work fine, they just need to be discussed first.