GHA/macos: improve, fix gcc/llvm, add new test matrix #14097
Conversation
vszakats
added
build
CI
vszakats
changed the title
bagder
left a comment
bagder
left a comment
There was a problem hiding this comment.
I have to agree, keeping them around is now only causing us grief
vszakats
changed the title
|
What's strange is that I cannot find any meaningful reference about this fallout on the internet, though something like this should hit elsewhere too. On the other hand I also cannot spot anything (obvious) in curl that could trigger such error. In theory an innocent typedef or keyword redefine could do this. Also In particular, #7128 also reported a GCC fallout, and that one was fixed by replacing one SDK header with another. |
|
Also noting that such innocent typedef / keyword-redefine may also occur in a dependency header. Like for example in this wolfSSL case with Though this fallout is unrelated to the TLS backend: As for the runner images: Notably GCC 12.4.0 (over 12.3.0): --- macos-14-arm64-Readme-0625.md
+++ macos-14-arm64-Readme-0701.md
@@ -6,7 +6,7 @@
# macOS 14
- OS Version: macOS 14.5 (23F79)
- Kernel Version: Darwin 23.5.0
-- Image Version: 20240625.2
+- Image Version: 20240701.9
## Installed Software
@@ -16,11 +16,11 @@
- Clang/LLVM 14.0.3
- Clang/LLVM (Homebrew) 15.0.7 - available on `$(brew --prefix llvm@15)/bin/clang`
- GCC 11 (Homebrew GCC 11.4.0) - available by `gcc-11` alias
-- GCC 12 (Homebrew GCC 12.3.0) - available by `gcc-12` alias
+- GCC 12 (Homebrew GCC 12.4.0) - available by `gcc-12` alias
- GCC 13 (Homebrew GCC 13.3.0) - available by `gcc-13` alias
- GCC 14 (Homebrew GCC 14.1.0_1) - available by `gcc-14` alias
- GNU Fortran 11 (Homebrew GCC 11.4.0) - available by `gfortran-11` alias
-- GNU Fortran 12 (Homebrew GCC 12.3.0) - available by `gfortran-12` alias
+- GNU Fortran 12 (Homebrew GCC 12.4.0) - available by `gfortran-12` alias
- GNU Fortran 13 (Homebrew GCC 13.3.0) - available by `gfortran-13` alias
- GNU Fortran 14 (Homebrew GCC 14.1.0_1) - available by `gfortran-14` alias
- Kotlin 2.0.0-release-341
@@ -34,7 +34,7 @@
- Bundler 2.5.14
- Carthage 0.39.1
- CocoaPods 1.15.2
-- Homebrew 4.3.7
+- Homebrew 4.3.8
- NPM 10.7.0
- NuGet 6.3.1.1
- Pip3 24.0 (python 3.12)
@@ -51,7 +51,7 @@
- 7-Zip 17.05
- aria2 1.37.0
- azcopy 10.25.1
-- bazel 7.2.0
+- bazel 7.2.1
- bazelisk 1.20.0
- bsdtar 3.5.3 - available by 'tar' alias
- Curl 8.6.0
@@ -69,19 +69,19 @@
- zstd 1.5.6
### Tools
-- AWS CLI 2.17.1
-- AWS SAM CLI 1.119.0
+- AWS CLI 2.17.5
+- AWS SAM CLI 1.120.0
- AWS Session Manager CLI 1.2.633.0
- Azure CLI 2.61.0
- Azure CLI (azure-devops) 1.0.1
- Bicep CLI 0.28.1
- Cmake 3.29.6
-- CodeQL Action Bundle 2.17.5
+- CodeQL Action Bundle 2.17.6
- Fastlane 2.221.1
- SwiftFormat 0.54.0
- Xcbeautify 2.4.0
- Xcode Command Line Tools 15.3.0.0.1.1708646388
-- Xcodes 1.4.1
+- Xcodes 1.5.0
### Linters
@@ -139,13 +139,13 @@
#### PowerShell Modules
- Az: 12.0.0
-- Pester: 5.6.0
+- Pester: 5.6.1
- PSScriptAnalyzer: 1.22.0
### Xcode
| Version | Build | Path |
| ---------------- | -------- | ------------------------------ |
-| 16.0 (beta) | 16A5171c | /Applications/Xcode_16.0.app |
+| 16.0 (beta) | 16A5171r | /Applications/Xcode_16.0.app |
| 15.4 | 15F31d | /Applications/Xcode_15.4.app |
| 15.3 | 15E204a | /Applications/Xcode_15.3.app |
| 15.2 | 15C500b | /Applications/Xcode_15.2.app |
@@ -198,17 +198,17 @@
| Simulator - watchOS 10.4 | watchsimulator10.4 | 15.3 |
| Simulator - watchOS 10.5 | watchsimulator10.5 | 15.4 |
| Simulator - watchOS 11.0 | watchsimulator11.0 | 16.0 |
-| visionOS 1.0 | xros1.0 | 15.2 |
| Simulator - visionOS 1.0 | xrsimulator1.0 | 15.2 |
-| Simulator - visionOS 1.1 | xrsimulator1.1 | 15.3 |
+| visionOS 1.0 | xros1.0 | 15.2 |
| visionOS 1.1 | xros1.1 | 15.3 |
+| Simulator - visionOS 1.1 | xrsimulator1.1 | 15.3 |
| Simulator - visionOS 1.2 | xrsimulator1.2 | 15.4 |
| visionOS 1.2 | xros1.2 | 15.4 |
| visionOS 2.0 | xros2.0 | 16.0 |
| Simulator - visionOS 2.0 | xrsimulator2.0 | 16.0 |
| Asset Runtime SDK for macOS hosts targeting watchOS 9.4 | assetruntime.host.macosx.target.watchos9.4 | 14.3.1 |
-| Asset Runtime SDK for macOS hosts targeting iOS 16.4 | assetruntime.host.macosx.target.iphoneos16.4 | 14.3.1 |
| Asset Runtime SDK for macOS hosts targeting tvOS 16.4 | assetruntime.host.macosx.target.appletvos16.4 | 14.3.1 |
+| Asset Runtime SDK for macOS hosts targeting iOS 16.4 | assetruntime.host.macosx.target.iphoneos16.4 | 14.3.1 |
| DriverKit 22.4 | driverkit22.4 | 14.3.1 |
| DriverKit 23.0 | driverkit23.0 | 15.0.1 |
| DriverKit 23.2 | driverkit23.2 | 15.1, 15.2 | |
|
autotools jobs with the new image compile without issues:
|
vszakats
force-pushed
the
gha-macos-del-gcc
branch
from
e57b6cc to
c9a9d42
Compare
|
Also not cmake related, it's just that we don't have an autotools job with gcc. (Despite calling |
vszakats
force-pushed
the
gha-macos-del-gcc
branch
from
1338a02 to
35593dd
Compare
|
edit: Nope, these fail with any gcc build. |
vszakats
force-pushed
the
gha-macos-del-gcc
branch
from
35593dd to
8877523
Compare
vszakats
force-pushed
the
gha-macos-del-gcc
branch
2 times, most recently
from
22e3cc7 to
ec72f3a
Compare
|
The best fix I could figure out is this ugly hack: --- a/lib/macos.c
+++ b/lib/macos.c
@@ -30,7 +30,14 @@
#include "macos.h"
+#if defined(__clang__)
#include <SystemConfiguration/SCDynamicStoreCopySpecific.h>
+#else
+#include <CoreFoundation/CFDictionary.h>
+typedef const struct CF_BRIDGED_TYPE(id) __SCDynamicStore * SCDynamicStoreRef;
+CFDictionaryRef __nullable
+SCDynamicStoreCopyProxies(SCDynamicStoreRef __nullable store);
+#endif
CURLcode Curl_macos_init(void)
{It works by avoiding the system headers that are incompatible at Another option is to find an alternative hack in place of what is Last but not least c-ares builds seem to be immune to this, because SecureTransport builds also require system headers and these Also it turns out that a bunch of tests fail in macOS + gcc builds.
|
|
Here's a CI run with gcc 11, 12, 13, 14 with SDK 14:
Is there anyone who can make some sense out of this? |
|
An even more bizarre run, with gcc 11, 12, 13, 14 on macos-12, 13, 14(latest) runners with both OpenSSL and SecureTransport.
There is a separate issue with gcc 11 + macos-14 + SecureTransport and autotools, where TL;DR:
|
vszakats
force-pushed
the
gha-macos-del-gcc
branch
from
036a09d to
dd0b584
Compare
|
So, while this patch for the hack restores some GCC compatibility, the compatibility situation between GCC and macOS SDK is fuzzy enough to make such builds fragile and overall impractical.
What do we prefer? |
vszakats
force-pushed
the
gha-macos-del-gcc
branch
6 times, most recently
from
a480435 to
a186cb9
Compare
This comment was marked as resolved.
This comment was marked as resolved.
|
OK, did a revert and a re-issue. |
As discovered earlier, Homebrew gcc is built against a specific Apple SDK and doesn't work when matched up with a different SDK, e.g. the one advertised as default by the macos runner image. Before this patch this was resolved with brute force by zapping the hack-layer gcc component to avoid the bad interference. This worked for us, but it's fragile, accidental and doesn't translate to any real-world build environment. Thus, impractical. This patch avoids this by explicitly configuring the SDK gcc was built for and meant to be used with. It assumes that the gcc binaries preinstalled on the runner images always ship with the SDK version they reference. It's non-trivial to guarantee that given the different build/release/update cycles of the projects involved, but since this is the only officially supported way to use them, hopefully this is ensured to be kept in sync. Since we don't do `brew update`, there is emphasis on preinstalled. In curl-for-win CI jobs we do `brew update` and expect that scenario to work too. Follow-up to c349bd6 curl#14097
As discovered earlier, Homebrew gcc is built against a specific Apple SDK version and doesn't work when matched up with a different version, e.g. the one advertised as default by the macos runner image. Before this patch this was resolved with brute force by zapping the hack-layer gcc component to avoid the bad interference. This worked for us, but it's fragile, accidental and doesn't translate to real-world build environments. Thus, impractical. Avoid this by explicitly selecting the SDK version gcc was built for and meant to be used with, as shown by `gcc --print-sysroot`. It assumes that the gcc binaries preinstalled on the runner images always ship with the SDK version they reference. It also assumes this works with and without `brew update`. Also: - add 4 quick build-only jobs to test all gcc/macos combos. - list SDKs offered via CommandLineTools. Suggested-by: Bo Anderson Ref: Homebrew/homebrew-core#194778 (comment) Follow-up to c349bd6 #14097 Closes #15518
Before this patch `--http2` did not work in gcc builds with Secure Transport, because ALPN relied on a compiler supporting the `HAVE_BUILTIN_AVAILABLE` aka `__builtin_available()` feature. This is clang-specific and missing from gcc (as of gcc v14). Add support for ALPN and HTTP/2 when this compiler feature is missing. Also drop test exceptions from GHA/macos in CI. Follow-up to 092f681 Ref: c349bd6 #14097 (issue 15.) Ref: #4314 Closes #16581
As discovered earlier, Homebrew gcc is built against a specific Apple SDK version and doesn't work when matched up with a different version, e.g. the one advertised as default by the macos runner image. Before this patch this was resolved with brute force by zapping the hack-layer gcc component to avoid the bad interference. This worked for us, but it's fragile, accidental and doesn't translate to real-world build environments. Thus, impractical. Avoid this by explicitly selecting the SDK version gcc was built for and meant to be used with, as shown by `gcc --print-sysroot`. It assumes that the gcc binaries preinstalled on the runner images always ship with the SDK version they reference. It also assumes this works with and without `brew update`. Also: - add 4 quick build-only jobs to test all gcc/macos combos. - list SDKs offered via CommandLineTools. Suggested-by: Bo Anderson Ref: Homebrew/homebrew-core#194778 (comment) Follow-up to c349bd6 curl#14097 Closes curl#15518
It fixes tests 1539, and 2402, 2404 (for non-Secure Transport), on macOS with the gcc compiler. Also unignore these tests in GHA/macos for non-secure transport. Ref: c349bd6 curl#14097 (issue 15.) Ref: 7b0240c curl#16539 Ref: 2ec0037 curl#16482 Closes curl#16580
To fix this test on macOS with the gcc compiler. Also unignore test 1156 in GHA/macos. Ref: c349bd6 curl#14097 (issue 15.) Ref: 7b0240c curl#16539 Ref: 2ec0037 curl#16482 Closes curl#16579
It runs fine now. Tested in all gcc-12 jobs after temporarly enabling debug in them all (test 2100 requires debug-enabled). Ref: c349bd6 curl#14097 (issue 15.) Closes curl#16582
Before this patch `--http2` did not work in gcc builds with Secure Transport, because ALPN relied on a compiler supporting the `HAVE_BUILTIN_AVAILABLE` aka `__builtin_available()` feature. This is clang-specific and missing from gcc (as of gcc v14). Add support for ALPN and HTTP/2 when this compiler feature is missing. Also drop test exceptions from GHA/macos in CI. Follow-up to 092f681 Ref: c349bd6 curl#14097 (issue 15.) Ref: curl#4314 Closes curl#16581
2 participants
This PR began as an attempt to drop GCC support, after repeated reports
on fallouts when trying to use it on macOS.
Then it transformed into a 3-week project turning up the issues causing
the fallouts, ending up including llvm and all available Xcode / macOS
SDK, macOS runner image, build tools and compiler vendors and versions.
Accumulating 400 sub-commits.
I developed and tested all fixes under this PR, then merged them as
separate patches.
This PR retained CI jobs updates, extensively reworking and extending
them: [1]
At first it seemed GCC and the Apple SDK is "naturally" growing more
incompatible, as Apple added further non-standard features to their
headers. This is partly true, but reality is more complicated.
Besides some issues local to curl, there were bugs in Apple SDK
headers, Homebrew GCC builds, feature missing in the old llvm version
pre-installed on GitHub CI runner images, and subtle incompatibilities
between GCC and llvm/clang when handling language extensions.
Resulting compiler errors seldom pointed to a useful direction, and
internet search was silent about these issues too. Thus, I had to peel
them off layer by layer, using trial and error, and by recognizing
patterns of failures accross 150-200 builds combinations. Exposing
configure logs, and curl_config.h in the CI logs helped too.
The toughest issue is GCC's built-in compatibility layer:
https://github.com/gcc-mirror/gcc/tree/master/fixincludes
This patch layer is further patched by a "Darwin compatibility" project
applied on top by Homebrew GCC via:
https://github.com/iains/gcc-12-branch
https://github.com/iains/gcc-13-branch
https://github.com/iains/gcc-14-branch
The hack layer is designed in a way that breaks more builds than it
fixes, esp. in context of GHA runners. The idea is to build GCC
specifically for the SDK for the target macOS version. The problem with
this approach is that the Xcode + SDK installed on the local/CI machine
often does not match with the SDK used on while building GCC on
Homebrew's build machines. In these cases the GCC compatibility layer
turns into an "uncompatibility" layer and consistently breaks builds.
curl cannot offer a fix for this, because the solution (I found) is to
patch the toolchain on the local machine. I implemented this for our CI
builds and curl-for-win. In other case the user must do this patching
manually, or choose a compatible GCC + Xcode/SDK combination.
An upstream fix doesn't seem trivial either, because the issue is
ingrained in the compatibility layer's design. Offering an
-fapplesdk(or recognizing
-target) option and/or fixing them within the compilerwould seem like a more robust option, and also how mainline llvm solves
this.
Here's a table summarizing the GCC + SDK combinations and curl build
failures: [2]
More info: #10356 (comment)
db135f8 #14119 macos: add workaround for gcc, non-c-ares, IPv6, compile error
Ref: curl/curl-for-win@e2db3c4
Ref: curl/curl-for-win@f5c58d7
availabilityextensionA recent minor Homebrew GCC upgrade caused major breakage. The "Darwin
compatibility" patch applied to GCC implemented the
availabilitycompiler attribute in GCC. Apple SDK detected this and enabled using
them, but as it turns out GCC accepts compiler attributes with slightly
different rules than llvm/clang, and how the Apple SDK uses them,
breaking builds.
Affected Homebrew GCC versions are: 12.4.0, 13.3.0 and 14.1.0.
Possibly tracked here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info: llvm/llvm-project#81767
Commit implementing the
availabilitymacro:gcc-12: iains/gcc-12-branch@fd5530b
gcc-13: iains/gcc-13-branch@cb7e4ec
gcc-14: iains/gcc-14-branch@ff62a10
That applied to Homebrew GCC (12.4.0):
Homebrew/homebrew-core@b904223#diff-89dd0b4176eca7fcc24b591943509bf8a8d6ea904d71e5dfcd6b78fed62fc574R44-R48
Ref: #13700
More info: #14091 (comment)
e91fcba #14155 macos: undo
availabilitymacro enabled by Homebrew gccApple SDK expects certain macros predefined by the compiler. Missing
them may causes odd issues. Mainline llvm is keeping up with Apple
clang, but it needs a fresh version, while the one installed on GitHub
runners is old (v15). I patched these in
lib/curl_setup.h.baa3270 #14134 build: fix llvm 16 or older + Xcode 15 or newer, and gcc
Without certain predefined macros, SDK headers can take a codepath where
it mis-defines its own
TARGET_OS_OSXmacro, which makes it break itsown headers later. I patched it in
lib/curl_setup.h.ff784af #14159 build: fix llvm 17 and older + macOS SDK 14.4 and newer
TargetConditionals.hrequiressys/types.hFixed in curl. It caused feature-detection failurs with autotools, and
could break builds in certain configurations.
e1f6192 #14130 configure: fix
SystemConfigurationdetectionFixed it by syncing compiler warning options.
59cadac #14128 build: sync warning options between autotools, cmake & compilers
Fixed it by improving detection of libidn2, with some more fixes
pending for the next feature window.
f43adc2 #14137 cmake: detect
libidn2also viapkg-configRef: #14136 cmake: detect
nghttp2viapkg-config, enable by defaultFixed the root cause and also the trigger in the CI config.
764fbab #14175 cmake: fix builds with detected libidn2 lib but undetected header
Fixed these warnings, which allowed to stop silencing them.
b05dc7e #14122 sectransp: fix
HAVE_BUILTIN_AVAILABLEchecks to not emit warnings5fa534b #14162 sectransp: fix clang compiler warnings, stop silencing them
d2ef625 #14182 cmake: sync CA bundle/path detection with autotools
Fixed by dropping unnecessary includes, makign test builds dependent
on dependency headers.
3765d75 #14172 cmake: fix building
unit1600due to missingssl/openssl.hcurl's CMake was missing bits for running the C preprocessor accurately.
It made tests 1119 and 1167 fail. I implemented the missing bits.
efc2c51 #14124 tests: include current directory when running test Perl commands
c09db8b #14129 cmake: create
configurehelp.pmlike autotools does67cc1e3 #14125 test1119: adapt for
.mdinput__builtin_available()supportcurl source code assumes this is available to enable certain codepaths.
It's also intermixed with monotonic timer support.
Detected by GCC, while it probably shouldn't be. llvm/clang detects it
depending on target OS version. I've been playing with this, but so far
without a conclusion or fix.
I couldn't find the reason for most of this. A bunch of RTSP tests fail
with GCC. SecureTransport + HTTP/2 is failing a bunch of tests. With
OpenSSL it fails two of those. SecureTransport builds also fail one DoH
test.
AppleIDN support received a fix with two more remaining.
fd02508 #14179 #14176 IDN: fix ß with AppleIDN
2c15aa5 GHA/macos: delete misplaced
CFLAGS, drop redundant CMake option80fb7c0 #14126 configure: limit
SystemConfigurationtest to non-c-ares, IPv6 buildscfd6f43 #14127 build: tidy up
__builtin_availablefeature checks (Apple)bae5553 #14174 runtests: show name and keywords for failed tests in summary
09cdf7e #14178 cmake: delete unused
HAVE_LIBSSH2,HAVE_LIBSOCKETmacrosd3595c7 #14186 configure: CA bundle/path detection fixes
58772b0 #14187 runtests: set
SOURCE_DATE_EPOCHto fix failing around midnight18f1cd7 #14183 tests: sync feature names with
curl -V4c22d97 #14181 build: use
#errorinstead of invalid syntaxPending merges:
-qs(quiet success) option #14185 runtests: fold test details for GitHub CI runs__builtin_availabletest to Darwin #14196 configure: limit__builtin_availabletest to DarwinSummary:
In general GCC doesn't seem to be a good fit with curl and macOS for
now. These "lucky" combinations (GitHub Actions runner) will build out
of the box now: macos-14 + Xcode 15.0.1 + gcc-11, gcc-12, gcc-14. The
rest builds with the ugly workaround in place, but all this still leaves
some runtime issues.
More info and links in the commit messages and source code.
[1]: This PR:
names and release years.
-Wno-deprecated-declarationsto suppress warnings.LDFLAGS=-wto suppress 'object file was built for newer macOSversion than being linked' warnings.
(there were tens of thousands of them in some jobs)
!instead ofno-andnon-.--enable-maintainer-mode./configureoption.results. (It's not yet clear why gcc-12 builds have different runtime
results than clang/llvm ones.)
e.g. 10.15 # Catalina (2019)
preinstalled packages and C compiler predefined macros for each job.
Useful for debugging all the strange problems these builds might have.
builds. Useful for debugging.
It supports building all possible compiler, runner image, Xcode/SDK
combinations, with cmake and autotools, target OS versions and with or
without SecureTransport. It's quick. GHA limits the maximum number of
matrix jobs at 256.
I used this as a test-rig to fix the macOS build fallouts with gcc and
llvm.
I settled with 16 jobs, trying to maximize fallout coverage.
each GHA images, with the defaults.
[2]: Homebrew GCC vs GHA runner images vs curl builds:
Source: https://github.com/curl/curl/actions/runs/9883956647/job/27299564218
Closes #14097