Skip to content

Some libc++abi symbols are missing with LLVM12+'s stdenv on Darwin #166205

@rrbutani

Description

@rrbutani

Problem

When using stdenvs (or even just libc++ versions, really) from llvmPackages corresponding to LLVM 12 and newer on macOS, symbols related to exceptions (such as __cxa_allocate_exception) are not found by the linker.

This can be reproduced by compiling and linking any C++ program that makes use of exceptions, i.e.:

clang++ -xc++ - <<<"int main() { throw 0; return 5; }" -o /tmp/foo

Results in:

Undefined symbols for architecture arm64:
  "___cxa_allocate_exception", referenced from:
      _main in --bbc70b.o
  "___cxa_throw", referenced from:
      _main in --bbc70b.o
ld: symbol(s) not found for architecture arm64
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

Comparing the libc++ dylibs used by the wrapped clang++ in the LLVM 12 and 13 stdenvs (i.e. llvmPackages_12.libcxxStdenv) to that used in the LLVM 11 stdenv reveals that while LLVM 11's libc++.dylib reexports such symbols, newer libc++.dylibs do not.

When run through llvm-nm -C and grepped for __cxa_allocate_exception, here is the output for LLVM 11's libc++ dylib:

                 U ___cxa_allocate_exception
                 I ___cxa_allocate_exception (indirect for ___cxa_allocate_exception)

And LLVM 13's:

                 U ___cxa_allocate_exception

Both of these libc++ dylibs have a dynamic dependency on libc++abi which actually provides a definition for this symbol (this can be observed with otool -L <dylib path> which shows the libc++abi.dylib dep and then llvm-nm -C <libc++abi.dylib path> which shows that the symbol is defined in the libc++abi dylib) however only libc++ versions prior to LLVM 12's reexport the symbol.

This is consistent with the libc++ source code. These symbols were initially removed from the main list of symbols reexported and gated on libc++ exception support. However later this logic and the special list of symbols was removed.

Open Questions

I am not certain I understand the changes in the commit linked above; it's not clear to me why libcxxabi was changed to reexport these symbols.

This seems like an ABI break but there is no mention of this being potentially problematic on the CL and the comments added to the changelog in this commit seem to insist that it only adds some reexports to libc++.

Potential Solutions

I am not sure what the "right" solution is but anecdotally it seems like it's common to pass in -lc++abi as a linkopt on macOS.

cc-wrapper already does this on Linux.

To Reproduce

Here's a flake:

{
  inputs = {
    # nixpkgs.url = github:nixos/nixpkgs?ref=1a8754caf8eb8831b904daaf34b2ee737053b383; # commit for the 13.0.1 LLVM bump
    nixpkgs.url = github:nixos/nixpkgs/nixpkgs-unstable; # LLVM 13 is broken on Darwin on 21.11; using < instead of ^ so we get caching
    flake-utils.url = github:numtide/flake-utils;
  };

  # The issue only affects Darwin. Both because symbol rexport is only a thing
  # on mach-o platforms
  # (https://github.com/llvm/llvm-project/blob/4f13b999297140486b2faa1b5d8d7c768fb40dfb/libcxx/src/CMakeLists.txt#L202-L215)
  # and because `-lc++abi` is always added to the cc wrapper for Linux:
  # https://github.com/NixOS/nixpkgs/blob/a785ec661fadba2ded58467f951b51c34631c269/pkgs/build-support/cc-wrapper/default.nix#L382
  outputs = { self, nixpkgs, flake-utils }: flake-utils.lib.eachSystem [ "aarch64-darwin" "x86_64-darwin" ] (system:
    let
      normal = import nixpkgs { inherit system; };
      imp = func: import nixpkgs { inherit system; config.replaceStdenv = { pkgs }: (func pkgs).libcxxStdenv; };
      l_11 = imp (n: n.llvmPackages_11);
      l_12 = imp (n: n.llvmPackages_12);
      l_13 = imp (n: n.llvmPackages_13);
      l_13' = import nixpkgs {
        inherit system;
        # config.replaceStdenv = { pkgs }: pkgs.llvmPackages_13.libcxxStdenv.overrideAttrs (old: {
        #   postFixup = old.postFixup + ''
        #     echo "-lc++abi" >> $out/nix-support/libcxx-ldflags
        #   '';
        # });
        config.replaceStdenv = { pkgs }:
          let
            clang' = pkgs.llvmPackages_13.libcxxClang.overrideAttrs (old: {
              # We _should_ use an overlay or something similar here but they allegedly don't interact well with `replaceStdenv`.
              #
              # We're essentially adding this: https://github.com/NixOS/nixpkgs/blob/3491c5ea290bca5437845b6348919fcb23950af9/pkgs/build-support/cc-wrapper/default.nix#L382
              postFixup = old.postFixup + ''
                echo "-lc++abi" >> $out/nix-support/libcxx-ldflags
              '';
            });

            # https://github.com/NixOS/nixpkgs/blob/904ed45698338365f086c22ab5af167adf8bee9a/pkgs/development/compilers/llvm/13/default.nix#L241
            stdenv' = pkgs.overrideCC pkgs.stdenv clang';
          in
            stdenv';
      };

      cmd = ''clang++ -xc++ - <<<"int main() { throw 4; return 0; }" -v'';
      test = np: np.runCommandCC "test-cxx" {} ''
        ${cmd} -o $out
      '';
      hook = ''
        cd "$(mktemp -d)"
        libcxx_dir="$(${cmd} -o out -Wl,-v |& tee /dev/stderr | grep -P '\t/nix/store/.*-libcxx-.*/lib' | tail -1)"
        libcxx=$(echo "$libcxx_dir/libc++.dylib" | xargs)

        echo -e "\nlibcxx: $libcxx"
        echo "contains:"
        ${normal.llvmPackages.llvm}/bin/llvm-nm -C "$libcxx" | grep __cxa_allocate_exception
      '';
    in {

    # `nix flake check` shows the error
    checks.l11 = test l_11; # Works fine
    checks.l12 = test l_12; # Fails
    checks.l13 = test l_13; # Fails
    checks.l13' = test l_13'; # A workaround

    devShells = {
      llvm11 = l_11.mkShell { shellHook = hook; };
      llvm12 = l_12.mkShell { shellHook = hook; };
      llvm13 = l_13.mkShell { shellHook = hook; };
    };
  });
}

Running nix flake check shows the error when compiling with the LLVM 12 and 13 stdenvs; entering the dev shells with nix develop prints the path of the libc++ dylib that's used and greps it for __cxa_allocate_exception (one of the missing symbols).

The above also contains an example of a workaround that adds -lc++abi to the wrapper used in the LLVM 13 stdenv.

I think this is reproducible with any nixpkgs version after the LLVM 12.0.0 package was added but just in case, here's my flake.lock file:

Click to expand
{
  "nodes": {
    "flake-utils": {
      "locked": {
        "lastModified": 1648297722,
        "narHash": "sha256-W+qlPsiZd8F3XkzXOzAoR+mpFqzm3ekQkJNa+PIh1BQ=",
        "owner": "numtide",
        "repo": "flake-utils",
        "rev": "0f8662f1319ad6abf89b3380dd2722369fc51ade",
        "type": "github"
      },
      "original": {
        "owner": "numtide",
        "repo": "flake-utils",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1648219316,
        "narHash": "sha256-Ctij+dOi0ZZIfX5eMhgwugfvB+WZSrvVNAyAuANOsnQ=",
        "owner": "nixos",
        "repo": "nixpkgs",
        "rev": "30d3d79b7d3607d56546dd2a6b49e156ba0ec634",
        "type": "github"
      },
      "original": {
        "owner": "nixos",
        "ref": "nixpkgs-unstable",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "root": {
      "inputs": {
        "flake-utils": "flake-utils",
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 7
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    0.kind: bugSomething is broken6.topic: darwinRunning or building packages on Darwin6.topic: llvm/clangIssues related to llvmPackages, clangStdenv and related

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions