darwin, python: nix-provided pip has trouble building some compiled packages in a virtualenv from within nix-shell #26876
Description
Issue description
Using a setup similar to that suggested in https://nixos.org/nixpkgs/manual/#how-to-consume-python-modules-using-pip-in-a-virtualenv-like-i-am-used-to-on-other-operating-systems works fine under Nix/Linux, but under darwin has trouble building some packages.
Steps to reproduce
One case is the pip package watchdog==0.8.3. A nix file such as:
{
pkgs ? import <nixpkgs> {},
pythonPackages ? pkgs.python27Packages
}:
{
testEnv = pkgs.stdenv.mkDerivation {
name = "test-env";
buildInputs = [
pythonPackages.virtualenv
pythonPackages.pip
pythonPackages.argh
pythonPackages.pathtools
pythonPackages.pyyaml
pkgs.stdenv
pkgs.darwin.apple_sdk.frameworks.CoreServices
pkgs.darwin.cf-private
];
hardeningDisable = [ "format" ];
shellHook = ''
VENV_DIR="venv${pythonPackages.python.pythonVersion}"
if [ ! -e $VENV_DIR ]; then
${pythonPackages.virtualenv}/bin/virtualenv --no-setuptools $VENV_DIR
fi
source $VENV_DIR/bin/activate
SOURCE_DATE_EPOCH=315532800 $VENV_DIR/bin/pip install --no-binary :all: watchdog==0.8.3
'';
};
}
on performing nix-shell . results in the error:
src/watchdog_fsevents.c:562:29: error: use of undeclared identifier 'kCFFileDescriptorReadCallBack'
kCFFileDescriptorReadCallBack);
^
src/watchdog_fsevents.c:565:29: error: use of undeclared identifier 'kCFFileDescriptorWriteCallBack'
kCFFileDescriptorWriteCallBack);
^
1864 warnings and 2 errors generated.
error: command 'clang' failed with exit status 1
----------------------------------------
Command "/Users/xxxxxx/git/yyyyyy/venv2.7/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/fg/vl0n7j2d0d9fcn__37bq6g5m000132/T/pip-build-9iIbG_/watchdog/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/fg/vl0n7j2d0d9fcn__37bq6g5m000132/T/pip-slKrxk-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/xxxxxx/git/yyyyyy/venv2.7/include/site/python2.7/watchdog" failed with error code 1 in /private/var/folders/fg/vl0n7j2d0d9fcn__37bq6g5m000132/T/pip-build-9iIbG_/watchdog/
Noted that this package supposedly requires access to macos system libraries, thus the inclusion of the lines pkgs.darwin.apple_sdk.frameworks.CoreServices and pkgs.darwin.cf-private (which I cribbed from the regular nix package for watchdog 0.8.3). But the weird thing is that those lines are the only "special" thing about the nix package definition - and the nix package compiles fine.
So I find that strange because I would have thought the build environment provided to the nix package would be very similar to that provided to pip in the nix-shell environment.
There are a couple of packages that have had a problem like this but most of the others I've been able to "fix" by using newer versions, so they don't really make good examples. But they're all examples that "work fine" on linux.
Technical details
- System: (MacOS 10.11.6)
- Nix version: 1.11.11
- Nixpkgs version: 17.03.1387.25a8ba5baf
- Sandboxing enabled: whatever's default on a straightforward darwin install of nix.