Skip to content

gh-90905: Allow cross-compilation on macOS#128385

Merged
ned-deily merged 3 commits intopython:mainfrom
zanieb:zb/cross-apple
Jan 13, 2025
Merged

gh-90905: Allow cross-compilation on macOS#128385
ned-deily merged 3 commits intopython:mainfrom
zanieb:zb/cross-apple

Conversation

@zanieb
Copy link
Contributor

@zanieb zanieb commented Dec 31, 2024 •
edited
Loading

Adds support for cross-compiling x86_64 from arm64 macOS. This is particularly useful due to the decreasing availability of x86_64 machines to perform builds on.

The changes here are loosely based on the patch that python-build-standalone uses for cross-compiling.

I tested compiling x86_64 from an arm64 machine with the following script:

#!/usr/bin/env bash

set -ex

ROOT=$(pwd)
OUT=${ROOT}/../cpython-out

# Assume that x86_64 Homebrew is installed
ibrew() { arch -x86_64 /usr/local/bin/brew "$@"; }
export -f ibrew

mkdir -p ${OUT}

# Compile the "build" aarch64 Python

make clean
rm -rf ${OUT}/cpython-aarch64

# Exclude modules we don't need that require system dependencies
./configure \
    py_cv_module__openssl=n/a \
    py_cv_module__hashlib=n/a \
    py_cv_module__gdbm=n/a \
    --without-ensurepip \
    --prefix ${OUT}/cpython-aarch64

make -j8
make -j sharedinstall
make -j install

# Build the "host" x86_64 Python

make clean
rm -rf ${OUT}/cpython-x86_64

CFLAGS="-arch x86_64" \
LDFLAGS="-arch x86_64" \
MACOSX_DEPLOYMENT_TARGET=10.15 \
GDBM_CFLAGS="-I$(ibrew --prefix gdbm)/include" \
GDBM_LIBS="-L$(ibrew --prefix gdbm)/lib -lgdbm" \
PKG_CONFIG="$(ibrew --prefix pkg-config)/bin/pkg-config" \
./configure \
    --with-pydebug \
    --with-system-libmpdec \
    --with-openssl="$(ibrew --prefix openssl@3)" \
    --build=aarch64-apple-darwin \
    --host=x86_64-apple-darwin \
    --prefix ${OUT}/cpython-x86_64 \
    --with-build-python=${OUT}/cpython-aarch64/bin/python3 \
    --disable-ipv6 \
    ac_cv_file__dev_ptc=no \
    ac_cv_file__dev_ptmx=no \
    ac_cv_func_sendfile=no

make -j8
make -j sharedinstall
make -j install

${OUT}/cpython-x86_64/bin/python3 -c "import platform; print(platform.machine())"

@bedevere-app bedevere-app bot mentioned this pull request Dec 31, 2024
Closed
@zanieb
Copy link
Contributor Author

zanieb commented Jan 1, 2025 •
edited
Loading

This seems a bit too easy. I'm curious if I'm missing something?

The justification for the configure flags are as follows:

--disable-ipv6

configure: error: You must get working getaddrinfo() function or pass the "--disable-ipv6" option to configure.

ac_cv_file__dev_ptmx=no

checking for /dev/ptmx... not set
configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling

ac_cv_file__dev_ptc=no

checking for /dev/ptc... not set
configure: error: set ac_cv_file__dev_ptc to yes/no in your CONFIG_SITE file when cross compiling

ac_cv_func_sendfile=no

./Modules/posixmodule.c:11871:15: error: call to undeclared function 'sendfile'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    ret = sendfile(in_fd, out_fd, offset, &sbytes, &sf, flags);
            ^

It's unclear to me if it's expected that the -arch needs to be added to CFLAGS and LDFLAGS explicitly.

I also checked the Platform in the sysconfig, as I know we have an extra patch for this in python-build-standalone but it doesn't look necessary here:

❯ ../cpython-out/cpython-x86_64/bin/python3 -m sysconfig | grep Platform
Platform: "macosx-10.15-x86_64"

@zanieb zanieb marked this pull request as ready for review January 1, 2025 04:59
@corona10 corona10 requested a review from a team January 1, 2025 09:11
@corona10
Copy link
Member

corona10 commented Jan 1, 2025

I think that @ned-deily and @ronaldoussoren have interests with this patch.

FFY00
FFY00 approved these changes Jan 10, 2025
View reviewed changes
Copy link
Member

@FFY00 FFY00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks correct to me.

I'll approve, but give @ned-deily some time to look it over before merging. @zanieb if this doesn't happen within a week, please ping me to move it forward. Thanks!

@ned-deily
Copy link
Member

ned-deily commented Jan 13, 2025

Thanks for the PR. The change looks OK so I'm merging it for 3.14.

But, IMO, I wouldn't recommend using this approach for most uses. There is a reason that we haven't bothered to support the "traditional" style of cross-building for macOS builds and that is because essentially the same result can be accomplished through the use of macOS universal builds, something that has been around in Python for many years and which is robust and well-tested. For example, the current universal2 flavor of universal builds produces fat builds including both arm64 and x86_64 architectures that can be built on either kind of Mac and will then run natively on both archs. The main differences are that the installed binary files are larger, likely negligible in most cases although the resultant binaries could be lipo-ed into single arch binaries, and it is necessary to build with universal builds of the third-party libraries used by the standard library that are not provided by macOS (i.e. openssl, xz, gdbm, tcl-tk, mpdecimal). AFAIK, Homebrew does not support universal builds of those libs; MacPorts does. We have had discussions about providing pre-built universal macOS and iOS binaries of at least some of these libs; that may yet happen in the coming months.

In the meantime, it might be a very useful interim experiment for someone to try to provide a script that lipo's and, as needed, install_name_tool's the Homebrew arm64 (/opt/homebrew) and x86_64 (/usr/local) versions of those few libs we need so that they could be cached and used in a GHA CI job to add macOS universal testing.

@ned-deily ned-deily merged commit 6ecb620 into python:main Jan 13, 2025
41 checks passed
@ned-deily
Copy link
Member

ned-deily commented Jan 13, 2025

As an example, a very simple universal build rough equivalent:

make clean || true

# If you need GDBM, uncomment the next two lines for MacPorts and/or edit.
# GDBM_CFLAGS="-I$(dirname $(dirname $(which port)))/include" \
# GDBM_LIBS="-L$(dirname $(dirname $(which port)))/lib -lgdbm" \
MACOSX_DEPLOYMENT_TARGET=10.15 \
./configure \
    --with-pydebug \
    --enable-universalsdk \
    --with-universal-archs=universal2 \
    --with-system-libmpdec \
    --prefix ${OUT}/cpython-x86_64 \

make -j8
make -j install

${OUT}/cpython-x86_64/bin/python3  -c "import platform; print(platform.machine())"
${OUT}/cpython-x86_64/bin/python3-intel64  -c "import platform; print(platform.machine())"

And, the one-time commands to build and install the universal libs from MacPorts:

#!/bin/sh

set -ex

# Assume that MacPorts base is installed and that "port" is on $PATH
#   (https://www.macports.org/install.php)

# Update ports info
sudo port selfupdate

# Install Python dependencies
sudo port -N install \
    pkgconfig \
    openssl +universal \
    xz +universal \
    gdbm +universal \
    mpdecimal +universal \
    # Tk disabled at the moment due to open MacPorts issue with univeral variant:
    # https://trac.macports.org/ticket/71415
    # tk +quartz +universal

@bedevere-bot
Copy link

bedevere-bot commented Jan 13, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.x has failed when building commit 6ecb620.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/345/builds/9923) and take a look at the build logs.
  4. Check if the failure is related to this commit (6ecb620) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/345/builds/9923

Failed tests:

  • test.test_multiprocessing_forkserver.test_misc

Failed subtests:

  • test_large_pool - test.test_multiprocessing_forkserver.test_misc.MiscTestCase.test_large_pool

Summary of the results of the build (if available):

==

Click to see traceback logs 
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 6607, in test_large_pool
    rc, out, err = script_helper.assert_python_ok(testfn)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 182, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 167, in _assert_python
    res.fail(cmd_line)
    ~~~~~~~~^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 80, in fail
    raise AssertionError(f"Process return code is {exitcode}\n"
    ...<10 lines>...
                         f"---")
AssertionError: Process return code is 1
command line: ['/root/buildarea/3.x.angelico-debian-amd64/build/python', '-X', 'faulthandler', '-I', '@test_3507502_tmpæ']


Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from multiprocessing.forkserver import main; main(10, 11, ['__main__'], **{'sys_path': ['/root/buildarea/3.x.angelico-debian-amd64/build/target/lib/python314.zip', '/root/buildarea/3.x.angelico-debian-amd64/build/Lib', '/root/buildarea/3.x.angelico-debian-amd64/build/build/lib.linux-x86_64-3.14'], 'authkey_r': 13})
                                                 ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 324, in main
    pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/build/test_python_3517661æ/@test_3517661_tmpæ", line 4, in <module>
    with multiprocessing.Pool(200) as p:
         ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
                context=self.get_context())
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 215, in __init__
    self._repopulate_pool()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
                                        self._processes,
                                        ^^^^^^^^^^^^^^^^
    ...<3 lines>...
                                        self._maxtasksperchild,
                                        ^^^^^^^^^^^^^^^^^^^^^^^
                                        self._wrap_exception)
                                        ^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 329, in _repopulate_pool_static
    w.start()
    ~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ~~~~~~~~~~~^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 300, in _Popen
    return Popen(process_obj)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
    ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 59, in _launch
    self.pid = forkserver.read_signed(self.sentinel)
               ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 390, in read_signed
    raise EOFError('unexpected EOF')
EOFError: unexpected EOF
---


Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 6607, in test_large_pool
    rc, out, err = script_helper.assert_python_ok(testfn)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 182, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 167, in _assert_python
    res.fail(cmd_line)
    ~~~~~~~~^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/script_helper.py", line 80, in fail
    raise AssertionError(f"Process return code is {exitcode}\n"
    ...<10 lines>...
                         f"---")
AssertionError: Process return code is 1
command line: ['/root/buildarea/3.x.angelico-debian-amd64/build/python', '-X', 'faulthandler', '-I', '@test_3517661_tmpæ']


Traceback (most recent call last):
  File "<string>", line 1, in <module>
    from multiprocessing.forkserver import main; main(10, 11, ['__main__'], **{'sys_path': ['/root/buildarea/3.x.angelico-debian-amd64/build/target/lib/python314.zip', '/root/buildarea/3.x.angelico-debian-amd64/build/Lib', '/root/buildarea/3.x.angelico-debian-amd64/build/build/lib.linux-x86_64-3.14'], 'authkey_r': 13})
                                                 ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 324, in main
    pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/build/test_python_3507502æ/@test_3507502_tmpæ", line 4, in <module>
    with multiprocessing.Pool(200) as p:
         ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
                context=self.get_context())
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 215, in __init__
    self._repopulate_pool()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
                                        self._processes,
                                        ^^^^^^^^^^^^^^^^
    ...<3 lines>...
                                        self._maxtasksperchild,
                                        ^^^^^^^^^^^^^^^^^^^^^^^
                                        self._wrap_exception)
                                        ^^^^^^^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/pool.py", line 329, in _repopulate_pool_static
    w.start()
    ~~~~~~~^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ~~~~~~~~~~~^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/context.py", line 300, in _Popen
    return Popen(process_obj)
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
    ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/popen_forkserver.py", line 59, in _launch
    self.pid = forkserver.read_signed(self.sentinel)
               ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/multiprocessing/forkserver.py", line 390, in read_signed
    raise EOFError('unexpected EOF')
EOFError: unexpected EOF
---

@zanieb
Copy link
Contributor Author

zanieb commented Jan 13, 2025

Thanks for the explanation @ned-deily — there is some discussion on universal builds downstream astral-sh/python-build-standalone#140

@colincornaby colincornaby mentioned this pull request Sep 21, 2025
Open
@Andrej730
Copy link
Contributor

Andrej730 commented Sep 25, 2025

So #90905 can be closed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FFY00 FFY00 FFY00 approved these changes

@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland is a code owner

@corona10 corona10 Awaiting requested review from corona10 corona10 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zanieb @corona10 @ned-deily @bedevere-bot @Andrej730 @FFY00