Skip to content

Fixes 2757: Add win/arm64 launchers to create arm64 executables#2758

Merged
jaraco merged 9 commits intopypa:mainfrom
niyas-sait:win_arm64_launchers
Oct 2, 2021
Merged

Fixes 2757: Add win/arm64 launchers to create arm64 executables#2758
jaraco merged 9 commits intopypa:mainfrom
niyas-sait:win_arm64_launchers

Conversation

@niyas-sait
Copy link
Contributor

Summary of changes

This change adds arm64 windows launchers executables generated from launcher.c for arm64 target and related changes to pick right launcher based on platform

Pull Request Checklist

  • Changes have tests
  • News fragment added in [changelog.d/].

@niyas-sait
Copy link
Contributor Author

@jaraco Can you please review this change?

This will be required to support cross compilation
Comment on lines 38 to 47
REM now for arm 64-bit
REM Cross compile for arm64
call VCVARSx86_arm64
if "%ERRORLEVEL%"=="0" (
cl /D "GUI=0" /D "WIN32_LEAN_AND_MEAN" launcher.c /O2 /link /MACHINE:arm64 /SUBSYSTEM:CONSOLE /out:setuptools/cli-arm64.exe
cl /D "GUI=1" /D "WIN32_LEAN_AND_MEAN" launcher.c /O2 /link /MACHINE:arm64 /SUBSYSTEM:WINDOWS /out:setuptools/gui-arm64.exe
) else (
echo Windows SDK 6.1 not found to build Windows 64-bit version
)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer if I could build the launcher. Do you know how I could acquire a Windows ARM machine (perhaps through Azure) that would build the executable? Did you have to do anything special to get the build to run?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think executables can be cross-compiled from an x64 machine as well. But for execution, you need to get one of the windows on arm machines. I have tested on a surface pro x machine. I don't think we have any machines available yet azure yet.

The launcher works out of the box, I just had to compile it with the right target machine.

Copy link
Member

@jaraco jaraco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm generally in favor of this change, even though it's adding functionality against a deprecated feature (easy_install wrapper scripts). Do you know what is the status of wrapper scripts in pip for arm on Windows executables? If pip/distutils already supports these, I'll be slightly inclined to leave them broken for easy_install, which is deprecated and to direct users to use pip to install and generate wrapper executables for ARM platforms.

@niyas-sait
Copy link
Contributor Author

I'm generally in favor of this change, even though it's adding functionality against a deprecated feature (easy_install wrapper scripts). Do you know what is the status of wrapper scripts in pip for arm on Windows executables? If pip/distutils already supports these, I'll be slightly inclined to leave them broken for easy_install, which is deprecated and to direct users to use pip to install and generate wrapper executables for ARM platforms.

Thanks, @jaraco . pip/distutils is also broken for the time being for woa. I have raised a PR for distutils and after merging and a distlib release the pip needs to be upgraded as well.

I think it will be helpful to have support in easy_install for woa as it might take a while to get woa launcher script support in pip.

@rubin55
Copy link

rubin55 commented Sep 19, 2021

Extra voice adding support for fixing this in setuptools, even though easy_install and friends are ye-olde deprecated, it will make working on ARM64 Windows that much more relaxed and nice when dealing with older packages or for some reason or the other being forced to use an older method of installing. Also, it would harmonize the experience of setuptools between the two (three) architectures.

@nsait-linaro: Do you have a link to the distutils PR and possibly a pip issue number with regards to launcher support for pip? I couldn't find it.

@niyas-sait
Copy link
Contributor Author

@rubin55 Here is the PR for distlib which seems to be the windows launcher used by pip.

And please check pip issue 10489

@niyas-sait
Copy link
Contributor Author

@jaraco Could you approve the pending workflow and merge the changes if you are happy ?

@jaraco
Copy link
Member

jaraco commented Sep 25, 2021

This change looks good to me, but I need some sort of verification that the changes work as intended... and preferably a verification that the compiled binaries were generated from the supplied sources. To do that, I think I need a machine with a compiler where I can compile the binaries myself. I'll try to acquire one using my MSDN subscription.

@niyas-sait
Copy link
Contributor Author

Re: workflow failure seems to be due to a network timeout error. I don't see how it is related to my patch so I guess it is a generic issue.

_____________ ERROR collecting setuptools/tests/test_virtualenv.py _____________

setuptools/tests/test_virtualenv.py:99: in <module>
    ???
setuptools/tests/test_virtualenv.py:64: in _get_pip_versions
    urlopen('https://pypi.org', timeout=1)
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/urllib/request.py:216: in urlopen
    return opener.open(url, data, timeout)
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/urllib/request.py:519: in open
    response = self._open(req, data)
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/urllib/request.py:536: in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/urllib/request.py:496: in _call_chain
    result = func(*args)
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/urllib/request.py:1391: in https_open
    return self.do_open(http.client.HTTPSConnection, req,
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/urllib/request.py:1352: in do_open
    r = h.getresponse()
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/http/client.py:1368: in getresponse
    response.begin()
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/http/client.py:317: in begin
    version, status, reason = self._read_status()
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/http/client.py:278: in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/socket.py:705: in readinto
    return self._sock.recv_into(b)
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/ssl.py:1273: in recv_into
    return self.read(nbytes, buffer)
/opt/hostedtoolcache/Python/3.10.0-rc.2/x64/lib/python3.10/ssl.py:1129: in read
    return self._sslobj.read(len, buffer)
E   TimeoutError: The read operation timed out

@niyas-sait
Copy link
Contributor Author

This change looks good to me, but I need some sort of verification that the changes work as intended... and preferably a verification that the compiled binaries were generated from the supplied sources. To do that, I think I need a machine with a compiler where I can compile the binaries myself. I'll try to acquire one using my MSDN subscription.

Thanks, @jaraco. Binaries for ARM64 can be cross-compiled from x86 or x64 windows machines. If you prefer you can amend the patch with the new binaries you create locally. I don't think there is any cloud VM available for woa yet so if you update the patch I can test new patch on my win/arm64 machine.

@jaraco
Copy link
Member

jaraco commented Sep 28, 2021

Oh, I hadn't realized the binaries could be cross-compiled. That's great.

Unfortunately, the current msvc_build_launcher.cmd is not working for me, using a recent install of Visual Studio 2019.

The link in that script takes me to a 404 page.

Did you use the script to build the results? If so, what did you do to set it up?

@jaraco
Copy link
Member

jaraco commented Sep 28, 2021

I have been using this Windows Docker container.

@niyas-sait
Copy link
Contributor Author

Thanks, @jaraco for trying it out. I couldn't install SDK 6.1 as well. I had to use VS 2019 for arm64 launchers. I've added a new script msvc-build-launcher-arm64.cmd to compile arm64 binaries using VS 2019. Could you give it a try with that ?

@jaraco
Copy link
Member

jaraco commented Oct 1, 2021

That's helpful, I think. I did try with the new script, but it also doesn't find visual studio as I've installed it:

PS C:\Program Files (x86)\Microsoft Visual Studio\2019> dir


    Directory: C:\Program Files (x86)\Microsoft Visual Studio\2019


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        7/13/2021  10:56 AM                BuildTools
d-----        7/13/2021  10:55 AM                TestAgent

It seems I only have MSBuild installed and not Visual Studio. I'm going to have to do more experimentation to determine what components are needed to provide the necessary tools.

Aha. I see now that the Docker image install MSVC into \Buildtools:

PS C:\> . '/Program Files (x86)/Microsoft Visual Studio/installer/vswhere' -latest -products * -property installationPath
C:\BuildTools

I probably installed into that directory because I struggled to install into a directory with spaces due to escaping challenges.

So probably, the script needs to invoke vswhere in order to locate where Visual Studio is installed. It probably should also indicate the required components to include the ARM compiler.

I'd recommend to get away from using a .cmd file. It's a very old syntax with a lot of legacy. I'd recommend to use Python or Powershell to write the script. I don't know if that's possible, though, with the use of "VCVARS*", which uses .bat files. Maybe the setuptools.msvc module provides some guidance on how to locate and invoke VC from Python.

Of course, all of this effort is moot if I can just get a pair of verified binaries. I'm starting to care less about getting a repeatable process as this is all a big mess and deprecated.

I started work on getting ARM support into my Docker image, but the build is failing, and I'm out of time to work on this today. I'll look into this more later.

@jaraco jaraco force-pushed the win_arm64_launchers branch from dba3dab to cbbdb86 Compare October 2, 2021 02:18
@jaraco jaraco merged commit 8519d4e into pypa:main Oct 2, 2021
@niyas-sait
Copy link
Contributor Author

Thanks, @jaraco. I've tested v58.2.0 and it all seems to work as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants