Skip to content

Conversation

@pbo-linaro
Copy link
Contributor

@pbo-linaro pbo-linaro commented Mar 6, 2023

Uses arm64 artifacts now available for dart/flutter. Most of the work is to define windows-arm64 as an host and target platforms, and modify some hardcoded windows-x64 variants.

The Flutter tool does not support cross-compilation as of this change:

  • On x64, you get an x64 app
  • On arm64, you get an arm64 app
  • On an arm64 machine emulating x64, you get an x64 app

The system bitness is identified thanks to the PROCESSOR_ARCHITECTURE environment variable, which is the idiomatic way to detect this.

This was successfully tested on flutter-gallery example app, after updating CMakeLists.txt files (from examples/) for this. Compiled debug, release and profil variants.

Solves native compilation for #62597 (not yet cross-compilation) and addresses #116196
This PR is an update of #113928

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

Uses arm64 artifacts now available for dart/flutter.
Most of the work is to define windows-arm64 as an host and target
platforms, and modify some hardcoded windows-x64 variants.

Cross compilation is not available, only native one.

On windows-arm64, it will produce an x64 or arm64 app depending on
PROCESSOR_ARCHITECTURE environment variable value.

This was successfully tested on flutter-gallery example app, after
updating CMakeLists.txt files (from examples/) for this. Compiled debug,
release and profil variants.
@flutter-dashboard flutter-dashboard bot added d: api docs Issues with https://api.flutter.dev/ d: examples Sample code and demos documentation c: contributor-productivity Team-specific productivity, code health, technical debt. tool Affects the "flutter" command-line tool. See also t: labels. labels Mar 6, 2023
@pbo-linaro
Copy link
Contributor Author

@loic-sharma Before fixing/adding tests, and or nit, I would like to discuss the content of this PR, and see if it would be approved later.

@stuartmorgan Your review is welcome on this, as the previous PR was not a good approach. I hope this one will be more satisfying.

My plan would be to clean/merge this, and then work on proper cross-compilation support (by adding a --target-platform option, like it's done for Linux). Having native only seems easier, and allows to define all host/target stuff, which is most of the work.

@stuartmorgan-g
Copy link
Contributor

Cross compilation is not available, only native one.

On windows-arm64, it will produce an x64 or arm64 app depending on PROCESSOR_ARCHITECTURE environment variable value.

These two sentences seem to directly contradict each other. Can you clarify the scope of this PR?

@pbo-linaro
Copy link
Contributor Author

As you may now, windows-arm64 (starting with windows 11) emulates x64 binaries. Thus, the second sentence means there is still a way to get an x64 flutter app, even on windows-arm64 machine.

However it's not cross compilation, but native compilation (with an emulated toolchain). If that's not clear, I can stilll remove this from PR description, don't want this to be confusing.

In summary, the scope is native compilation. On x64, you get an x64 app. On arm64, an arm64 one.

@loic-sharma
Copy link
Member

Thanks for the contribution! I'll take a closer look at the approach this afternoon. FYI we will need buy-in from the Flutter tool team as well, I'll make sure to loop them in when we're ready!

Thus, the second sentence means there is still a way to get an x64 flutter app, even on windows-arm64 machine ... However it's not cross compilation, but native compilation (with an emulated toolchain)

What are the steps to force x64 emulation?

I like the phrasing in your last comment, consider using the same phrasing in your pull request description. Perhaps something like:

The Flutter tool does not support cross-compilation as of this change:

  • On x64, you get an x64 app
  • On arm64, you get an arm64 app
  • On an arm64 machine emulating x64, you get an x64 app

@pbo-linaro
Copy link
Contributor Author

pbo-linaro commented Mar 7, 2023

I just updated the description, it's much clearer this way 👍 .

To force x64 emulation, there is some corner cases. I'm not very delighted by the way Windows handle this, but it is the way it is. Emulation is transparent (windows process loader handles this), however, setting the desired bitness is much more tricky.

From what we observed, an x64 process will always have PROCESSOR_ARCHITECTURE set to AMD64, and an arm64 process, ARM64 value. Even if you setenv(PROCESSOR_ARCHITECTURE, ...), it will be overridden when process is created. So far, so good.

There comes one specificity: cmd/powershell. For those "shell" processes, and only them, they seem to inherit the initial bitness of the parent process. So even, if you manually set PROCESSOR_ARCHITECTURE in a .bat script, this value won't be propagated to new children processes. I think it was implemented because their bitness is always arm64, so it would result in PROCESSOR_ARCHITECTURE always being set to ARM64. But it's very confusing... and nothing is documented.


Since Flutter bootstrap itself using a bat script, we are impacted by this. So the only way to force x64 "emulation" in this case, is to launch flutter.bat from an arm64 or x64 parent process. In my case, I used python (x64 or arm64) to be able to change bitness. All this is very implicit and cumbersome alas.

Once the initial dart is downloaded, we are stuck with one bitness as it won't be re-downloaded, and it's PROCESSOR_ARCHITECTURE value will be set from its bitness.


If you survived until the end of this long answer, IMHO, a simple solution to this could be to create a new wrapper flutter_arm64.bat, which would download explicit arm64 version using powershell script, and would launch an error on windows-x64.

But considering we'll introduce proper cross-compilation later (x64 -> arm64 and arm64 -> x64), I think it's not worth doing this.

@override
List<List<String>> getBinaryDirs() {
// Currently only Linux supports both arm64 and x64.
// Linux and Windows both support arm64 and x64.
Copy link
Member

@loic-sharma loic-sharma Mar 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cbracken Doesn't macOS support ARM64 too with Apple silicon? Lines 227 and 232 below manually specify x64 instead of using $arch, is that intended? If so, perhaps we should add a quick comment here explaining the situation.

${FLUTTER_TOOL_ENVIRONMENT}
"${FLUTTER_ROOT}/packages/flutter_tools/bin/tool_backend.bat"
windows-x64 $<CONFIG>
${FLUTTER_TARGET_PLATFORM} $<CONFIG>
Copy link
Member

@loic-sharma loic-sharma Mar 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will want to create a migration to automate this for our users. Here's the corresponding Linux migration:

// CMake's add_custom_command() should add FLUTTER_TARGET_PLATFORM to support multi-architecture.
// However, developers would get the following warning every time if we do nothing.
// ------------------------------
// CMake Warning:
// Manually-specified variables were not used by the project:
// FLUTTER_TARGET_PLATFORM
// ------------------------------
if (addCustomCommandOriginal?.contains('linux-x64') ?? false) {
newProjectContents = newProjectContents.replaceAll('linux-x64', r'${FLUTTER_TARGET_PLATFORM}');
}

This is kicked off here:

final List<ProjectMigrator> migrators = <ProjectMigrator>[
CmakeCustomCommandMigration(windowsProject, globals.logger),
];
final ProjectMigration migration = ProjectMigration(migrators);
migration.run();

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, I didn't see there was this tool. The related change in gallery example application change mention they use flutter create instead, and updated files from there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries. I didn't know about this either until very recently 😆

'-G',
generator,
'-A',
getCmakeWindowsArch(targetPlatform),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gives the following error if you've ran or built your Windows app before this change:

CMake Error: Error: generator platform: x64
Does not match the platform used previously:
Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.
Exception: Unable to generate build files

Users will need to run flutter clean to work around this. This will affect all Windows developers once when they upgrade their Flutter SDK. This will also affect Windows developers each time they provide a different --target-platform value. This seems too impactful.

Is the -A option strictly necessary? What is it used for? Can we work around this somehow?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you're right, the actual build layout for windows does not reflect architecture, so that would need to be changed.
Another solution is to clean the folder when a change in target-platform is detected, so that would be less impactful.

Maybe that can wait the second round where we implement --target-platform?
For now, people would be stuck with arm64 or x64 build after flutter bootstrapped itself.

the -A is needed as it's the one who dictates if you create an arm64 or x64 objects, that will be linked with flutter.dll in the end.

@loic-sharma
Copy link
Member

a simple solution to this could be to create a new wrapper flutter_arm64.bat, ..., I think it's not worth doing this.

Agreed. Thank you for the thorough explanation!

Have you considered how cross-platform builds will be implemented? How will CMake generation be affected? What will the build directory look like? I ask because I wonder if we should update the build directory layout first before adding support for native compilation.

Otherwise, your changes seem reasonable. I'm still reviewing but I'll start looping in other folks. Thank you for all the excellent work!

@driver1998
Copy link

driver1998 commented Mar 8, 2023

What are the steps to force x64 emulation?

Run your shell as x64 and start from here.

Note that both Windows Powershell and CMD are ARM64X hybrid binaries, kind of like (but not 100% equal to) Universal binaries on macOS.

The Modern PowerShell (7.0+) are not hybrid binaries and come with separate x64 and arm64 versions.

ARM64X binaries will run as ARM64 if you start them from ARM64 processes (e.g: the shell), or x64 if you start from x64 processes.

If you want to force these as x64, run them like this in CMD:

start /machine amd64 cmd.exe

This command is added in Windows 11 22H2.

With 21H2, you'll need to build a special launcher like this:

#define _WIN32_WINNT 0x0A00      // _WIN32_WINNT_WIN10
#define NTDDI_VERSION 0x0A00000B // NTDDI_WIN10_CO

#include <assert.h>
#include <stdlib.h>
#include <string.h>
#include <windows.h>

int main(void) {
  char cmdLine[32767];

  const char *app = "cmd.exe";
  memcpy(cmdLine, app, strlen(app) + 1);

  STARTUPINFOEX si = {0};
  si.StartupInfo.cb = sizeof(STARTUPINFOEX);

  BOOL bRet;
  size_t attrListSize = 0;
  InitializeProcThreadAttributeList(NULL, 1, 0, &attrListSize);
  assert(attrListSize > 0);

  si.lpAttributeList = malloc(attrListSize);
  bRet = InitializeProcThreadAttributeList(si.lpAttributeList, 1, 0,
                                           &attrListSize);
  assert(bRet);

  // Specify the target architecture here
  WORD processMachineType = IMAGE_FILE_MACHINE_AMD64;
  bRet = UpdateProcThreadAttribute(
      si.lpAttributeList, 0, PROC_THREAD_ATTRIBUTE_MACHINE_TYPE,
      (void *)&processMachineType, sizeof(processMachineType), NULL, NULL);
  assert(bRet);

  PROCESS_INFORMATION pi = {0};
  bRet = CreateProcess(NULL, cmdLine, NULL, NULL, FALSE,
                       EXTENDED_STARTUPINFO_PRESENT | CREATE_NEW_CONSOLE, NULL,
                       NULL, (STARTUPINFO *)&si, &pi);

  assert(bRet);

  DeleteProcThreadAttributeList(si.lpAttributeList);
  free(si.lpAttributeList);

  CloseHandle(pi.hProcess);
  CloseHandle(pi.hThread);

  return 0;
}

... or just build your launcher as x64.

@pbo-linaro
Copy link
Contributor Author

Thanks soooo much @driver1998, this is exactly the information I was looking for.

All the machines I use are running 22h2, so start /machine is accessible 👍 . For people looking for a way to launch a process on 21H2 without having to compile your own launcher, you can simply use a python script, and execute it without python-x86, python-x64 or python-arm64 depending on your need.

Out of curiosity: From where did you get it? Do you work for MSFT? Any "official" documentation or link to provide (exception for start manual)?
I'm really looking for places (blog, changelog, git repo) where this kind of thing can be found.

@pbo-linaro
Copy link
Contributor Author

a simple solution to this could be to create a new wrapper flutter_arm64.bat, ..., I think it's not worth doing this.

Agreed. Thank you for the thorough explanation!

Have you considered how cross-platform builds will be implemented? How will CMake generation be affected? What will the build directory look like? I ask because I wonder if we should update the build directory layout first before adding support for native compilation.

To be honest, I would prefer to go straight to implement --target-platform, and solve all related issues (build path, etc). The thing is that I expect this to take a long time to review and argue, as in the last PR, it was mentioned that introducing an option would require a design document, and all those kind of things.

So, before further work, would that be possible to agree if we should do it in one or two steps (native, and then cross-compilation with --target-platform)?
If we do it in one step, can I get the exact list of requirements and documents you or any other flutter team member need to give its approval?
The thing I want to avoid, is spending several weeks to implement something/fix nits/add tests, and have a new member of the team coming to say "I don't agree". Hope that's understandable.

@driver1998
Copy link

driver1998 commented Mar 8, 2023

PROC_THREAD_ATTRIBUTE_MACHINE_TYPE is documented here: https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-updateprocthreadattribute

But as you can see how to use it exactly is not very appearant from the docs so I provided a sample code.

And no, I don't work for MS...

@pbo-linaro
Copy link
Contributor Author

PROC_THREAD_ATTRIBUTE_MACHINE_TYPE is documented here: https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-updateprocthreadattribute

But as you can see how to use it exactly is not very appearant from the docs so I provided a sample code.

Instead of the code part, I meant more about the fact cmd.exe is an ARM64X binary, and the semantic of PROCESSOR_ARCHITECTURE being derived from the parent process. Which does not seem to be documented anywhere, but maybe I missed something.

@driver1998
Copy link

I afraid the behavior of ARM64X executables (not DLLs) are not officially documented, since it is basically for internal use in order to share the same system32 directory.
These stuff are mostly gathered from trial-and-error.

@pbo-linaro
Copy link
Contributor Author

pbo-linaro commented Mar 8, 2023

I afraid the behavior of ARM64X executables (not DLLs) are not officially documented, since it is basically for internal use in order to share the same system32 directory. These stuff are mostly gathered from trial-and-error.

Thanks again for taking the time to share this here.

If someone needs it as a reference, I summarized information about PROCESSOR_ARCHITECTURE on this link:
https://linaro.atlassian.net/wiki/spaces/WOAR/pages/28869623809/Identify+CPU+architecture

@stuartmorgan-g
Copy link
Contributor

Have you considered how cross-platform builds will be implemented? How will CMake generation be affected? What will the build directory look like? I ask because I wonder if we should update the build directory layout first before adding support for native compilation.

To be honest, I would prefer to go straight to implement --target-platform, and solve all related issues (build path, etc). The thing is that I expect this to take a long time to review and argue, as in the last PR, it was mentioned that introducing an option would require a design document, and all those kind of things.

This is exactly the kind of thing that the design document would need to address, so given that we need to answer the questions at this stage, we should do the design document now.

Since there have been several comparisons to the Linux work above, I should point out that the reason we need a new design here is that the Windows build is fundamentally different from the Linux build. Linux uses a single-config generation, while Windows uses a multi-config. We need to have a clear plan for how architecture fits into a multi-config system.

So, before further work, would that be possible to agree if we should do it in one or two steps (native, and then cross-compilation with --target-platform)?

Given that we're already running into questions about how generation would work, the design document seems like the right place to figure that out.

If we do it in one step, can I get the exact list of requirements and documents you or any other flutter team member need to give its approval? The thing I want to avoid, is spending several weeks to implement something/fix nits/add tests, and have a new member of the team coming to say "I don't agree". Hope that's understandable.

Avoiding that situation is exactly the point of doing a design document before doing a non-trivial PR. The list of requirements is part of what the discussion in a design document would generate.

@loic-sharma
Copy link
Member

loic-sharma commented Mar 8, 2023

The thing I want to avoid, is spending several weeks to implement something/fix nits/add tests, and have a new member of the team coming to say "I don't agree". Hope that's understandable.

That's completely understandable, and I agree with Stuart that a design doc is the best way to prevent this. Adding Windows ARM64 support will require input from folks that work on infrastructure, plugins, the Flutter tool, and Windows (obviously). The design doc will help get that feedback early and build alignment.

Here are instructions on how to write design docs: https://github.com/flutter/flutter/wiki/Design-Documents#soliciting-feedback
Here is the template for design docs: flutter.dev/go/template
You can find previous design docs here: https://docs.flutter.dev/resources/design-docs

I'd be happy to iterate with you on this design doc. You can find me on the desktop team's chat: https://discord.com/channels/608014603317936148/608020180177780791

@pbo-linaro
Copy link
Contributor Author

Thanks to both of you, @stuartmorgan @loic-sharma, for your help and guidance on this. You're right, let's do the design document, will be a good way to discuss about pending changes.

I'll do my best for the first draft, even though I don't know all the details of flutter tool architecture. For now, I'll simply close this PR.

@pbo-linaro pbo-linaro closed this Mar 9, 2023
final String arch = _platform.environment['PROCESSOR_ARCHITECTURE'] ?? '';
_hostPlatform = (arch == 'ARM64') ? HostPlatform.windows_arm64 :
HostPlatform.windows_x64;
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be used by flutter app as well to identify architecture.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c: contributor-productivity Team-specific productivity, code health, technical debt. d: api docs Issues with https://api.flutter.dev/ d: examples Sample code and demos tool Affects the "flutter" command-line tool. See also t: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants