Use PE/COFF AOT snapshots for Windows x64 - #187594

Thanks for sending this patch! FYI, we'll hold off on reviewing this until the Dart SDK CL lands: https://dart-review.googlesource.com/c/sdk/+/511320

Dart has merge the DLL support in https://dart-review.googlesource.com/c/sdk/+/511320. So we are able to go on, first I should change author to [email protected] before I go on

Is there a reason why Dart SDK change only implements PE/COFF for x86_64? It seems quite weird still requiring ELF loader on arm64.

Is there a reason why Dart SDK change only implements PE/COFF for x86_64? It seems quite weird still requiring ELF loader on arm64.

There are three main reasons for this change:

1. It allows crash reporting to unwind through AOT frames. Before this patch, once the stack walker entered an AOT frame, unwinding stopped there, so any frames below that AOT frame were invisible.

2. It provides detailed information for AOT frames in crash/ANR reports, such as source file and line number, making it easier for developers to understand what happened.

3. It makes module loading on Windows more consistent. Instead of introducing a separate ELF-loading path, we can use the native Windows loader and make AOT modules visible to the system and debuggers.

@linzj I think knopp was asking a different question: should we use PE/COFF AOT snapshots for Windows Arm64 too?

This pull request is not mergeable in its current state, likely because of a merge conflict. Pre-submit CI jobs were not triggered. Pushing a new commit to this branch that resolves the issue will result in pre-submit jobs being scheduled.

fyi this caused a 20% improvement in startup time:

https://flutter-flutter-perf.luci.app/e/?begin=1791860242&enable_chart_tooltip=true&end=1791911489&keys=Xf38e43c39228221e9392d98dcea3452b&plotSummary=true&selected=commit%3D2810319%26name%3D%252Carch%253Darm%252Cbranch%253Dmaster%252Cconfig%253Ddefault%252Cdevice_type%253Dnone%252Cdevice_version%253Dnone%252Chost_type%253Dwin%252Csub_result%253DtimeToFirstFrameMicros%252Ctest%253Dflutter_gallery_win_desktop__start_up%252C%26tableCol%3D2%26tableRow%3D1726&use_titles=true&xbaroffset=2810319

cc @walley892 since you've been looking into startup time

risk

Test failure looks like a flake, rerunning...

@loic-sharma Is it okay to add an optional param for building in old ELF? New PE DLL format has a lot of negatives sides i think people don't want to pay just to add a better debug info for cases that happen too rarely (i see the 20% improvement in startup time but even if it was -90%, the downsides are still worse)

• --split-debug-info does not reduce app.so as good as ELF (file is 22% bigger)
• The PDB in the split symbols directory replaces app.windows-x64.symbols. It works for native debuggers, but it cannot decode Dart stack traces, so obfuscated frames like bUa (Byb:21) can no longer be mapped back to source
• dll now keeps absolute pdb path from developers machine (there's no /PDBALTPATH:app.pdb param) like C:\src\aotapps\counter_pecoff\.dart_tool\flutter_build\931430df…\app.pdb
• each build is built with new hash because there's no /Brepro param to remove timestamps
• and finally even default flutter counter app is incorrectly displayed as a virus now, i think apps bigger then default counter functionality should be even worse here.
  I understand it's 1 detection that seems tiny, but it's just a counter app, without any extra packages or features. Writing to false positive emails of antivirus companies on each mini-update does not seem good

Old format did not have these issues

• and finally even default flutter counter app is incorrectly displayed as a virus now, i think apps bigger then default counter functionality should be even worse here.
  I understand it's 1 detection that seems tiny, but it's just a counter app, without any extra packages or features. Writing to false positive emails of antivirus companies on each mini-update does not seem good

The other issues needs to be looked at, but the false positive needs to be reported until they fix their broken heuristics. There's no way around that. We can't be expected to ship ELF on windows just so that we please some broken antivirus heuristics. Btw. only one antivirus flagging this is actually surprisingly good result, sadly.

• and finally even default flutter counter app is incorrectly displayed as a virus now, i think apps bigger then default counter functionality should be even worse here.
  I understand it's 1 detection that seems tiny, but it's just a counter app, without any extra packages or features. Writing to false positive emails of antivirus companies on each mini-update does not seem good

The other issues needs to be looked at, but the false positive needs to be reported until they fix their broken heuristics. There's no way around that. We can't be expected to ship ELF on windows just so that we please some broken antivirus heuristics. Btw. only one antivirus flagging this is actually surprisingly good result, sadly.

But ELF did not have this problem. I understand heuristics are broken, but what they'll do is just add 1 file hash to their clean list and problem won't be fixed, 1 is surprisingly good but it's default flutter counter app - no extra packages, no functions. Will PR with extra flag for old format be accepted if proposed? New will remain as is, just 1 param to build with what was already stable and working fine for those who don't want to change anything to worse

• and finally even default flutter counter app is incorrectly displayed as a virus now, i think apps bigger then default counter functionality should be even worse here.
  I understand it's 1 detection that seems tiny, but it's just a counter app, without any extra packages or features. Writing to false positive emails of antivirus companies on each mini-update does not seem good

The other issues needs to be looked at, but the false positive needs to be reported until they fix their broken heuristics. There's no way around that. We can't be expected to ship ELF on windows just so that we please some broken antivirus heuristics. Btw. only one antivirus flagging this is actually surprisingly good result, sadly.

But ELF did not have this problem. I understand heuristics are broken, but what they'll do is just add 1 file hash to their clean list and problem won't be fixed, 1 is surprisingly good but it's default flutter counter app - no extra packages, no functions.

The size doesn't really matter. I had false positives consistently generated for NSIS uninstaller executable which is tiny.

It's also not just adding hashes. I never had false positive with cynet but I had with many other anti-viruses and after several reports they all actually fixed the heuristics.

Will PR with extra flag for old format be accepted if proposed? New will remain as is, just 1 param to build with what was already stable and working fine for those who don't want to change anything to worse

I can't really answer that, but personally I'd rather see the issues diagnosed and fixed. Shipping ELF on Windows was always a hack. I think we would prefer to ship native binaries.

Btw. can you explain what the point of obsfucating the Dart code on windows is? There are no debug symbols inside PE/COFF files, what is the point of additional obsfucation? It think it has been conceded that the obsfucation will not work for windows.

Btw. can you explain what the point of obsfucating the Dart code on windows is? There are no debug symbols inside PE/COFF files, what is the point of additional obsfucation? It think it has been conceded that the obsfucation will not work for windows.

Personally i never used it, just a scenario of how people use the current format

Obfuscation doesn't depend on debug symbols. --obfuscate renames Dart identifiers stored in the snapshot itself, so it applies to the PE/COFF app.so as well. A stack trace from a PE build of a small test app shows it:

#0      bUa (Byb:21)
#1      aUa (Byb:18)
#2      main (Byb:10)

The same PE build without --obfuscate prints innerFrame (package:symtest/main.dart:21)

A few open sources repos i found withing 2 minutes on git:

What changed for them with PE/COFF: --split-debug-info now writes app.windows-x64.pdb instead of app.windows-x64.symbols. The PDB has 2 public symbols, no procedures and no line tables, so the frames above can't be mapped back to source anymore. The stack trace tables also stay in app.so, which makes it 22% larger than the ELF build with the same flags (3,395,584 vs 2,786,264 bytes on a counter app).

Is it a problem for them? I don't know. I personally did not like the false positives trigger out of nowhere, absolute pdb path from machine and different hash on each build

### I understand flutter can not depend on a few repos and will not try to avoid breaking changes for a tiny amount of people who use that, but i can't see anything wrong for keeping support for format that's already working for years and does not have any obvious issues except for that debug stack trace problem

Again, what is the point of renaming symbol if the symbols only exist in the PDB file that is never shipped to customers? PE/COFF does not have any symbol names in the binary itself.

The stack trace tables also stay in app.so, which makes it 22% larger than the ELF build with the same flags (3,395,584 vs 2,786,264 bytes on a counter app).

There shouldn't be any symbols in the COFF. It's not impossible but it's deprecated. The symbols should only be inside PDBs. I assumed this was already happening but maybe the assumption was wrong.

I understand flutter can not depend on a few repos and will not try to avoid breaking changes for a tiny amount of people who use that, but i can't see anything wrong for keeping support for format that's already working for years and does not have any obvious issues except for that debug stack trace problem

It's something that adds complexisty and that we need to support and keep maintaining. There is some cost associated with it.

Again, what is the point of renaming symbol if the symbols only exist in the PDB file that is never shipped to customers? PE/COFF does not have any symbol names in the binary itself.

The stack trace tables also stay in app.so, which makes it 22% larger than the ELF build with the same flags (3,395,584 vs 2,786,264 bytes on a counter app).

There shouldn't be any symbols in the COFF. It's not impossible but it's deprecated. The symbols should only be inside PDBs. I assumed this was already happening but maybe the assumption was wrong.

I understand flutter can not depend on a few repos and will not try to avoid breaking changes for a tiny amount of people who use that, but i can't see anything wrong for keeping support for format that's already working for years and does not have any obvious issues except for that debug stack trace problem

It's something that adds complexisty and that we need to support and keep maintaining. There is some cost associated with it.

I'm not sure what cost are you referring to, format remains for ARM and Linux, it's already there, it was already used for years. dart compile exe and dart compile aot-snapshot still use ELF. Dart runtime loads these snapshots on Windows with its own ELF loader (Dart_LoadELF, via TryReadAppSnapshotElf for aot-snapshot and TryReadAppendedAppSnapshotFromPE for exe, in runtime/bin/snapshot_utils.cc). It doesn't seem to be a hack

The change to add a flag for optional ELF format in Flutter is about 116 added and 28 removed lines outside of tests: an option in flutter_tools and a format check on load in the engine.

Obfuscation doesn't touch native symbols. It renames Dart class, function and library names that are stored in the snapshot data itself, in .rdata. The runtime uses them for stack traces and runtimeType.toString(), without any PDB. In the counter app the COFF symbol table is empty (NumberOfSymbols = 0), yet .rdata still contains strings like package:counter/main.dart. That's why a PE build prints innerFrame (package:symtest/main.dart:21) without reading the PDB, and bUa (Byb:21) with --obfuscate.

The extra size is Dart's own stack trace data (CodeSourceMaps). The ELF build drops it with --dwarf-stack-traces, the PE build doesn't pass that flag. With the flag added by hand the PE app.so is 2,771,968 bytes vs 2,786,264 for ELF. But the PDB has no Dart functions or line info (2 public symbols, 0 procedures), so such stack traces can't be symbolized. That's why it isn't passed

'm not sure what cost are you referring to, format remains for ARM and Linux, it's already there, it was already used for years. dart compile exe and dart compile aot-snapshot still use ELF. Dart runtime loads these snapshots on Windows with its own ELF loader (Dart_LoadELF, via TryReadAppSnapshotElf for aot-snapshot and TryReadAppendedAppSnapshotFromPE for exe, in runtime/bin/snapshot_utils.cc). It doesn't seem to be a hack

I'm referring to the complexity of having something configurable and have to keep supporting two paths on single platform. Yes, ARM64 still needs ELF but that's missing implementation rather than by choice. And it's not a flag.

Obfuscation doesn't touch native symbols.

The source of native symbols are dart classes and methods.

The extra size is Dart's own stack trace data (CodeSourceMaps). The ELF build drops it with --dwarf-stack-traces, the PE build doesn't pass that flag. With the flag added by hand the PE app.so is 2,771,968 bytes vs 2,786,264 for ELF. But the PDB has no Dart functions or line info (2 public symbols, 0 procedures), so such stack traces can't be symbolized. That's why it isn't passed

I assumed this also works with PDB but I guess I was wrong. In this case optionally still supporting ELF on windows makes more sense.

@krll-kov thanks for the detailed testing. The "2 public symbols, 0 procedures" observation is correct, but it's a bug in how flutter_tools drives gen_snapshot, not a limitation of PE/COFF.

Root cause

flutter_tools passes --strip to gen_snapshot by default. For PE/COFF output, --strip means the VM doesn't create a DWARF object for the snapshot writer (Dart_CreateAppAOTSnapshotAsBinary in dart_api_impl.cc), and CoffWriter::EmitCodeView() returns early when there's no DWARF (coff.cc). So the .obj has no .debug$S/.debug$T, and link /DEBUG has only the two exported data symbols to put in the PDB. The VM already has a CodeView writer (codeview.cc, which emits S_GPROC32_ID + DEBUG_S_LINES), but it never runs in Flutter builds.

The same check also gates .pdata/.xdata (x64 unwind info). So today's stripped builds also ship the Dart code without unwind tables.

Verification

I used a small Dart program with the same gen_snapshot and link.exe flags flutter_tools uses, changing only --strip:

• llvm-symbolizer --obj=app.so resolves addresses inside innerFrame to innerFrame main.dart:4.
• With --obfuscate and no --strip, the PDB has the original names (innerFrame, outerFrame), and app.so contains none of them. That's the same split as the ELF .symbols file.
• With /Brepro /PDBALTPATH:%_PDB%, linking the same .obj twice gives a byte-identical app.so. The debug directory then records only app.pdb instead of the absolute build path.

Planned follow-up

1. For Windows x64 builds with --split-debug-info, stop passing --strip to gen_snapshot. Then app.windows-x64.pdb will contain the Dart functions and line tables, with de-obfuscated names.
2. Add /Brepro and /PDBALTPATH:%_PDB% to the link step. This fixes the build-to-build hash changes and the leaked absolute PDB path.
3. Separately: passing --dwarf-stack-traces (which removes the ~22% CodeSourceMaps overhead) only makes sense once PC-based stack traces can be symbolized against the PDB. Today flutter symbolize / package:native_stack_traces only read DWARF. I'll look into a PDB-based symbolization path and file an issue for it, rather than turning the flag on before traces can be decoded.
4. I'll also ask on the Dart side whether unwind data should be emitted for stripped builds too.

This doesn't address DLL file locking or the antivirus heuristics. Those are product trade-offs for the ELF-option discussion in #194854.

@krll-kov A huge thank you for the excellent feedback before this landed on stable.

@linzj Thanks for the analysis. From my understanding of your planned follow-up work:

• After item 1: When using --obfuscate --split-debug-info=..., you'll be able to symbolicate using native debugging tools, but not with flutter symbolize. Also, Dart code cannot de-obfuscate StackTrace.current.
• After item 2: Builds will be reproducible again
• After item 3: flutter symbolize will support PDBs and Dart code can de-obfuscate StackTrace.current. The PE/COFF mode will no longer be ~22% larger than the ELF mode when using --split-debug-info.

And like you said, this won't address the DLL file locking or antivirus heuristics.

Is my understanding correct?

Assuming my understanding is correct, I'd recommend we revert this PR. Item 1 seems like a definite prerequisite before we reland this PR. However, I lean towards also making item 3 a prequisite before relanding. What do people think?

@krll-kov A huge thank you for the excellent feedback before this landed on stable.

@linzj Thanks for the analysis. From my understanding of your planned follow-up work:

* After item 1: When using `--obfuscate --split-debug-info=...`, you'll be able to symbolicate using native debugging tools, but not with `flutter symbolize`. Also, Dart code cannot de-obfuscate `StackTrace.current`.

* After item 2: Builds will be reproducible again

* After item 3: `flutter symbolize` will support PDBs and Dart code can de-obfuscate `StackTrace.current`. The PE/COFF mode will no longer be ~22% larger than the ELF mode when using `--split-debug-info`.

And like you said, this won't address the DLL file locking or antivirus heuristics.

Is my understanding correct?

Assuming my understanding is correct, I'd recommend we revert this PR. Item 1 seems like a definite prerequisite before we reland this PR. However, I lean towards also making item 3 a prequisite before relanding. What do people think?

If decided not to revert or decision is delayed until the next stable release, is it okay to enable old format as optional flag in flutter build command, not many developers need those tradeoffs for debugging problem, which is rare? #194854

@loic-sharma Mostly yes, with one correction on item 3:

• Item 1: Correct. The PDB gets the Dart functions and line tables, so native debuggers and crash dumps can symbolicate. flutter symbolize can't use it.
• Item 2: Correct.
• Item 3: Neither format de-obfuscates StackTrace.current inside the app. With --split-debug-info, ELF builds print PC-based stack traces that are decoded offline with flutter symbolize and the .symbols file. Item 3 gives PE/COFF the same workflow, and removes the ~22% size overhead.

The regression only affects apps built with --split-debug-info (size) or --obfuscate (stack traces can't be decoded). Non-obfuscated PE/COFF builds still print readable stack traces in-process.

Item 3 prototype

I prototyped item 3 to check it's feasible. gen_snapshot can write the same separate DWARF debug file for PE/COFF that it already writes for ELF and iOS. The Dart change is small:

• Allow --save-debugging-info for app-aot-pecoff-obj (gen_snapshot.cc).
• Don't force generate_debug = false for PE/COFF (dart_api_impl.cc).
• Let CoffWriter::AssertConsistency accept an ELF debug writer (coff.h). The decoder resolves frames by _kDartSnapshotText+offset, so the debug file doesn't need to match the DLL's virtual addresses. This is the same model as assembly snapshots with a separate debug ELF.

flutter_tools then passes the same flags it uses on ELF platforms: --dwarf-stack-traces --resolve-dwarf-paths --save-debugging-info=app.windows-x64.symbols.

Results on a small Dart program built with --obfuscate, run with dartaotruntime (which loads the PE DLL through LoadLibrary):

So with item 3, PE/COFF ends up slightly smaller than ELF instead of larger.

One dependency: flutter symbolize is currently broken on all platforms with this Dart SDK (#191879). flutter_tools still uses native_stack_traces 0.6.1, which doesn't know the renamed _kDartSnapshotText symbol; the ELF build above fails the same way. #192473 bumps it to 0.7.0, which is what I used to decode the traces above.

Still open

• PE/COFF stack traces don't print a build_id, and the debug file has its own. flutter symbolize -d doesn't need it, but tools that match debug files by build ID (e.g. Crashlytics, Sentry) do, so the DLL needs to carry the same ID.
• For PE/COFF, the virt field prints the same value as abs. The decoder doesn't use it here, but it should print the RVA or be omitted.
• The AssertConsistency change needs a proper design with the Dart team.
• I've only tested this with dartaotruntime, not yet in a full Flutter Windows app.

Revert

I'm fine with reverting. My plan for relanding:

1. Dart SDK: PE/COFF split debug info (item 3), including the build ID.
2. Update packages, ensuring native_stack_traces is bumped to 0.7.0. #192473 (native_stack_traces 0.7.0).
3. Reland this PR with items 1–3 in flutter_tools.

Update: I've started on item 3 on the Dart side: https://dart-review.googlesource.com/c/sdk/+/568321

The CL:

• Lets gen_snapshot write --save-debugging-info for PE/COFF snapshots, as a separate DWARF ELF file (the same model as assembly snapshots).
• Adds a build ID to PE/COFF snapshots that matches the one in the debug file, so tools that match debug files by build ID work too.
• Fixes the DSO base / virt fields in PE/COFF stack traces.

It adds a PE/COFF variant to vm/dart/use_dwarf_stack_traces_flag_test, which passes locally. Once it lands and rolls into Flutter, the flutter_tools side is mostly passing the same --dwarf-stack-traces --save-debugging-info flags for Windows x64 as for ELF platforms. flutter symbolize will also need #192473 (native_stack_traces 0.7.0).

I want to provide high-level commentary: we absolutely certainly want to use native executable/object/debugging information formats on all platforms. Dart AOT snapshots should just be understood by native tooling and OS specific dynamic linkers. This means using ELF on Linux, Mach-O on Mac and it PE/COFF on Windows. We used ELF on Windows because it was simpler this way. If we get PE/COFF output to parity with ELF output in terms of capabilities we most certainly want to switch away from ELF format and delete all and any support for using ELF on Windows.

In that vein: I think debugging information should also be represented by OS specific format. I don't think it makes sense to bend backwards and use DWARF on Windows. It should be PDB based. llvm-symbolizer can handle PDBs. So I think we should just support PDB format in native_stack_traces as well. Another option we could consider is to follow crashpad footsteps and introduce platform independent format which native_stack_traces could parse - but that's a change which requires more consideration because we would need to tell users that we know produce two different debugging formats (one for consumption by native tools and one for consumption by Flutter tooling).

Following @mraleph's suggestion, I've dropped the DWARF-on-Windows approach (abandoned https://dart-review.googlesource.com/c/sdk/+/568321) and switched to PDBs. I'm starting with the Dart SDK side:

1. https://dart-review.googlesource.com/c/sdk/+/568825 (package:native_stack_traces): reads the PDB of a PE/COFF snapshot, so flutter symbolize can decode PE/COFF stack traces with the .pdb. This covers procedures, line tables and inlined frames, and uses the PDB signature as the build ID.
2. https://dart-review.googlesource.com/c/sdk/+/568826 (VM): makes the CodeView records from gen_snapshot complete enough for that:
   • inlined frames (S_INLINESITE) and columns
   • invisible functions hidden, like artificial DWARF subprograms
   • virt addresses as RVAs that match the PDB
   • the PDB GUID+age as the build ID of the PE image, so tools that match debug files by build ID (Crashlytics, Sentry) can find the PDB

With these, the .pdb in the --split-debug-info directory is the only debug file for Windows x64. Native debuggers and flutter symbolize both use it.

Once they land and roll into Flutter, I'll do the flutter_tools side:

• Stop passing --strip for Windows x64 when --split-debug-info is set, so the PDB contains the Dart functions and line tables.
• Add /Brepro /PDBALTPATH:%_PDB% to the link step, for reproducible builds with no absolute PDB path.
• Pass --dwarf-stack-traces, which removes the ~22% size overhead, and bump native_stack_traces to a version with PDB support.

I'll post updates here as the CLs progress.

Reason for revert: See the comments above:

• Use PE/COFF AOT snapshots for Windows x64 #187594 (comment)
• Use PE/COFF AOT snapshots for Windows x64 #187594 (comment)

Failed to revert commit 95bdb98 cleanly. Please resolve conflicts manually.

Reason for revert: See the comments above:

• Use PE/COFF AOT snapshots for Windows x64 #187594 (comment)
• Use PE/COFF AOT snapshots for Windows x64 #187594 (comment)

Thanks @loic-sharma. To confirm the reland plan: I'll finish the Dart SDK side first (PDB support in package:native_stack_traces in https://dart-review.googlesource.com/c/sdk/+/568825, and the CodeView changes in gen_snapshot in https://dart-review.googlesource.com/c/sdk/+/568826), then reland this PR with the flutter_tools changes from my latest update (no --strip with --split-debug-info, /Brepro /PDBALTPATH:%_PDB%, and --dwarf-stack-traces with a PDB-aware native_stack_traces). Does that sound right?

Also, the revert bot couldn't revert this cleanly because of conflicts. Would you like me to open a manual revert PR?