Looking forward to the PR!

Thanks for this contribution! What is the status of this PR? Is there anything we can do to help?

Note: we are still waiting for dart-lang/sdk#48767 (or an alternative implementation) to land to avoid breaking flutter symbolize. @sstrickl is working on it here

obfuscation doesn't produce BCSymbolMaps so the stack traces are not deobfuscated like they are on Android, i.e.

I am puzzled. BCSymbolMaps are AFAIK only relevant for bitcode (and bitcode is dead anyway).

I would have expected that debug information contained in dSYM is itself deobfuscated - so symbolizing using dSYM should produce deobfuscated result. Maybe we are missing something. @sstrickl could you take a look at that as well, while you are working on symbolization?

I am puzzled. BCSymbolMaps are AFAIK only relevant for bitcode (and bitcode is dead anyway).

indeed it works fine with the new dSYM support in native_stack_traces by @sstrickl

Something like this, which would actually leverage the target build system, caching, inputs and outputs, depfile:

  List<Target> get dependencies => const <Target>[
    KernelSnapshot(),
+    DwarfSymbolsGenerator(),
  ];

class DwarfSymbolsGenerator extends Target {
  const DwarfSymbolsGenerator();

  @override
  String get name => 'dsym_generator';

  @override
  List<Source> get inputs => const <Source>[
    Source.pattern('{OUTPUT_DIR}/App.framework/App'),
  ];

  @override
  List<Source> get outputs => const <Source>[
    Source.pattern('{OUTPUT_DIR}/App.framework.dSYM'),
  ];

  @override
  List<String> get depfiles => <String>[
    'flutter_assets.d',
  ];

  @override
  List<Target> get dependencies => <Target>[];

  @override
  Future<void> build(Environment environment) async {
    final File frameworkBinary = environment.outputDir.childDirectory('App.framework').childFile('App');
    final File dsymFile = environment.outputDir.childFile('App.framework.dSYM');

    final ProcessResult result = environment.processManager.runSync(<String>[
      'xcrun',
      'dsymutil',
      '-o',
      dsymFile.path,
      frameworkBinary.path,
    ]);
    if (result.exitCode != 0) {
      throw Exception('Failed to generate App.framework.dSYM.\n${result.stderr}');
    }
  }
}

Then the App.dSYM is always available, and included in submitted Xcode archive bundles. This could be in addition to the one optionally generated to the --split-debug-info path so flutter can also symbolicate.

which would actually leverage the target build system

Actually that Target extension is probably overkill, but I think the dsymutil logic would be better closer to

void _generateDSYMs(Environment environment, String binaryPath, BuildMode buildMode) {
  final String? splitDebugInfo = environment.defines[kSplitDebugInfo];
  // Only generate dSYMs for release mode, or if the user is storing debug info separately.
  if (buildMode != BuildMode.release && splitDebugInfo == null) {
    return;
  }

  const String dsymBasename = 'App.framework.dSYM';
  final Directory buildDir = environment.buildDir;
  final Directory dSYMOutput = buildDir.childDirectory(dsymBasename);

  final ProcessResult result = environment.processManager.runSync(<String>[
    'xcrun',
    'dsymutil',
    '-o',
    dSYMOutput.path,
    binaryPath,
  ]);
  if (result.exitCode != 0) {
    throw Exception('Failed to generate $dsymBasename.\n${result.stderr}');
  }

  // Copy to split debug directory.
  if (splitDebugInfo != null && splitDebugInfo.isNotEmpty) {
    copyDirectory(dSYMOutput, environment.fileSystem.directory(splitDebugInfo).childDirectory(dsymBasename));
  }
  // In release mode, also copy to the output directory, which will sit next to App.framework
  // and therefore be picked up by Xcode when archived and uploaded to the App Store.
  if (buildMode == BuildMode.release) {
    copyDirectory(dSYMOutput, environment.outputDir.childDirectory(dsymBasename));
  }
}

That worked archiving from Xcode:

 ll /path/to/Xcode/Archives/2022-07-15/Runner\ 7-15-22,\ 9.36\ PM.xcarchive/dSYMs
total 0
drwxr-xr-x  4 magder  primarygroup  128 Jul 15 21:36 .
drwxr-xr-x  6 magder  primarygroup  192 Jul 15 21:36 ..
drwxr-xr-x  3 magder  primarygroup   96 Jul 15 21:36 App.framework.dSYM
drwxr-xr-x  3 magder  primarygroup   96 Jul 15 21:36 Runner.app.dSYM

and from flutter build ios --release --split-debug-info=symbols

$ ll symbols
total 1352
drwxr-xr-x   4 magder  primarygroup     128 Jul 15 21:48 .
drwxr-xr-x  17 magder  primarygroup     544 Jul 15 21:47 ..
drwxr-xr-x   3 magder  primarygroup      96 Jul 15 21:50 App.framework.dSYM

Though it didn't from flutter build xcarchive, likely because this rsync needs to also move the dSYM:

That would still require the AOTSnapshotter shouldSplitDebugInfo logic.

Just a note, the pkg/native_stack_traces support for dSYM (and DWARF5, since there's some non-backwards compatible changes there we ran into while testing the dSYM support with assembly snapshots) has relanded in dart-lang/sdk@27f6c6d and been pushed to pub :)

@jmagman extracting the dsymutil call to build_system/targets/ios.dart looked like a great idea and I've experimented with it here: https://github.com/vaind/flutter/commit/17d17ed5ae2887ddfca9a66daffd400c7bfeddd0

The problem is that the produced dSYM actually doesn't contain any debug information, because the AOT snapshot is already stripped by build.dart AOTSnapshotter. Seems like dsymutil will happily create the file even on a stripped binary...

We could have the logic to exclude strip in build.dart, as it is in this PR, while keeping the extraction in ios.dart target but that seems counterproductive to keeping things "closer".

Additionally, I've found an issue in this PR - it doesn't actually strip after creating the dSYM. This is especially an issue if obfuscate is enabled. Therefore, I will be updating it to strip manually just after creating the dSYM.

This approach is feeling a bit hacky and it really would be much nicer if we could get dSYM style dwarf out of dart's gen_snapshot directly, but I guess that's quite a stretch, right @mraleph @sstrickl?

The problem is that the produced dSYM actually doesn't contain any debug information, because the AOT snapshot is already stripped by build.dart AOTSnapshotter. Seems like dsymutil will happily create the file even on a stripped binary...

I think we would still need "the AOTSnapshotter shouldSplitDebugInfo logic" from this PR to always avoid stripping on macOS and iOS, and then strip later if the dSYM is generated?

I've updated this PR significantly, including generation of dSYMs and stripping. Still need to do tests though (the existing one is outdated, so ignore)

I've had to add lipo call to merge multiple architectures (there seems to be support for this for iOS although IIRC only arm64 is currently supported, but that doesn't matter...). After the change, native_stack_traces called on a fat dSYM created by lipo prints there are no debug symbols, after making the following changes to stop it from crashing:

diff --git a/pkg/native_stack_traces/lib/src/dwarf.dart b/pkg/native_stack_traces/lib/src/dwarf.dart
index 9944f2d628d..93d235f1373 100644
--- a/pkg/native_stack_traces/lib/src/dwarf.dart
+++ b/pkg/native_stack_traces/lib/src/dwarf.dart
@@ -31,7 +31,10 @@ enum _Tag {
   // Snake case used to match DWARF specification.
   compile_unit(0x11),
   inlined_subroutine(0x1d),
-  subprogram(0x2e);
+  subprogram(0x2e),
+
+  // Internal value if we don't recognize the tag.
+  _unknown(-1);
 
   final int code;
 
@@ -39,15 +42,13 @@ enum _Tag {
 
   static const _prefix = 'DW_TAG';
 
-  static _Tag fromReader(Reader reader) {
+  static _Tag? fromReader(Reader reader) {
     final code = reader.readLEB128EncodedInteger();
     for (final name in values) {
       if (name.code == code) {
         return name;
       }
     }
-    throw FormatException(
-        'Unexpected $_prefix code 0x${code.toRadixString(16)}');
   }
 
   @override
@@ -88,8 +89,6 @@ enum _AttributeName {
         return name;
       }
     }
-    throw FormatException(
-        'Unexpected $_prefix code 0x${code.toRadixString(16)}');
   }
 
   @override
@@ -221,11 +220,12 @@ class _Attribute {
   static _Attribute? fromReader(Reader reader) {
     final name = _AttributeName.fromReader(reader);
     final form = _AttributeForm.fromReader(reader);
+
+    // Skip attributes we don't know (or need to read).
     if (name == null || form == null) {
-      // If one is null, the other should be null.
-      assert(name == null && form == null);
       return null;
     }
+
     return _Attribute._(name, form);
   }
 
@@ -253,6 +253,8 @@ class _Abbreviation {
     final code = reader.readLEB128EncodedInteger();
     if (code == 0) return null;
     final tag = _Tag.fromReader(reader);
+    // Note: returning `null` would break the reader loop.
+    // Instead we pass `_Tag._uknown` below and later filter it out.
     final childrenByte = reader.readByte();
     if (childrenByte != _dwChildrenNo && childrenByte != _dwChildrenYes) {
       throw FormatException('Expected DW_CHILDREN_no or DW_CHILDREN_yes: '
@@ -260,7 +262,7 @@ class _Abbreviation {
     }
     final children = childrenByte == _dwChildrenYes;
     final attributes = reader.readRepeated(_Attribute.fromReader).toList();
-    return _Abbreviation._(code, tag, children, attributes);
+    return _Abbreviation._(code, tag ?? _Tag._unknown, children, attributes);
   }
 
   void writeToStringBuffer(StringBuffer buffer) {
@@ -298,6 +300,7 @@ class _AbbreviationsTable {
   static _AbbreviationsTable? fromReader(Reader reader) {
     final abbreviations = Map.fromEntries(reader
         .readRepeated(_Abbreviation.fromReader)
+        .where((abbr) => abbr.tag != _Tag._unknown)
         .map((abbr) => MapEntry(abbr.code, abbr)));
     return _AbbreviationsTable._(abbreviations);
   }

@sstrickl Do you happen to know what would need to change in native_stack_traces for it to support dSYM files merged by lipo, it can be simulated like this (assuming an app dSYM):

lipo ./.dart_tool/flutter_build/b7d263367c5969e84da4d8dc464b837f/arm64/App.framework.dSYM -create -output fat-

running decode in native_stack_traces fails 
```shell-script
dart run bin/decode.dart dump /app-path/fat-dwarf
# Error: file "..." does not contain debugging information.

while running the same command on arm64 dwarf works fine

dart run bin/decode.dart dump /app-path/.dart_tool/flutter_build/b7d263367c5969e84da4d8dc464b837f/arm64/App.framework.dSYM/Contents/Resources/DWARF/App | wc -l
 1589249

@sstrickl Do you happen to know what would need to change in native_stack_traces for it to support dSYM files merged by lipo, it can be simulated like this (assuming an app dSYM):

lipo ./.dart_tool/flutter_build/b7d263367c5969e84da4d8dc464b837f/arm64/App.framework.dSYM -create -output fat-

Hmm, don't know! I'll take a look after discharging initial gardening duties, see what's going on there.

Actually, wait, I might have an idea. Just to check, what's the directory structure of the .dSYM generated?

From my own tests, I assumed that if you're given .dSYM, that the appropriate Mach-O file is at <file>.dSYM/Content/Resources/DWARF/<file>. I'm guessing that the file in the DWARF directory is not the same as the basename of the dSYM (without the .dSYM extension though). I should probably just look to see if that directory contains one file, and if so use it.

So I bet that's all it is.

Yeah

Actually, wait, I might have an idea. Just to check, what's the directory structure of the .dSYM generated?

From my own tests, I assumed that if you're given .dSYM, that the appropriate Mach-O file is at <file>.dSYM/Content/Resources/DWARF/<file>. I'm guessing that the file in the DWARF directory is not the same as the basename of the dSYM (without the .dSYM extension though). I should probably just look to see if that directory contains one file, and if so use it.

So I bet that's all it is.

I've noticed that assumption in the code and that doesn't work with shat this PR produces so I'm passing the actual dwarf file directly instead of the dSYM container. It reads the content, just isn't able to recognize it when it's a fat binary.

Ah, sorry, misunderstood. Well, I did try fixing the logic in this CL https://dart-review.googlesource.com/c/sdk/+/252821, along with my take on your handling of unknown DW_AT/DW_TAG values. Could you try that out for me while I look into the fat binary format and see what I need to do?

Ah, looks like it's just literally concatenated MachO files with a header, so I'll just need to add a parser for the header part. On it.

Okay, have updated https://dart-review.googlesource.com/c/sdk/+/252821 with a universal macOS binary parser as well, go ahead and try it out if you get a chance while I work on an appropriate test so I can send it for review.

Okay, have updated https://dart-review.googlesource.com/c/sdk/+/252821 with a universal macOS binary parser as well, go ahead and try it out if you get a chance while I work on an appropriate test so I can send it for review.

Works fine with the same binaries I got laying around from Monday - both fat and plain arm64 and also the automatic dSYM dwarf file is picked up, as expected. Thanks, @sstrickl!

auto label is removed for flutter/flutter, pr: 101586, Failed to merge pr#: 101586 with OperationException(linkException: null, graphqlErrors: [GraphQLError(message: Base branch was modified. Review and try the merge again., locations: [ErrorLocation(line: 2, column: 3)], path: [mergePullRequest], extensions: null)]).