Generate ELF shared libraries and allow multi-abi libs in APKs and App bundles #33696

blasten · 2019-06-01T00:22:57Z

Description

Include ELF shared library instead of AOT snapshots in APKs and make the flag target-platform multi-option for build apk and build appbundle.

After this change, an APK built for release or profile will have the following structure:

    2460  Defl:N      889  64% 1980-00-00 00:00 f40cf37a  AndroidManifest.xml
     765  Defl:N      595  22% 1980-00-00 00:00 dce0efa9  META-INF/CERT.RSA
    1522  Defl:N      722  53% 1980-00-00 00:00 39881335  META-INF/CERT.SF
    1479  Defl:N      695  53% 1980-00-00 00:00 ce61d107  META-INF/MANIFEST.MF
     109  Defl:N       53  51% 1980-00-00 00:00 e83e6b87  assets/flutter_assets/AssetManifest.json
     208  Defl:N      113  46% 1980-00-00 00:00 92d58e5f  assets/flutter_assets/FontManifest.json
  644419  Defl:N    58421  91% 1980-00-00 00:00 548a9a0b  assets/flutter_assets/LICENSE
  134640  Defl:N    63011  53% 1980-00-00 00:00 c7c94b78  assets/flutter_assets/fonts/MaterialIcons-Regular.ttf
   97680  Defl:N    54923  44% 1980-00-00 00:00 273d62de  assets/flutter_assets/packages/cupertino_icons/assets/CupertinoIcons.ttf
- 1821616  Defl:N   773152  58% 1980-00-00 00:00 10467cf1  assets/isolate_snapshot_data
- 3060736  Defl:N   832709  73% 1980-00-00 00:00 0406652a  assets/isolate_snapshot_instr
-   23592  Defl:N    11409  52% 1980-00-00 00:00 0fa678e0  assets/vm_snapshot_data
-   12704  Defl:N     2807  78% 1980-00-00 00:00 b5e3f0f0  assets/vm_snapshot_instr
  258100  Defl:N   107181  59% 1980-00-00 00:00 5bc8eac3  classes.dex
+ 5632000  Defl:N  1850034  67% 1980-00-00 00:00 6415dcf7  lib/armeabi-v7a/libapp.so
 6032316  Defl:N  3293989  45% 1980-00-00 00:00 c82f4361  lib/armeabi-v7a/libflutter.so
     344  Defl:N      186  46% 1980-00-00 00:00 55202761  res/drawable/launch_background.xml
     507  Stored      507   0% 1980-00-00 00:00 dab4e8f1  res/mipmap-hdpi-v4/ic_launcher.png
     405  Stored      405   0% 1980-00-00 00:00 ad6e0a73  res/mipmap-mdpi-v4/ic_launcher.png
     684  Stored      684   0% 1980-00-00 00:00 77d229a5  res/mipmap-xhdpi-v4/ic_launcher.png
     994  Stored      994   0% 1980-00-00 00:00 547189ae  res/mipmap-xxhdpi-v4/ic_launcher.png
    1406  Stored     1406   0% 1980-00-00 00:00 63bc7c87  res/mipmap-xxxhdpi-v4/ic_launcher.png

Fat APK

$ flutter build apk

# Effectively the same as
$ flutter build apk --target-platform android-arm,android-arm64

# Also possible in the near future
$ flutter build apk --target-platform android-arm,android-arm64,android_x64

The tool displays a message:

You are building a fat APK that includes binaries for android-arm, android-arm64.
If you are deploying the app to the Play Store, it's recommended to use app bundles or split the APK to reduce the APK size.
    To generate an app bundle, run:
        flutter build appbundle --target-platform android-arm,android-arm64
        Learn more on: https://developer.android.com/guide/app-bundle
    To split the APKs per ABI, run:
        flutter build apk --target-platform android-arm,android-arm64 --split-per-abi
        Learn more on:  https://developer.android.com/studio/build/configure-apk-splits#configure-abi-split

Resulting APK:

    2512  Defl:N      901  64% 00-00-1980 00:00 c7f247d8  AndroidManifest.xml
     765  Defl:N      597  22% 00-00-1980 00:00 b105c0a9  META-INF/CERT.RSA
    1678  Defl:N      777  54% 00-00-1980 00:00 45833d1e  META-INF/CERT.SF
    1635  Defl:N      747  54% 00-00-1980 00:00 a3af0ca9  META-INF/MANIFEST.MF
     109  Defl:N       53  51% 00-00-1980 00:00 e83e6b87  assets/flutter_assets/AssetManifest.json
     208  Defl:N      113  46% 00-00-1980 00:00 92d58e5f  assets/flutter_assets/FontManifest.json
  644419  Defl:N    58421  91% 00-00-1980 00:00 548a9a0b  assets/flutter_assets/LICENSE
  134640  Defl:N    63011  53% 00-00-1980 00:00 c7c94b78  assets/flutter_assets/fonts/MaterialIcons-Regular.ttf
   97680  Defl:N    54923  44% 00-00-1980 00:00 273d62de  assets/flutter_assets/packages/cupertino_icons/assets/CupertinoIcons.ttf
  257176  Defl:N   107353  58% 00-00-1980 00:00 8e495fa1  classes.dex
 6311936  Defl:N  1801944  72% 00-00-1980 00:00 ed1058f5  lib/arm64-v8a/libapp.so
 8473344  Defl:N  3657267  57% 00-00-1980 00:00 540aacd2  lib/arm64-v8a/libflutter.so
 5632000  Defl:N  1850033  67% 00-00-1980 00:00 8abc54a5  lib/armeabi-v7a/libapp.so
 6032316  Defl:N  3293989  45% 00-00-1980 00:00 c82f4361  lib/armeabi-v7a/libflutter.so
     344  Defl:N      186  46% 00-00-1980 00:00 55202761  res/drawable/launch_background.xml
     507  Stored      507   0% 00-00-1980 00:00 dab4e8f1  res/mipmap-hdpi-v4/ic_launcher.png
     405  Stored      405   0% 00-00-1980 00:00 ad6e0a73  res/mipmap-mdpi-v4/ic_launcher.png
     684  Stored      684   0% 00-00-1980 00:00 77d229a5  res/mipmap-xhdpi-v4/ic_launcher.png
     994  Stored      994   0% 00-00-1980 00:00 547189ae  res/mipmap-xxhdpi-v4/ic_launcher.png
    1406  Stored     1406   0% 00-00-1980 00:00 63bc7c87  res/mipmap-xxxhdpi-v4/ic_launcher.png
    1560  Stored     1560   0% 00-00-1980 00:00 71db73fd  resources.arsc

ABI split per APK

$ flutter build apk --split-per-abi

This will generate two APKs with the appropriate version for the the Play Store:

Built build/app/outputs/apk/release/app-armeabi-v7a-release.apk.
Built build/app/outputs/apk/release/app-arm64-v8a-release.apk.

Related Issues

#18494
#31922

Tests

Unit tests to the AOT snapshotter.
Unpack the APK and app bundle and assert that the expected libraries are contained.

Checklist

Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I signed the CLA.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I updated/added relevant documentation (doc comments with ///).
All existing and new tests are passing.
The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
I am willing to follow-up on review comments in a timely manner.

Breaking Change

Does your PR require Flutter developers to manually update their apps to accommodate your change?

Yes, this is a breaking change (Please read Handling breaking changes). Replace this with a link to the e-mail where you asked for input on this proposed change.
No, this is not a breaking change.

goderbauer · 2019-06-03T18:22:55Z

Wondering what this change does to APK size. From the diff in the description it looks like it adds ~220KiB compressed?

jason-simmons · 2019-06-03T21:16:46Z

I tried building a default template app with flutter build apk --release and flutter build apk --release --build-shared-library

The regular APK was 5.23MB and the build-shared-library APK was 5.46MB.

The uncompressed app.so library was 5.96MB. Versus 4.97MB for the total of isolate_snapshot_data, isolate_snapshot_instr, vm_snapshot_data, and vm_snapshot_instr in the regular APK.

Note that we had to disable the debug symbol stripping that Android usually does to .so libraries packaged into an APK. See dart-lang/sdk#37146

@rmacnak-google

goderbauer · 2019-06-03T22:04:26Z

Is the size increase likely due to the debug symbols?

blasten · 2019-06-04T05:28:36Z

FYI: the config to avoid stripping debug symbols does not work in Flutter modules. It seems that the root cause is that the Gradle plugin is inserted as an implementation dependency:

dependencies {
    implementation project(':flutter')
}

In which, any config set in the android { } block by the plugin is completely ignored. The app's .android/app/build.gradle needs to set doNotStrip.

We may need to land a fix for dart-lang/sdk#37146 before this one goes in.

cc @rmacnak-google @jason-simmons @tvolkert

blasten · 2019-06-04T05:50:43Z

The CI seems to be running into: https://github.com/dart-lang/sdk/blob/a6bbc58d09569ed84f437d6ff55c01166745b3d4/runtime/vm/dart_api_impl.cc#L6075. The target is still arm, but the command is run from Windows. Is that expected?

blasten · 2019-06-05T04:30:35Z

Pending AIs:

The issue about Android stripping symbols was fixed in dart-lang/sdk@da0de06 (Confirmation pending, AI: remove doNotStrip "*/${abi}/libapp.so" from Gradle).
The failure on Windows will be fixed by https://github.com/flutter/engine/pull/9190/files
Add integration test that installs an APK in release mode from build apk for app and add to app.
Add integration test that installs an APK in release mode from build appbundle. Requires to update the image in devicelab to include bundletool for app and add to app.

tvolkert · 2019-06-05T17:00:41Z

The CI seems to be running into: https://github.com/dart-lang/sdk/blob/a6bbc58d09569ed84f437d6ff55c01166745b3d4/runtime/vm/dart_api_impl.cc#L6075. The target is still arm, but the command is run from Windows. Is that expected?

@rmacnak-google

blasten · 2019-06-05T17:05:50Z

@tvolkert I think https://github.com/flutter/engine/pull/9190/files will fix it.

jason-simmons · 2019-06-05T17:23:26Z

See also flutter/buildroot#258 and https://dart-review.googlesource.com/c/sdk/+/104950

dev/devicelab/bin/tasks/gradle_plugin_test.dart

mklim

LGTM pending LGTM from other reviewers as well. :)

dev/devicelab/bin/tasks/gradle_plugin_test.dart

packages/flutter_tools/gradle/flutter.gradle

dev/devicelab/bin/tasks/gradle_plugin_test.dart

packages/flutter_tools/gradle/flutter.gradle

tvolkert

Other than a few nits, LGTM

blasten · 2019-06-07T17:54:09Z

@tvolkert

How do we know it's a profile or release build at this point?

There's a check before that. I have updated both error messages, so it's more clear.

jonahwilliams

LGTM

tvolkert

LGTM

tvolkert · 2019-06-08T07:44:26Z

packages/flutter_tools/gradle/flutter.gradle

    }

+    /**
+     * Returns the platform that is used to extract the `libflutter.so` and the .class files.


Is this really used to used to extract libflutter.so? It looks like it's only used to extract the .class files.

If I'm mis-reading it (and it is used to extract libflutter.so), is it not a problem that it's returning PLATFORM_ARM64 if that's merely one of the target platforms?

It extracts libflutter.so as well, then it adds the libflutter.so for the rest of the ABIs.
Alternatively, we could extract the .class only before the afterEvaluate block (otherwise Gradle complains that we are referencing classes that aren't defined).

I filed an issue to track this in #33059 and #33064

The current engine artifacts for Android are:

android-arm/ android-arm-profile/ android-arm-release/ android-arm64/ android-arm64-profile/ android-arm64-release/ android-x64/ android-x86/

x64 and x86 are for debug only, and their corresponding libflutter.so is always added even for arm and arm64.

packages/flutter_tools/gradle/flutter.gradle

jonahwilliams · 2019-06-09T01:33:47Z

This change has doubled the measured size in the basic_material_app_android__compile benchmark.

…s and App bundles (#33696)" This reverts commit 966b15b.

…s and App bundles (#33696)" (#34121)

blasten · 2019-06-09T02:34:13Z

@jonahwilliams that's intended. Looks like the goal/baseline is defined in a db?

blasten added platform-android Android applications specifically t: gradle "flutter build" and "flutter run" on Android labels Jun 1, 2019

blasten requested a review from jason-simmons June 1, 2019 00:22

googlebot added the cla: yes label Jun 1, 2019

jason-simmons approved these changes Jun 1, 2019

View reviewed changes

Piinks added the tool Affects the "flutter" command-line tool. See also t: labels. label Jun 3, 2019

blasten force-pushed the elf branch 2 times, most recently from 6af89f8 to 0dcfa51 Compare June 5, 2019 00:15

blasten changed the title ~~Generate ELF shared libraries instead of AOT snapshots~~ Generate ELF shared libraries instead of AOT snapshots and allow multi-abi libs in APKs and App bundles Jun 5, 2019

blasten changed the title ~~Generate ELF shared libraries instead of AOT snapshots and allow multi-abi libs in APKs and App bundles~~ Generate ELF shared libraries and allow multi-abi libs in APKs and App bundles Jun 5, 2019

blasten requested review from jonahwilliams, mklim and tvolkert June 5, 2019 04:39

blasten mentioned this pull request Jun 5, 2019

Support app bundles with 32-bit and 64-bit binaries within them #31922

Closed

tvolkert reviewed Jun 5, 2019

View reviewed changes

dev/devicelab/bin/tasks/gradle_plugin_test.dart Outdated Show resolved Hide resolved

tvolkert reviewed Jun 5, 2019

View reviewed changes

dev/devicelab/bin/tasks/gradle_plugin_test.dart Outdated Show resolved Hide resolved

tvolkert reviewed Jun 5, 2019

View reviewed changes

dev/devicelab/bin/tasks/gradle_plugin_test.dart Outdated Show resolved Hide resolved

Emmanuel Garcia and others added 4 commits June 5, 2019 15:41

Generate ELF shared libraries instead of AOT snapshots

24d2d65

Allow comma-separated target platforms

f10efb3

Look for app.so in intermediates

b7d6c4f

Format code

0ec35f2

blasten requested a review from tvolkert June 7, 2019 01:15

mklim approved these changes Jun 7, 2019

View reviewed changes

tvolkert reviewed Jun 7, 2019

View reviewed changes

packages/flutter_tools/gradle/flutter.gradle Outdated Show resolved Hide resolved

tvolkert approved these changes Jun 7, 2019

View reviewed changes

Address feedback

ecc1782

blasten force-pushed the elf branch from 46b21bc to ecc1782 Compare June 7, 2019 17:53

jonahwilliams approved these changes Jun 7, 2019

View reviewed changes

Merge branch 'master' into elf

a8310eb

chinmaygarde mentioned this pull request Jun 8, 2019

[engine] Fix builds targeting Android from a Windows host gen_snapshot flutter/engine#9190

Merged

tvolkert mentioned this pull request Jun 8, 2019

Support APKs with 32-bit and 64-bit binaries within them #18494

Closed

tvolkert approved these changes Jun 8, 2019

View reviewed changes

blasten added 3 commits June 8, 2019 08:47

targetPlatform cannot be null anymore

15f4b85

Merge branch 'master' into elf

4378f17

assembly -> elf

bcf21cd

blasten force-pushed the elf branch from 6cb7892 to bcf21cd Compare June 8, 2019 20:28

Fix issue in win

2de74ed

blasten merged commit 966b15b into flutter:master Jun 9, 2019

blasten deleted the elf branch June 9, 2019 01:02

jonahwilliams mentioned this pull request Jun 9, 2019

basic_material_app_android__compile size doubled #34120

Closed

jonahwilliams pushed a commit that referenced this pull request Jun 9, 2019

Revert "Generate ELF shared libraries and allow multi-abi libs in APK…

b3b31ef

…s and App bundles (#33696)" This reverts commit 966b15b.

jonahwilliams mentioned this pull request Jun 9, 2019

Revert "Generate ELF shared libraries and allow multi-abi libs in APKs and App bundles" #34121

Merged

jonahwilliams pushed a commit that referenced this pull request Jun 9, 2019

Revert "Generate ELF shared libraries and allow multi-abi libs in APK…

f8f7b89

…s and App bundles (#33696)" (#34121)

blasten mentioned this pull request Jun 9, 2019

Generate ELF shared libraries and allow multi-abi libs in APKs and App bundles #34123

Merged

jmagman mentioned this pull request Jul 29, 2020

output.versionCodeOverride from project's build.gradle is ignored #62005

Closed

github-actions bot locked as resolved and limited conversation to collaborators Aug 6, 2021

Generate ELF shared libraries and allow multi-abi libs in APKs and App bundles #33696

Generate ELF shared libraries and allow multi-abi libs in APKs and App bundles #33696

Uh oh!

Conversation

blasten commented Jun 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Fat APK

ABI split per APK

Related Issues

Tests

Checklist

Breaking Change

Uh oh!

goderbauer commented Jun 3, 2019

Uh oh!

jason-simmons commented Jun 3, 2019

Uh oh!

goderbauer commented Jun 3, 2019

Uh oh!

blasten commented Jun 4, 2019

Uh oh!

blasten commented Jun 4, 2019

Uh oh!

blasten commented Jun 5, 2019

Uh oh!

tvolkert commented Jun 5, 2019

Uh oh!

blasten commented Jun 5, 2019

Uh oh!

jason-simmons commented Jun 5, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mklim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tvolkert left a comment

Choose a reason for hiding this comment

Uh oh!

blasten commented Jun 7, 2019

Uh oh!

jonahwilliams left a comment

Choose a reason for hiding this comment

Uh oh!

tvolkert left a comment

Choose a reason for hiding this comment

Uh oh!

tvolkert Jun 8, 2019

Choose a reason for hiding this comment

Uh oh!

blasten Jun 8, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jonahwilliams commented Jun 9, 2019

Uh oh!

blasten commented Jun 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

blasten commented Jun 1, 2019 •

edited

Loading