[flutter_tools] Deferred components build system #76192

GaryQian · 2021-02-17T09:37:10Z

This PR adds the full build system that allows building deferred components apps using flutter build appbundle

Behavior is controlled via two negatable flags: --[no-]deferred-components and --[no-]-verify-deferred-components

Split off of #63773

GaryQian · 2021-02-25T22:24:47Z

Still plan on adding additional tests, but requesting review now for more preliminary feedback!

packages/flutter_tools/lib/src/android/gradle.dart

jonahwilliams · 2021-02-25T22:26:41Z

packages/flutter_tools/lib/src/android/gradle.dart

Should there be some more validation here? Could you have deferred components and no component names?

That particular case is a valid use case. Developers can include no component names to generate the loading units without packing them into dynamic features during development.

packages/flutter_tools/lib/src/asset.dart

jonahwilliams · 2021-02-25T22:27:17Z

packages/flutter_tools/lib/src/asset.dart

document what this is

jonahwilliams · 2021-02-25T22:27:42Z

packages/flutter_tools/lib/src/asset.dart

Pull this into a helper method

jonahwilliams · 2021-02-25T22:27:54Z

packages/flutter_tools/lib/src/asset.dart

globals.fs.path should have a path separator.

jonahwilliams · 2021-02-25T22:29:18Z

packages/flutter_tools/lib/src/asset.dart

Since we can't build a split aot in debug mode, could we structure this so that the calling code always passes false for debug and otherwise leaves the build mode out of the asset system?

jonahwilliams · 2021-02-25T22:30:00Z

packages/flutter_tools/lib/src/build_system/targets/android.dart

The manifest doesn't get produced if its not a split AOT right?

Correct. I'll change this to track the manifest in the depfile in case the app does not include any loading units.

jonahwilliams · 2021-02-25T22:31:51Z

packages/flutter_tools/lib/src/commands/assemble.dart

Pass this through as a build define instead of creating a special argument for it. The build apk/appbundle et cetera commands should have the well documented arg

jonahwilliams · 2021-02-25T22:32:22Z

packages/flutter_tools/lib/src/commands/assemble.dart

This should all be performed in the build system, why move it out here?

jonahwilliams · 2021-02-25T22:33:05Z

packages/flutter_tools/lib/src/commands/build_appbundle.dart

Like we talked about, no usage of the build system outside of assemble

jonahwilliams · 2021-03-03T16:40:24Z

We don't have a super verbose mode, but if you want to see why the targets are re-running, I would recommend adding logging to https://github.com/flutter/flutter/blob/master/packages/flutter_tools/lib/src/build_system/build_system.dart#L806-L856

t # This is a combination of 3 commits. Separate prebuild and build tooling, early working(ish) version Refactor, cleanup Integrate dart library names golden generation refactor REmove extra deps Remove conflicts Sync version numbers with local.properties in modules Remove no-use-bare-instructions Use strings.xml for loading unit -> module mapping Copy assets prototype Begin assets packaging Pack assets into split mdoule Package assets Fix after rebase Add shrinking disable back to support multi-apk builds Rename to deferred_component_config Move commands to proper location Move commands to new locations, cleanup Additional cleanup, rename to deferred d components Begin transition into single command setup Basic one-command verification and setup Build multiple ABIs Assets work Working intermediates system Shift .so multiplexing into dart, cleanup Consolidation and additional cleanup Goldens work Migrate so copy to new target Move setup to prebuild Working prebuild for setup Setup verification work Working setup system Cleanup and begin backwards/debug compatibility Cleanup More cleanup: Refactor verification and setup into calss, split into two parts: pre and post assembly Refactor setup code, address comments Refactoring, begin templates impl, address comments Working template based generation Docs, refactor, rename Refactor, libs copy depfile More depfiles work, debug run assets working Add dynamicFeatures build.gradle programmatically More refactoring, depfile fixes, improvement to copying and loading unit parsing Include unused loading units in base module Crash fixes: Prevent validation on debug builds, fix flutter run behavior Flutter run working again AndroidManifest.xml validation and writing Move check to Target to track dependencies. Resolve various crashes and flutter run issues Fix multi-split assets Generate diff Message formatting, async fixes Fix strings.xml edits and log recommended command to apply diff Improve strings.xml algorithm Improve regex, Add unit tests Begin validator tests, custom loggers Null safety Passing initial tests Add many validator tests Component android dir tests Add tests for AndroidManifest mapping and string.xml names Finish validator tests Test for deferred components .so copying Add validator Target test flutter manifest tests Round 1 analyzer fixes Refactor, passes analyzer Fix casts Merge with upstream Merge Fix abi mismatch when switching targets between builds Pass abi to validator Switch to BufferLogger Begin integration tests for tooling Building deferred components apps works in integration test Passing build appbundle integration tests Fix tests and analyzer, cleanup: Whitespaces Update diff Sync with validator PR Sync with validator PR tests Integrate with new validator Sync with splitbuild Fix null dep Merge changes Add tests for target

… command

Refactor abi extraction to within target

Analyzer Fix null bool

Analyzer Buildsystem Non-null splitDeferredAssets

Remove extra input Oops

Fix analyzer Fix edge case

GaryQian · 2021-03-04T22:02:06Z

dev/bots/test.dart

    if (!_allTargetsCached(file)) {
      print('${red}Not all build targets cached after second run.$reset');
-      print('The target performance data was: ${file.readAsStringSync()}');
+      print('The target performance data was: ${file.readAsStringSync().replaceAll('},', '},\n')}');


Makes the log much more readable. Quick fix for me totally missing the "skipped: false" entries in this line because it was 90% cut off onscreen.

jonahwilliams

LGTM

I would like to see some of the usages of flutter project to pass config in the build rules cleaned up a bit, but we can handle that in a follow up

jonahwilliams · 2021-02-26T21:40:53Z

packages/flutter_tools/lib/src/asset.dart

You could lose a level of indentation here by doing an early return if components is null.

jonahwilliams · 2021-03-04T22:10:37Z

dev/bots/test.dart

    if (!_allTargetsCached(file)) {
      print('${red}Not all build targets cached after second run.$reset');
-      print('The target performance data was: ${file.readAsStringSync()}');
+      print('The target performance data was: ${file.readAsStringSync().replaceAll('},', '},\n')}');


GaryQian · 2021-03-04T22:23:33Z

I'll definitely follow up with additional cleanup and I do also want to add additional tests in areas.

fluttergithubbot · 2021-03-04T23:29:02Z

This pull request is not suitable for automatic merging in its current state.

The status or check suite Google testing has failed. Please fix the issues identified (or deflake) before re-applying this label.

jonahwilliams · 2021-03-04T23:31:54Z

Ahh, the change in the asset manager API needs to be updated in g3 too. You'll need to make a g3fix using frob

flutter-dashboard bot added the tool Affects the "flutter" command-line tool. See also t: labels. label Feb 17, 2021

google-cla bot added the cla: yes label Feb 17, 2021

GaryQian force-pushed the splitbuild branch from afa3d34 to f171b2c Compare February 25, 2021 20:43

GaryQian requested a review from jonahwilliams February 25, 2021 22:24

jonahwilliams reviewed Feb 25, 2021

View reviewed changes

packages/flutter_tools/lib/src/android/gradle.dart Outdated Show resolved Hide resolved

jonahwilliams reviewed Feb 25, 2021

View reviewed changes

packages/flutter_tools/lib/src/asset.dart Outdated Show resolved Hide resolved

jonahwilliams reviewed Feb 25, 2021

View reviewed changes

packages/flutter_tools/lib/src/asset.dart Outdated

Copy link

Contributor

jonahwilliams Feb 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

document what this is

GaryQian reacted with thumbs up emoji

jonahwilliams reviewed Feb 25, 2021

View reviewed changes

packages/flutter_tools/lib/src/asset.dart Outdated

Copy link

Contributor

jonahwilliams Feb 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull this into a helper method

GaryQian reacted with thumbs up emoji

jonahwilliams reviewed Feb 25, 2021

View reviewed changes

GaryQian force-pushed the splitbuild branch from 8ba57ea to 0351409 Compare March 1, 2021 21:38

GaryQian requested a review from jonahwilliams March 1, 2021 21:40

GaryQian force-pushed the splitbuild branch 2 times, most recently from 8e596b7 to 09a2dcc Compare March 2, 2021 01:05

GaryQian mentioned this pull request Mar 3, 2021

☂️ Deferred Components: Build and load split AOT Android App Bundles #57617

Closed

18 tasks

GaryQian force-pushed the splitbuild branch from 30179bb to eff2f61 Compare March 3, 2021 09:59

GaryQian added 7 commits March 4, 2021 02:08

Fix validator runs with new system

30d1c10

Consolidate naming

369fe9f

Fix analyzer and imports

8bd856b

Configure assets, debug checks

65180d7

Remove dependency on build_system in prebuild validator and appbundle…

644ff8c

… command

Fix tests and analyzer

fcfe0cb

GaryQian added 13 commits March 4, 2021 02:08

Address comments for assets

8413a54

Refactor deferred components asset parsing into helper

9d4bad1

Refactor abi extraction to within target

Assets tests

3061ac9

Analyzer Fix null bool

Test for .so existence in integration

076116f

Analyzer Buildsystem Non-null splitDeferredAssets

Remove build_system deps from prebuild tests

14a4667

Verify assets packing

5c31198

Proper handling of rebuilding asset bundles, more tests

8c09d29

Handle mixed targets in assemble, Standardize naming verify->validate

703f239

Use depfile in AndroidAotBundle task

cb6798a

Remove extra input Oops

Fix depfile not being written

9ed9438

Allow force writing depfile

ad1dcb0

Use pathSeparator

bb2f135

Fix analyzer Fix edge case

Fix test

d5cef0a

GaryQian force-pushed the splitbuild branch from 58c26c0 to d5cef0a Compare March 4, 2021 11:35

Fix tooltip formatting

9e7a4c9

GaryQian commented Mar 4, 2021

View reviewed changes

jonahwilliams approved these changes Mar 4, 2021

View reviewed changes

nits, sync verify->validate naming

dc5d6bd

GaryQian added c: new feature Nothing broken; request for a new capability waiting for tree to go green labels Mar 4, 2021

fluttergithubbot removed the waiting for tree to go green label Mar 4, 2021

GaryQian added the waiting for tree to go green label Mar 5, 2021

fluttergithubbot merged commit ee41782 into flutter:master Mar 5, 2021

This was referenced Mar 5, 2021

Deferred Components tooling #63773

Closed

Deferred Components: Tooling and build process #73345

Closed

jmagman mentioned this pull request Mar 6, 2021

deferred_components_tests fail, "The shrinker may have failed to optimize the Java bytecode" #77445

Closed

jmagman mentioned this pull request Mar 23, 2021

Some plugins breaks on release mode on Android #78625

Closed

[flutter_tools] Deferred components build system #76192

[flutter_tools] Deferred components build system #76192

Uh oh!

Conversation

GaryQian commented Feb 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GaryQian commented Feb 25, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonahwilliams commented Mar 3, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonahwilliams left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GaryQian commented Mar 4, 2021

Uh oh!

fluttergithubbot commented Mar 4, 2021

Uh oh!

jonahwilliams commented Mar 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GaryQian commented Feb 17, 2021 •

edited

Loading