Skip to content

Conversation

@dcharkes
Copy link
Contributor

@dcharkes dcharkes commented Feb 1, 2024

Native assets in other build systems are not built with package:native_assets_builder invoking build.dart scripts. Instead all packages have their own blaze rules. Therefore we'd like to not depend on package:native_assets_builder from flutter tools in g3 at all.

This PR aims to move the imports of native_assets_builder and native_assets_cli into the isolated/ directory and into the files with a main function that are not used in with other build systems.

In order to be able to remove all imports in files used by other build systems, two new interfaces are added HotRunnerNativeAssetsBuilder and TestCompilerNativeAssetsBuilder. New parameters are then piped all the way through from the entry points:

  • bin/fuchsia_tester.dart
  • lib/executable.dart

The build_system/targets dir is already excluded in other build systems.

So, after this PR only the two above files and build_system/targets import from isolated/native_assets/ and only isolated/native_assets/ import package:native_assets_cli and package:native_assets_builder.

Context:

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

@github-actions github-actions bot added tool Affects the "flutter" command-line tool. See also t: labels. a: desktop Running on desktop labels Feb 1, 2024
@dcharkes dcharkes requested a review from chingjun February 1, 2024 15:07
@dcharkes
Copy link
Contributor Author

dcharkes commented Feb 1, 2024

Copy link
Contributor

@reidbaker reidbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RSLGTM

@jmagman
Copy link
Member

jmagman commented Feb 1, 2024

cc @christopherfujino do we have precedent for this kind of g3 import issue?

@christopherfujino
Copy link
Contributor

cc @christopherfujino do we have precedent for this kind of g3 import issue?

we have precedent for the issue, but not real great solution. In this particular situation, where we can move the code in question to isolated/ this seems ideal (i think?), as we're reducing the API surface we need to roll to Google.

@chingjun
Copy link
Contributor

chingjun commented Feb 1, 2024

flutter_tools/lib/src/test/test_compiler.dart is fine, it is now used internally.

For third_party/dart/flutter_tools/lib/src/run_hot.dart, can we move NativeAssetsBuildRunnerImpl into isolated, and have it (or a factory of it) be injected from executable.dart? That's how other classes in isolated are used usually.

For flutter_tools/lib/src/build_system/targets/common.dart, I don't have a better solution for that. I don't think the tools support applying patches currently, so we either need to change the tools, or use a different approach here. Ideally we should not depend on anything in build_system internally. I'll see if it's possible to isolate everything that depends on build_system/targets.


@jmagman There are precedence, which is why isolated was created, but the reasons are slightly different. Previously it was mostly to avoid introducing new dependencies. For native_assets, it is because of the version skew introduced by different roll schedule of the Dart SDK and Flutter SDK, causing difficulties for the Dart team to make changes to the native_assets tools.

I agree that this is not a good long term solution. The proper solution is to refactor flutter_tools into different parts for building, running, debugging, test setup, user interfacing etc. Internal toolings could then use only the parts that are relevant.

@dcharkes
Copy link
Contributor Author

dcharkes commented Feb 2, 2024

For third_party/dart/flutter_tools/lib/src/run_hot.dart, can we move NativeAssetsBuildRunnerImpl into isolated, and have it (or a factory of it) be injected from executable.dart? That's how other classes in isolated are used usually.

I'm not entirely sure how this would solve avoiding to import package:native_assets_builder and package:native_assets_cli, because the NativeAssetsBuildRunner type has types from those packages in the interface.

Ideally we should not depend on anything in build_system internally. I'll see if it's possible to isolate everything that depends on build_system/targets.

Okay, let me know once you have an update.

auto-submit bot pushed a commit that referenced this pull request Feb 2, 2024
…p level entrypoints in flutter_tools. (#142760)

Add a new `BuildTargets` class that provides commonly used build targets. And avoid importing files from `build_system/targets` except from the top level entrypoints or from top level commands.

Also move `scene_importer.dart` and `shader_compiler.dart` into `build_system/tools` because they are not `Target` classes, but wrapper for certain tools.

With this change, we can ignore all files in `build_system/targets` internally and make PR #142709 easier to land internally. See cl/603434066 for the corresponding internal change.

Related to:
#142709
#142041

Also note that I have opted to add a new variable in `globals.dart` for `BuildTargets` in this PR, but I know that we are trying to get rid of globals. Several alternatives that I was considering:

1. Add a new field in `BuildSystem` that returns a `BuildTargets` instance. Since `BuildSystem` is already in `globals`, we can access build targets using `globals.buildSystem.buildTargets` without adding a new global variable.
2. Properly inject the `BuildTargetsImpl` instance from the top level `executable.dart` and top level commands.

Let me know if you want me to do one of the above instead. Thanks!
@chingjun
Copy link
Contributor

chingjun commented Feb 2, 2024

Change that allows excluding build_system internally is submitted here: #142760

I'm not entirely sure how this would solve avoiding to import package:native_assets_builder and package:native_assets_cli, because the NativeAssetsBuildRunner type has types from those packages in the interface.

The main idea is to take out the parts in run_hot.dart where it references native_assets, and move them into a part that is replaceable.

In this case specifically, run_hot.dart contains 3 references to native_assets.dart

  1. Reference to the type NativeAssetsBuildRunner.
  2. Invoking the constructor NativeAssetsBuildRunnerImpl.
  3. Calling dryRunNativeAssets.

We want to avoid (2), avoid (1) or make it a type that does not depend on native_assets, and inject (3).

My idea after looking at the code is to have a new class:

abstract class NativeAssetsBuildRunnerRunner {
  Future<String> dryRunNativeAssets({
    Uri projectUri,
    PackageConfig packageConfig,
    FileSystem fileSystem,
    Logger logger,
    List<FlutterDevice> flutterDevices,
  });
}

And another class in isolated

class NativeAssetsBuildRunnerRunnerImpl extends NativeAssetsBuildRunnerRunner {
  @override
  Future<String> dryRunNativeAssets({...}) {
    final buildRunner = NativeAssetsBuildRunner(...);
    return dryRunNativeAssets(...);
  }
}

(The name is ugly I know, we could try to come up with a better name.)

HotRunner will now take NativeAssetsBuildRunnerRunner instead, and we pipe that dependency all the way from executable.dart.

What do you think?

@dcharkes
Copy link
Contributor Author

dcharkes commented Feb 2, 2024

The main idea is to take out the parts in run_hot.dart where it references native_assets, and move them into a part that is replaceable.

In this case specifically, run_hot.dart contains 3 references to native_assets.dart

  1. Reference to the type NativeAssetsBuildRunner.
  2. Invoking the constructor NativeAssetsBuildRunnerImpl.
  3. Calling dryRunNativeAssets.

We want to avoid (2), avoid (1) or make it a type that does not depend on native_assets, and inject (3).

My idea after looking at the code is to have a new class:

abstract class NativeAssetsBuildRunnerRunner {
  Future<String> dryRunNativeAssets({
    Uri projectUri,
    PackageConfig packageConfig,
    FileSystem fileSystem,
    Logger logger,
    List<FlutterDevice> flutterDevices,
  });
}

Yeah, basically pass in everything that's needed for the constructor and invocation, I was thinking in the same direction. It's just not very pretty.

And another class in isolated

class NativeAssetsBuildRunnerRunnerImpl extends NativeAssetsBuildRunnerRunner {
  @override
  Future<String> dryRunNativeAssets({...}) {
    final buildRunner = NativeAssetsBuildRunner(...);
    return dryRunNativeAssets(...);
  }
}

(The name is ugly I know, we could try to come up with a better name.)

HotRunner will now take NativeAssetsBuildRunnerRunner instead, and we pipe that dependency all the way from executable.dart.

What do you think?

I don't have any better solutions. 🙃

Change that allows excluding build_system internally is submitted here: #142760

Nice! So, so now we can move the NativeAssets extends back to where it was.

I'll take another spin at this on Monday. Have a nice weekend!

@dcharkes
Copy link
Contributor Author

dcharkes commented Feb 5, 2024

In this case specifically, run_hot.dart contains 3 references to native_assets.dart

  1. Reference to the type NativeAssetsBuildRunner.
  2. Invoking the constructor NativeAssetsBuildRunnerImpl.
  3. Calling dryRunNativeAssets.

I have solved 2. by moving the constructor call into isolated/ because the dryRunNativeAssets method is only invoked from run_hot.dart.

  1. Reference to the type NativeAssetsBuildRunner.

Can we define NativeAssetsBuildRunner to be an empty classes in g3? That should also pass all analysis.

  1. Calling dryRunNativeAssets.

This is the same issue as in test_compiler.dart

And 3.b. is that the methods we call als require in import statement in the top.

@dcharkes
Copy link
Contributor Author

dcharkes commented Feb 6, 2024

@chingjun I managed to make it green together with the patch CL for g3 that removes the imports. 🎉

We probably want to do some cleanup (naming, etc.). PTAL.

Copy link
Contributor

@chingjun chingjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice and it's actually cleaner than I imagined. Left some comments but mostly LGTM, thank you!

bool machine = true,
String? userIdentifier,
bool enableDevTools = true,
required HotRunnerNativeAssetsBuilder? buildRunner,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm slightly surprised that this startApp is not used from the same file. That makes the change easier than I initially thought :)

Copy link
Contributor

@christopherfujino christopherfujino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, let's ship it

@dcharkes dcharkes added the autosubmit Merge PR when tree becomes green via auto submit App label Feb 6, 2024
@auto-submit auto-submit bot merged commit a069e62 into master Feb 6, 2024
@auto-submit auto-submit bot deleted the move-native-assets-to-isolated branch February 6, 2024 20:59
@vashworth
Copy link
Contributor

@vashworth vashworth added the revert Autorevert PR (with "Reason for revert:" comment) label Feb 6, 2024
auto-submit bot pushed a commit that referenced this pull request Feb 7, 2024
@auto-submit auto-submit bot removed the revert Autorevert PR (with "Reason for revert:" comment) label Feb 7, 2024
auto-submit bot added a commit that referenced this pull request Feb 7, 2024
Reverts #142709

Initiated by: vashworth

Reason for reverting: `Mac tool_tests_general` started failing on this commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_tests_general/15552/overview

Original PR Author: dcharkes

Reviewed By: {christopherfujino, chingjun, reidbaker}

This change reverts the following previous change:
Original Description:
Native assets in other build systems are not built with `package:native_assets_builder` invoking `build.dart` scripts. Instead all packages have their own blaze rules. Therefore we'd like to not depend on `package:native_assets_builder` from flutter tools in g3 at all.

This PR aims to move the imports of `native_assets_builder` and `native_assets_cli` into the `isolated/` directory and into the files with a `main` function that are not used in with other build systems.

In order to be able to remove all imports in files used by other build systems, two new interfaces are added `HotRunnerNativeAssetsBuilder` and `TestCompilerNativeAssetsBuilder`. New parameters are then piped all the way through from the entry points:

* bin/fuchsia_tester.dart
* lib/executable.dart

The build_system/targets dir is already excluded in other build systems.

So, after this PR only the two above files and build_system/targets import from `isolated/native_assets/` and only `isolated/native_assets/` import `package:native_assets_cli` and `package:native_assets_builder`.

Context:

* #142041
@jmagman
Copy link
Member

jmagman commented Feb 7, 2024

Ah, the test actually failed in presubmit on this PR

 00:29 +3088 ~5 -1: test/general.shard/resident_runner_test.dart: use the nativeAssetsYamlFile when provided [E]                                                                                        
   Null check operator used on a null value
   package:flutter_tools/src/run_hot.dart 506:48         HotRunner._updateDevFS
   ===== asynchronous gap ===========================
   package:flutter_tools/src/run_hot.dart 290:40         HotRunner.attach
   ===== asynchronous gap ===========================
   test/general.shard/resident_runner_test.dart 2357:29  main.<fn>.<fn>
   ===== asynchronous gap ===========================
   test/src/testbed.dart 131:13                          Testbed.run.<fn>.<fn>.<fn>
   ===== asynchronous gap ===========================
   package:flutter_tools/src/base/context.dart 153:19    AppContext.run.<fn>
   ===== asynchronous gap ===========================
   package:flutter_tools/src/base/context.dart 153:19    AppContext.run.<fn>
   ===== asynchronous gap ===========================
   test/src/context.dart 151:26                          testUsingContext.<fn>.<fn>.<fn>.<fn>.<fn>
   ===== asynchronous gap ===========================
   package:flutter_tools/src/base/context.dart 153:19    AppContext.run.<fn>
   ===== asynchronous gap ===========================
   test/src/context.dart 140:22                          testUsingContext.<fn>.<fn>.<fn>.<fn>

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8756812476208304049/+/u/run_test.dart_for_tool_tests_shard_and_subshard_general/stdout

You maybe hit the brief window where the checks weren't reporting test failures: #142998

@dcharkes
Copy link
Contributor Author

dcharkes commented Feb 7, 2024

You maybe hit the brief window where the checks weren't reporting test failures: #142998

That's unfortunate, sorry for the noise!

I've fixed the test here: #143055.

dumazy added a commit to dumazy/flutter that referenced this pull request Feb 7, 2024
* master: (32 commits)
  [reland] Add `AnimationStyle` to `showSnackBar` (flutter#143052)
  Run Mac x64 build tests in postsubmit only (flutter#142334)
  Copy the flutter version JSON file into the simulated Flutter SDK used by update_packages (flutter#143035)
  Mark `Windows_android hot_mode_dev_cycle_win__benchmark` as no longer flaky (flutter#143016)
  Dispose precached image info (flutter#143017)
  Instrument CurvedAnimation. (flutter#143007)
  Update _goldens_io.dart to generate failure images during a size mism… (flutter#142177)
  Reverts "Activate InkSparkle on CanvasKit" (flutter#143036)
  Activate InkSparkle on CanvasKit (flutter#138545)
  [Windows] Fix signed/unsigned int comparison (flutter#142341)
  Reverts "Move native assets to `isolated/` directory" (flutter#143027)
  Reverts "Roll Flutter Engine from 808886312e2b to 07cdaab7f531 (18 revisions)" (flutter#143025)
  Make destructiveRed a CupertinoDynamicColor (flutter#141364)
  Move native assets to `isolated/` directory (flutter#142709)
  Roll Flutter Engine from 808886312e2b to 07cdaab7f531 (18 revisions) (flutter#143005)
  Fix CupertinoPageScaffold resizeToAvoidBottomInset (flutter#142776)
  Reverts "Add `AnimationStyle` to `showSnackBar`" (flutter#143001)
  Material 3 - Tab indicator stretch animation (flutter#141954)
  Add `AnimationStyle` to `showSnackBar` (flutter#142825)
  Roll Packages from ae3494d to 1a5a7ce (2 revisions) (flutter#142985)
  ...
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 7, 2024
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 7, 2024
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 7, 2024
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 7, 2024
auto-submit bot pushed a commit to flutter/packages that referenced this pull request Feb 7, 2024
flutter/flutter@e6ba809...8431cae

2024-02-07 [email protected] Roll Flutter Engine from 1ac6beb8a3c2 to 6807342305e4 (6 revisions) (flutter/flutter#143082)
2024-02-07 [email protected] Manual roll Flutter Engine from 808886312e2b to 1ac6beb8a3c2 (22 revisions) (flutter/flutter#143039)
2024-02-07 [email protected] Roll Packages from 1a5a7ce to e4ea6bf (4 revisions) (flutter/flutter#143076)
2024-02-07 [email protected] Fix M3 text field height + initial step for input decorator M3 test migration (flutter/flutter#142981)
2024-02-07 [email protected] [reland] Add `AnimationStyle` to `showSnackBar` (flutter/flutter#143052)
2024-02-07 [email protected] Run Mac x64 build tests in postsubmit only (flutter/flutter#142334)
2024-02-07 [email protected] Copy the flutter version JSON file into the simulated Flutter SDK used by update_packages (flutter/flutter#143035)
2024-02-07 [email protected] Mark `Windows_android hot_mode_dev_cycle_win__benchmark` as no longer flaky (flutter/flutter#143016)
2024-02-07 [email protected] Dispose precached image info (flutter/flutter#143017)
2024-02-07 [email protected] Instrument CurvedAnimation. (flutter/flutter#143007)
2024-02-07 [email protected] Update _goldens_io.dart to generate failure images during a size mismâ�¦ (flutter/flutter#142177)
2024-02-07 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Activate InkSparkle on CanvasKit" (flutter/flutter#143036)
2024-02-07 [email protected] Activate InkSparkle on CanvasKit (flutter/flutter#138545)
2024-02-07 [email protected] [Windows] Fix signed/unsigned int comparison (flutter/flutter#142341)
2024-02-07 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Move native assets to `isolated/` directory" (flutter/flutter#143027)
2024-02-06 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Roll Flutter Engine from 808886312e2b to 07cdaab7f531 (18 revisions)" (flutter/flutter#143025)
2024-02-06 [email protected] Make destructiveRed a CupertinoDynamicColor (flutter/flutter#141364)
2024-02-06 [email protected] Move native assets to `isolated/` directory (flutter/flutter#142709)
2024-02-06 [email protected] Roll Flutter Engine from 808886312e2b to 07cdaab7f531 (18 revisions) (flutter/flutter#143005)
2024-02-06 [email protected] Fix CupertinoPageScaffold resizeToAvoidBottomInset (flutter/flutter#142776)
2024-02-06 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Add `AnimationStyle` to `showSnackBar`" (flutter/flutter#143001)
2024-02-06 [email protected] Material 3 - Tab indicator stretch animation (flutter/flutter#141954)
2024-02-06 [email protected] Add `AnimationStyle` to `showSnackBar` (flutter/flutter#142825)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human
is aware of the problem.

To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
auto-submit bot pushed a commit that referenced this pull request Feb 8, 2024
Reland of #142709.

The revert of the revert is in the first commit, the fix in the commit on top.

The move of the fakes for packages/flutter_tools/test/general.shard/resident_runner_test.dart was erroneous before, as it was trying to use setters instead of a private field. This PR changes the private `_devFS` field in the fake to be a public `fakeDevFS` in line with other fakes.

## Original PR description

Native assets in other build systems are not built with `package:native_assets_builder` invoking `build.dart` scripts. Instead all packages have their own blaze rules. Therefore we'd like to not depend on `package:native_assets_builder` from flutter tools in g3 at all.

This PR aims to move the imports of `native_assets_builder` and `native_assets_cli` into the `isolated/` directory and into the files with a `main` function that are not used in with other build systems.

In order to be able to remove all imports in files used by other build systems, two new interfaces are added `HotRunnerNativeAssetsBuilder` and `TestCompilerNativeAssetsBuilder`. New parameters are then piped all the way through from the entry points:

* bin/fuchsia_tester.dart
* lib/executable.dart

The build_system/targets dir is already excluded in other build systems.

So, after this PR only the two above files and build_system/targets import from `isolated/native_assets/` and only `isolated/native_assets/` import `package:native_assets_cli` and `package:native_assets_builder`.

Context:

* #142041
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a: desktop Running on desktop autosubmit Merge PR when tree becomes green via auto submit App tool Affects the "flutter" command-line tool. See also t: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants