@simolus3 is there a way now (e.g. option) to force disable native asset compilation for flutter tests? If not, then having clang installed might have a negative impact on tests performance and they might behave differently depending on where they are run. At least if I understand correctly the native modules are compiled when the buildtools are found, aren't they?

is there a way now (e.g. option) to force disable native asset compilation for flutter tests?

There isn't, and (as mentioned in the issue) I don't think there should be one. If a package has a build hook, it can expect that hook to run whenever the package is used. So having an option to not run hooks would easily break packages (I know you want to mock the package anyways, but this still violates assumptions).

At least if I understand correctly the native modules are compiled when the buildtools are found, aren't they?

We still unconditionally invoke build hooks. It's just that if we don't find a compiler toolchain, we will let each hook worry about finding a compiler on its own (if it needs one). Previously, we'd immediately fail the build instead.

The sqlite3 package for instance does not do any native compilation by default, it downloads a precompiled binary instead. And since a native toolchain is not required since this PR, I imagine most other packages using hooks will adopt a similar approach.
If you want to skip the build for sqlite3 entirely, you can use something like this. But adding these escape hatches is the responsibility of a package author using hooks, not of the tool invoking them.

There isn't, and (as mentioned in the issue) I don't think there should be one. If a package has a build hook, it can expect that hook to run whenever the package is used. So having an option to not run hooks would easily break packages (I know you want to mock the package anyways, but this still violates assumptions).

My problem is: when more packages start to use build hooks (see there are a couple of them already example projects), then running any flutter test will build all the assets no matter whether I use them or not (overhead; a benchmark would be nice to have). The assumption for testing is that no matter where/when I run tests, they must behave the same. If not, then the tests are flaky. Seems like native assets (so flutter 3.38) might make all tests flaky according to your explanation as they will compile and use native assets if they can, otherwise they just run without them (so the output depends on the runner env).

If I change the download url in your code by an invalid one (similar to when the right one is down or blocked), then running flutter test fails with Building native assets failed. See the logs for more details. even if I don't use your package, but it is added as a dependency. If the test still ran then it would have behaved differently than when the native assets were there. But currently as you can see, when the download fails, then I can't run any test in a project ⚠️

You may say caching the downloaded stuff would minimize the likelihood of this happening, but as I see outputDirectoryShared points to the project's .dart_tool directory instead of PUB_CACHE or something controllable. When running tests in CI, each job will trigger each packages' downloads everytime as caching a project directory is not that trivial.

The sqlite3 package for instance does not do any native compilation by default, it downloads a precompiled binary instead.

This looks like a very big security hole tbh. Npm has already been there. So now any package can start to download anything from the net (as I see there is no central repository for these; e.g. sqlite tries to download from Github, but this is up to the plugin authors what do they use and if they check checksums after downloading) and execute basically anything when building a flutter app or just running its tests without any visible indicator? Even transient dependencies can do this?

Sorry for being picky, but I just realized these things the hard way after upgrading a prod project to 3.38. I'm not blaming you (or anyone) for any of these. I'm not involved in the native assets/code_assets developments, so I can't really recommend good/optimal solutions for my problems, sadly all I can do is to highlight my concerns which may look rude (not intended), but I still want to make them transparent. I don't think this PR could/should fix the mentioned core issues, just saying that this PR only improves the situation if I intentionally don't install clang or other build tools, then I might be able to get the intended/recommended behavior for tests (except the non-transparent precompiled stuff downloading). In my opinion not trying to compile/download anything for tests would solve many problems. @stuartmorgan-g can I summon you here to share your thoughts on this please? I know you have a lot of experience with testing/CI and these things.

A breaking change entry is definitely missing for 3.38 regarding this native assets/hooks topic.

There isn't, and (as mentioned in the issue) I don't think there should be one. If a package has a build hook, it can expect that hook to run whenever the package is used. So having an option to not run hooks would easily break packages (I know you want to mock the package anyways, but this still violates assumptions).

Please continue the discussion on a relevant issue:

• [hooks] Exclude a platform from being built by dependency's build hook dart-lang/native#2480 (comment)

My problem is: when more packages start to use build hooks (see there are a couple of them already example projects), then running any flutter test will build all the assets no matter whether I use them or not (overhead; a benchmark would be nice to have). The assumption for testing is that no matter where/when I run tests, they must behave the same. If not, then the tests are flaky. Seems like native assets (so flutter 3.38) might make all tests flaky according to your explanation as they will compile and use native assets if they can, otherwise they just run without them (so the output depends on the runner env).

If I change the download url in your code by an invalid one (similar to when the right one is down or blocked), then running flutter test fails with Building native assets failed. See the logs for more details. even if I don't use your package, but it is added as a dependency. If the test still ran then it would have behaved differently than when the native assets were there. But currently as you can see, when the download fails, then I can't run any test in a project ⚠️

At the risk of repeating myself. This was already the situation with plugins when doing flutter build and flutter run. This merely makes flutter test consistent with the other commands relieving the need for flutter drive and relieving the need to even stub dependencies out when they can run on the host.

You may say caching the downloaded stuff would minimize the likelihood of this happening, but as I see outputDirectoryShared points to the project's .dart_tool directory instead of PUB_CACHE or something controllable. When running tests in CI, each job will trigger each packages' downloads everytime as caching a project directory is not that trivial.

The final goal is to have the assets hosted on pub as well:

• Native assets on pub dart-lang/pub#3693

In the meantime, package authors can use the approaches listed in:

• [native_assets_cli] Input.outputDirectorySystemShared dart-lang/native#1962 (comment)
  • Upload your dylibs in the package via pub publish
  • Use a user-define to let users point to a cache dir (so they know where to clean it).
  • Use a global cache and provide a bin/clean.dart.

The sqlite3 package for instance does not do any native compilation by default, it downloads a precompiled binary instead.

This looks like a very big security hole tbh. Npm has already been there. So now any package can start to download anything from the net (as I see there is no central repository for these; e.g. sqlite tries to download from Github, but this is up to the plugin authors what do they use and if they check checksums after downloading) and execute basically anything when building a flutter app or just running its tests without any visible indicator? Even transient dependencies can do this?

This was already the case with Flutter plugins, you can shell out to arbitrary scripts from the Gradle/CocoaPods/SPM build scripts. (And various packages were doing this to run shell scripts to download prebuilt dylibs.) Build hooks equalize the flutter test to be consistent with flutter build and flutter run.

If you have specific feature requests please file them on dart-lang/native where most of the shared code (between Dart and Flutter) for build hooks lives.

Let's keep the discussion on this issue to the issue that this PR is solving, we are unanimous that this PR is an improvement over the pre-PR situation!

@stuartmorgan-g can I summon you here to share your thoughts on this please?

This PR seems reasonable to me; unconditionally failing due to lack of a tool we are likely not to need is clearly not ideal behavior.

I'm happy to discuss the general topic further on a relevant issue.

A breaking change entry is definitely missing for 3.38 regarding this native assets/hooks topic.

That's also not related to this PR, but to save the trouble of filing a new issue that will be closed: adding new, opt-in functionality is by definition not a breaking change. There might need to be more guidance to package authors about how to adopt it without affecting clients, but that's not the same thing.

Failed to create CP due to merge conflicts.
You will need to create the PR manually. See the cherrypick wiki for more info.