[hooks] build.dart / link.dart web backends and WasmCode and JsCode assets

@mosuem

At some point, we'd like to output assets for web:

DataAsset ([native_assets_cli] data assets #154)
WasmCode (Compilation of native code to WASM #156)
JSCode

And WasmCode and it should have a dialect field that is linear_memory, wasm_gc. (I'm assuming the linking might work differently for the two dialects.)

Besides asset types to output, we should also have the BuildConfig communicate that we're targeting "web" instead of "native".
The question is how we should surface this in the protocol and Dart API. And this depends on differences in target.

I believe build.dart and link.dart should consider dart compile js the same as flutter build web and dart compile wasm the same as flutter build web --wasm. I believe it would be an anti-pattern if build.dart or link.dart would branch on Dart vs Flutter, if we want packages to work both in Dart and Flutter. (It's fine to branch on web vs Android vs MacOS.) #946 (comment)

@mosuem suggested to extend the API with: OS.web, Architecture.js and Architecture.wasm (and extend the protocol ditto).

I don't know enough about dart2js/dart2wasm and Flutter for web to know if it's reasonable to let users branch on whether we're compiling dart to js or wasm.
I'm thinking that theoretically it might be enough to just introduce OS.web, and have the dart-js-interop and dart-wasm-interop work for any backend (js or wasm). However, that would require the dart code interoperating with js and wasm to work with both the js and wasm backends. In practice, I expect dart2js to not support wasm interop but to require a wrapper JS layer. Therefore the build.dart would need to output that wrapper when targeting Architecture.js. Hence, we need to introduce Architecture.js and Architecture.wasm alongside OS.web.

@eyebrowsoffire @mkustermann @osa1 Could you comment on how WASM and WASM GC code files are linked into a flutter build web --wasm / flutter build web / dart compile wasm application? (I'm aware that we can currently only have a single linear memory module, but conceptually how would this work?) Pointers to examples and the implementation in the Dart and Flutter SDK would be greatly appreciated. 🙏

@sigmundch Who is the right person to ask about the (new) JS interop? Does the (new) JS interop work seamlessly when compiling Dart to JS and compiling Dart to WASM? Does dart compile js have any support for including WASM code in a web app? (e.g. if Architecture.js should we allow outputting WasmCode assets at all?

Note, we don't need to answer all these questions now, or commit to supporting this. I just want to make sure that while designing the API we design it in a way so that we can support these use cases later.

@srujzs

Who is the right person to ask about the (new) JS interop?

@srujzs is our main point of contact 😃

Does the (new) JS interop work seamlessly when compiling Dart to JS and compiling Dart to WASM?

That's the goal, for the most part. There are some caveats/inconsistencies that we have to live with, though (e.g. like differences between null/undefined, different default conversions of arrays, among others). That aside, yes, the new dart:js_interop model is meant to work on both. This is what enables the future flutter web based on dart:js_interop and package:web (DOM APIs layered on top of this new interop) to work on both JS and WasmGC seamlessly.

That said, the JS interop story centers on accessing JavaScript from Dart and not on accessing Wasm modules. Today, we don't have much to support the latter. Users would need to manually create an export layer and expose it via JavaScript as another JS interop interface. We've considered that a JS-2-wasm and wasm-2-wasm interface is more aligned with a Dart-2-native interface, and as such, it we could potentially use dart:ffi's syntax in the future to support it.

Does dart compile js have any support for including WASM code in a web app?

Nope. Nothing is done automatically, if users manage their own assets and provide a JS interface, then that works.

It's probably worth noting that we also don't do any management for JS assets, either. Today most developers manage this by hand. Some include assets in their lib/ folders with the Dart code to allow sharing across packages. This is not ideal as it can interleave Dart and JS build toolchains unnecessarily. Internally, in bazel, we do have some BUILD macros that assist in the process and allows disentangling the dependencies. This is then used to link all js assets when creating application targets.

@dcharkes

I believe build.dart and link.dart should consider dart compile js the same as flutter build web and dart compile wasm the same as flutter build web --wasm

dart compile wasm is similar to dart compile exe - it's mainly focused on compiling dart code itself.

Wouldn't dart build <js/exe/...> (the new command @dcharkes added under experimental flag) be the analogous to flutter build web?

I believe it would be an anti-pattern if build.dart or link.dart would branch on Dart vs Flutter, if we want packages to work both in Dart and Flutter.

Generally true, but the flutter build command may offer more functionality, more asset types, etc than dart build does. When building for Android it may e.g. allow building & including java code for example.

@eyebrowsoffire @mkustermann @osa1 Could you comment on how WASM and WASM GC code files are linked into a flutter build web --wasm / flutter build web / dart compile wasm application? (I'm aware that we can currently only have a single linear memory module, but conceptually how would this work?) Pointers to examples and the implementation in the Dart and Flutter SDK would be greatly appreciated. 🙏

You can call dart compile wasm hello.dart and see the output: It will produce an hello.mjs file that can e.g. be used by an index.html (like flutter does). This hello.mjs file has an exported instantiate function that can given the wasm module and things that module needs (via imports). It also exposes a invoke function that will invoke the main function.

So outside code (like flutter or non-flutter) is responsible for

loading the hello.wasm file
loading the hello.mjs file & calling instantiate and providing all necessary imports
calling the invoke

Any used linear memory e.g. is provided to the wasm module via the imports (e.g. a WebAssembly.Memory). A wasm module could import functions that are exported by another wasm module. Then multiple wasm modules can operate on the same memory.

Though one may think about whether it would make sense to statically link wasm modules together.

Does the (new) JS interop work seamlessly when compiling Dart to JS and compiling Dart to WASM?

The new JS interop is designed for Dart code to interop with JavaScript. Yes, ignoring a few corner cases, code written using the new JS interop should behave the same whether compiled with dart2js or dart2wasm.

Though none of the two have an official interop with linear memory wasm (dart2wasm supports a subset of dart:ffi - sufficient enough to interact with skwasm/skia - but this isn't supported to 3rd party users yet).

@MichealReed

Some sketches based on the above comments and the following.

Well, adding an OS.web and adding support for code assets in web builds are two different issues ;)

At least for the near-term, for use with flutter, we would only need the produced wasm and js of a emscripten compiled library moved to the web build folder. This would at least allow us to build data assets. Through the html template, the library can be included and invoked.

Perhaps a similar approach could be considered where dart build web provides a html template that includes relevant js/wasm libraries (rather than trying to link it all into a fat library based on if js or wasm), which then frameworks could include where appropriate to make the calls available?

Originally posted by @MichealReed in #2108 (comment)

The new JS interop [...]

According to this, JSAssets should be made available in the global namespace, so most likely only a file is needed for JS assets

The hook output should then have JSAssets in a way like this.

{
  "assets" : [
    {
      "type": "js",
      "file": "output_dir/some_file.js"
    }
  ]
}

(For CodeAssets we use the @Native(assetId: "my_asset") to avoid having to have all C functions in a global namespace. But apparently this is less common for JS?)

[...] to work on both JS and WasmGC seamlessly.

So, for JS-assets we don't really need to know if the web_target is js or wasm_gc.

In that case, it would be enough to simply add js to the build_asset_types in the input:

{
  "config": {
    "build_asset_types": [ "js" ]
  }
}

For linear-wasm assets, there's a question whether we're interoperating with them through the dart:ffi interface (dart2wasm-only, not yet available for 3rd party) or via a JS wrapper.

If it's via JS wrapper, then they should probably be in the global namespace, but if used via dart:ffi they need an asset_id.

The input then needs to signal whether dart:ffi is available (dart2wasm, when we make it available for 3rd party). So for such potential future:

{
  "config": {
    "build_asset_types": [ "wasm" ],
    "wasm" : {
      "supported_link_modes": ["js_wrapper", "dart_ffi"]
    }
  }
}

(Instead of having config for if it's dart2js or dart2wasmgc, we put in the config what ways WASM assets can be consumed in such target. In the situation some SDK ships a WASM runtime next to the Dart VM that SDK might actually support WASM assets even though it's not dart2js or dart2wasm!)

For signalling if WASM assets are meant for the global namespace through a JS wrapper or for use via an asset ID via @Native in dart:ffi we probably want a link_mode for WASM assets.

{
  "assets": [
    {
      "type": "wasm",
      "file": "output_dir/some_file.wasm",
      "link_mode": {
        "type": "js_wrapper"
      },
    },
    {
      "type": "wasm",
      "file": "output_dir/some_other_file.wasm",
      "link_mode": {
        "type": "dart_ffi",
        "asset_id": "package:my_package/src/use_in_foo.dart"
      }
    }
  ]
}

This is a rough whiteboard sketch. If there requirements not satisfied with this sketch, please give a shout.

So, for JS-assets we don't really need to know if the web_target is js or wasm_gc.

We might not need to know this for either with a way to target web via dart (instead of js or wasm).

(Instead of having config for if it's dart2js or dart2wasmgc, we put in the config what ways WASM assets can be consumed in such target. In the situation some SDK ships a WASM runtime next to the Dart VM that SDK might actually support WASM assets even though it's not dart2js or dart2wasm!)

The wasm blob should be usable in both contexts. If wasm is provided for web, as far as I know, bindings to JS would need to be produced to work with js_interop. If wasm is provided for a native context it should be usable without such bindings.

Very nice sketch!

@srujzs

Just wanted to say that I would really love to see this feature, and it'd be helpful for Dart code depending on JS code/NPM packages.

When I was doing GSoC with @srujzs this past summer, when working on the web_generator project, one of our main concerns for post-project was how code would be bundled for users, especially considering some of the ways we could do this (like an intermediate js file, using direct imports, having it accessible to the global namespace (for web only)), and the fact that some TS APIs would need the output JS code to be connected to the Dart code in a certain way. It would be nice if such could be handled by the build hooks API, so that, in the same way users no longer have to call DynamicLibrary.open() to make use of native FFI APIs, users would also not have to manually handle assets and can just have them accessible, and we would know how such APIs would need to be connected to the JS code, making generation work easier.

The benefits could extend to non-web environments, as well, such as in Node.

Would love to help out in getting JS/WASM included for the hooks API (including builders) if possible.

@srujzs

I'd love to have someone work on this! 🙏

My gut feeling is we should focus on JS assets first, and then later look at WASM assets. (WASM is still in flux, and there's wasm gc, wasm non-gc etc.)

I think the way that this should work is:

Start a package js_asset, and define a JSAsset type. (Similar to how data_asset's DataAsset works.)
Add an experiment flag to Dart and Flutter. (You can look at the data assets experiments in package:dartdev and package:flutter_tools. Relevant code: https://dart-review.googlesource.com/c/sdk/+/450781 Add data assets flutter/flutter#174685).
The interesting part: Figure out how correctly wire up the JSAssets outputted from the hook to make them callable both in Flutter for web (with the js development compiler, and the deployment wasm compiler), and dart2js, and dart2wasm.
a. Maybe we need to come up with a syntax which uses asset IDs like we use for code assets to connect the Dart code with the JS asset.
b. In Flutter possibly we should add a JS asset manifest json (similar to how we have an asset manifest json and native asset manifest to json).
c. In dart2js and dart2wasm output I'm not familiar with how things are set up.

I'm happy to guide you! Step 1 and 2 are clear to me. For step 3 we could probably use some more input from @srujzs and others.

@dcharkes

Agreed that now that we have a generator for .d.ts files, being able to bundle assets would be quite nice.

(For CodeAssets we use the @Native(assetId: "my_asset") to avoid having to have all C functions in a global namespace. But apparently this is less common for JS?)

The use of the global namespace is an existing issue for JS interop today. It's presumably less of an issue because you can put things in nested namespaces, but there's nothing forcing users to use that practice and even if they did, collisions can still occur. So if you're an app that has lots of JS interop libraries, it's a bit of a dance to set up the environment properly such that every library works and you hope there are no collisions. We've looked at ways we can improve this in the future, including configuring modules/libraries in the Dart runtime before calling main, but for now, putting libraries in the global namespace via some JS glue code is the only way.

I'll need to a bit of digging to see how we can wire this up in Flutter web. I'm imagining the build hooks will output the JS glue code file that when loaded will load modules in the appropriate namespaces, and that will need to be loaded before main is called. I'll set something up with you at some point to get more info, @dcharkes.

I also need this, and I would be happy to contribute if this is not currently being worked on.

The hook output should then have JSAssets in a way like this.

So if I read these comments correctly, basically we only expose some kind of relative URL for JSAssets, correct? And then Flutter would potentially reference them in an HTML template, and other users can do things like importModule() if that's sensible for them?

I wonder if in this case we really need a distinction between JS and WASM assets. Obviously WASM assets are different in that there's a potential future where dart2wasm emits object files and then something like wasm-ld does static linking, but that doesn't seem realistic in the short term. Similarly, JSAssets feel special in the sense that users might expect them to be bundled into the main JS file emitted by a webdev or Flutter build.

If all we expose is a URI and let either the embedder or the package worry about what to do with it, maybe the asset type could just be called something like WebUriAsset? Then in the future we could still have a JSModuleAsset or a WasmAsset with different linking modes (as object files, wasm components, ...). But WebUriAsset would just be "here's a file, give me a URI to it at runtime" and my package gets to interpret these however that makes sense:

For external JS libraries, I would use a dynamic ESM import(uri).
For shared or dedicated workers, I would use new Worker(uri).
For custom WebAssembly assets, I would use WebAssembly.instantiateStreaming(fetch(uri), ...).

I'd very much appreciate if a solution were possible that could work across environments (i.e. not just in Flutter, or webdev, but also in non-web environments). If need be, we could probably make improvements if running on Flutter Web, although there probably won't be much needed.
In that case, the most likely of solutions would be to have the code bundled in the output JS file (or for dart2wasm, the output WASM file).

...and other users can do things like importModule() if that's sensible for them?

It probably may be up to the user if the user would want to do that, but if code were being generated, for instance, then I don't think users should import() the code (similar to how things already are in Dart C FFI).

For me, one motivation was that I want to provide a worker file to run from the application, which would be "JavaScript asset" but one that must never be linked into the main program (because I need the separate URI).

Providing URLs lets you import the code and load it independently for say a web worker, but if Dart were to bundle these assets automatically, we could only import the code into the main application. So I think providing URIs first and then adding specialized asset types to do bundling later might be easier, and it still gives you access to everything (even though it's a bit more manual). Outside of web environments, you could still resolve the URL against the current script path on node/bun.

Ah, that makes sense. This would also be a similar issue if you were to try using concurrency in any JS runtime at the moment.

So I think providing URIs first and then adding specialized asset types to do bundling later might be easier, and it still gives you access to everything (even though it's a bit more manual). Outside of web environments, you could still resolve the URL against the current script path on node/bun.

After reading #2829, the concerns still hold somewhat. In order to support this, we'd now have to make platform-specific APIs (possibly for assets) for getting the URL. I'm in support for that, mainly because as long as we use relative file URLs, JS/WASM code requiring passing URLs or file paths should work correctly, including dynamic imports (if distributed as a JS package).

(While not the best way, it could be possible to support this by making a URI from the raw ByteAsset/StringAsset, but that would be a last-effort choice anyways)

@srujzs

If all we expose is a URI and let either the embedder or the package worry about what to do with it, maybe the asset type could just be called something like WebUriAsset? Then in the future we could still have a JSModuleAsset or a WasmAsset with different linking modes (as object files, wasm components, ...). But WebUriAsset would just be "here's a file, give me a URI to it at runtime" and my package gets to interpret these however that makes sense:

For external JS libraries, I would use a dynamic ESM import(uri).

For shared or dedicated workers, I would use new Worker(uri).

For custom WebAssembly assets, I would use WebAssembly.instantiateStreaming(fetch(uri), ...).

I like the idea of a WebUriAsset! We don't do any specific JS or WASM "linking" (in a broad sense, e.g. including no minifying etc.)

The embedder must be responsible for (1) bundling the assets, and (2) implementing an API that takes an asset-id and returns the uri.

We need to decide (A) where to introduce that API, and (B) how embedders get to implement that API.

RE (A):

dart:something: Uri webUriAssetUri(String assetId).
pro: dart: signals it's something embedder-agnostic that multiple embedders can implement.
con: introducing a dart: will require significant politics.
package:web_assets: static Uri WebUriAsset.runtimeUri(String assetId).
pro: same class as used for reporting the asset in the hook.
con: a package: that has special magic in the compiler/embedder to hook things up.

RE (B):

Every backend needs to register some kind of callback that does the resolution. IIRC the dart2js output has some logic to register callbacks. @srujzs

I'd very much appreciate if a solution were possible that could work across environments (i.e. not just in Flutter, or webdev, but also in non-web environments).

I think for that we would need a JSModuleAsset or a WasmAsset. Because for non-web targets the embedder must link the code.

My feeling is that tackling WebUriAssets first before JSModuleAsset or a WasmAsset is the easiest route.

Also, I have a slight feeling that there might be some overlap between the WebUriAsset and DataAssets. The way that data assets currently work in Flutter only give you access to the bytes or the string contents. However, for some use cases one might want access to the file path on platforms where a file path is supported. If you want to write truly platform independent code you might want to use a web uri asset for an image for web targets and a file data asset non-web targets (e.g. flutter desktop, but also node?)

DataAsset: only gives access to the bytes. Works on every imaginable backend. Access to bytes via assetId (possibly this must be async with a Future).
FileAsset: only gives access to a file path. Works on targets in which files can be bundled and the compiled Dart code has the permissions to read those files. Access to file path via assetId.
WebUriAsset or WebAsset: only gives access to a web url. Works on web targets. Access to uri via assetId.
JsModuleAsset: Can link / minify. Actually understands the JS modules. Works on targets that have a JS runtime. Should be integrated with the js interop directly: In a way similar to @Native(assetId: '...') external functions in dart:ffi.
WasmAsset: ditto to JsModuleAsset, but for targets that have a wasm runtime.

Possibly supporting WebUriAsset or FileAsset implies supporting DataAsset, since DataAsset is implementable in terms of the others.

And I'd like this design to fit in with other SDKs/embedders aswell. Jaspr should be able to support WebUriAssets. cc @schultek

decide (A) where to introduce that API

Why is the method a Uri webUriAssetUri(String assetId), wouldn't we have to guarantee that String assetId is a compile-time constant? To also take custom tree-shaking behavior via link hooks into account, it seems easier to expose this as an annotation and let the language guarantee everything is a constant? Perhaps something like:

@WebUriAsset('package:sqlite3/web/wasm_asset_id')
external Uri get sqlite3Wasm;

If it's an annotation, we could also start with something like

@pragma('dart:unstable-web-uri-asset', 'package:sqlite3/web/wasm_asset_id')
external Uri get sqlite3Wasm;

and then turn it into either a package or someting like a dart:assets(?) library once it's closer to being stable.

Jaspr should be able to support WebUriAssets

All Dart web frameworks should support WebUriAssets, and I don't think duplicating this logic in every one of them is reasonable. Jaspr already relies on build_runner (and specifically build_web_compilers) to emit assets, so that feels like a better place to implement this. Changing build_runner to also run hooks if a compiling builder is involved is probably a large change, but it would be nice if all build_web_compilers / webdev-based workflows could just work.

Why is the method a Uri webUriAssetUri(String assetId), wouldn't we have to guarantee that String assetId is a compile-time constant?

I like your thinking, and that's exactly where we started when reasoning about various asset types.

However, guaranteeing that it is a compile-time constant doesn't work for building abstractions on top of assets. That's why the record_use and link-hooks is a separate feature. You'd write:

@RecordUse()
Uri get sqlite3Wasm => WebAsset.runtimeUri('package:sqlite3/web/wasm_asset_id')

Yes, that does require you to map sqlite3Wasm map to 'package:sqlite3/web/wasm_asset_id'. However it alos enables writing.

// in package:sqlite3
Uri get sqlite3WasmDynamic => WebAsset.runtimeUri('package:sqlite3/web/wasm_asset_id')

// in package:bla

@RecordUse()
dynamic getDatabase(String database = 'sqlite3') {
  switch(database) {
    case 'sqlite3':
      return sqlite3WasmDynamic;
    // ...
  }
}

In which case package:blas hook/link.dart can translate its higher level abstraction to the lower level abstractions of other packages and send its info to the hook/link.dart of package:sqlite3.

It's the same reason why for tree-shaking for CodeAssets users will also have to write @RecordUse() @Native() external function.

All Dart web frameworks should support WebUriAssets, and I don't think duplicating this logic in every one of them is reasonable.

Agreed. Probably dart2js/webdev needs to output something that can easily be reused by all frameworks. And if dart2js is used from build_runner, then we should be able to write a builder to automate that. However, I don't know too much about the web framework builds, so please correct me if I'm wrong.

From what I'm understanding (correct me if I'm wrong), JSAsset and WasmAsset will inherit from WebAsset?

From what I'm understanding (correct me if I'm wrong), JSAsset and WasmAsset will inherit from WebAsset?

No, JS and Wasm assets should work in a non-web-context as long as there is a JS or Wasm runtime. E.g. Wasm assets should work if you have DartVM+Wasmer.

I don't think there should be an inheritance relationship. Theoretically a DataAsset (available as bytes) could be also available as WebAsset or FileAsset (a way to read a file which can get you the bytes), but we can't express that as an inheritance relationship. The developer writing the hook should decide as what kind of asset type an asset is available if there are multiple options, and then use that asset in the code at runtime in the correct way.

I keep stumbling on this: For packages providing additional web assets to load, having this would be so useful as the current alternative is to provide external download links and ask users to re-download files after upgrading the package which is a terrible experience all around.

It sounds like this will require support across a few SDK and non-SDK components, so I wanted to start by sketching out a rough implementation plan based on what was discussed in this issue. Looking at the concept of uri assets (hooks contribute files to bundle with the app, the only way to access them at runtime is by getting a relative URL) first, it looks like the steps would be to:

In this repository, add a package:web_assets:
- Its runtime API would be Uri webUriAssetUri(String assetId) (there was a "same class as used for reporting the asset in the hook" static method idea in this thread, but I assume we don't want package:hooks to be a transitive import of compiled apps? At least it feels kind of wrong to me, happy to be corrected here). In fact, I don't think we need a new asset type for this at all since a DataAsset gives us every thing we need? The runtime API would essentially interpret data assets as URLs.
Implement support for this in dartdevc, dart2js and dart2wasm.
- Maybe the implementation would be to give webUriAssetUri a throwing body, then annotate it with a pragma to make it identifiable and compatible compilers would have a performModularTransformationsOnLibraries to replace the body with a runtime lookup? We can't pass available assets to the compiler directly since link hooks running afterwards might contribute assets too.
- We would then make available assets available at runtime:
  - For dart2wasm, allow providing assets as a parameter in the support .mjs files, embedders would pass them there as a JS object.
  - For dart2js, expect a global named dartUriAssets to be defined, embedders would add its definition as a prefix to the compiled JavaScript fle.
  - I'm not familiar with ddc but I'm sure there's a way to do that there as well?
Support this in Flutter web, which should be fairly easy to do by copying data assets into the final bundle.
Support this in build_web_compilers (effectively the only other Dart web build tool apart from Flutter web).
- I can see some controversy here, builders like build_web_compilers should be hermetic while build/link hooks are allowed to download files and more. It wouldn't be hard for build_web_compilers to "cheat" by invoking the hooks build runner in the root directory, but I don't know how a proper hooks <-> build_runner integration would look like (especially in this case where builders would pass their own custom config). It might be worth discussing that first, should I open a dart-lang/build issue for that?

Again, I'm happy to contribute at least parts of this, but given the complexity it would be good to agree on a plan first.

but I assume we don't want package:hooks to be a transitive import of compiled apps?

That's an interesting observation. For CodeAssets we have @Native in the Dart SDK. And for DataAssets (experiment) we have the Flutter asset bundle and no API yet for Dart. So, so far we don't have a transitive import of package:hooks.

For the runtime API, the question is who provides it, and who implements it.

When using Flutter for web, the implementor should be flutter_tool. Probably making the code that it generates that calls instantiate on the dart compile wasm output.

However, for "third party" wrappers of the Dart SDK, these will generate some code that calls the instantiate function.

In fact, I don't think we need a new asset type for this at all since a DataAsset gives us every thing we need? The runtime API would essentially interpret data assets as URLs.

I don't think making it data assets works. The communication between the Dart SDK and third parties should communicate what the intent of asset type is and the Dart SDK commands should list what asset types are supported in the hooks. E.g. dart build cli should not support JS and WASM assets, but it should support code and data assets.

In this repository, add a package:web_assets:

Yes, and that's then for the communication between the Dart SDK and packages. So these asset types need to refer to a file (like Data assets) when being reported.

For the runtime API, the question is who provides it

For Data assets we have a prototype that provides it in a package, and then recognize it in the Dart compiler. I think this is rather ugly. The Dart compiler should not know about external packages! So, I'd much rather define the runtime API in dart:assets.

If we do end up defining the runtime API in the package, we can avoid importing hooks at runtime by having the runtime API in package:web_assets/runtime.dart. And having the API for the hooks defined in package:web_assets/hooks.dart (or package:web_assets/web_assets.dart).

Maybe the implementation would be to give webUriAssetUri a throwing body, then annotate it with a pragma to make it identifiable and compatible compilers would have a performModularTransformationsOnLibraries to replace the body with a runtime lookup?

Yeah if the runtime API is defined in a package, we'll need to replace the implementation in the compiler.

The implementation should be wired to the embedder API. E.g. a function from asset id to uri. That function will need to be provided in:

That way flutter web's and build_web_compilers/web and Jaspr can provide the mapping.

Since the dart does not provide a way to build a web application end-to-end dart build web does not really make sense. Rather, the Dart SDK can provide a bundle that contains (1) the compiled Dart code, (2) all the web assets, and (3) a JSON mapping asset-ids to web assets file paths.

If the the JS and WASM compiled code is close enough we could have dart build web-bundle and dart build web-bundle --dart-target=js (WASM as default). or if they are vastly different maybe something like dart build js-bundle and dart build wasm-bundle.

I can see some controversy here, builders like build_web_compilers should be hermetic while build/link hooks are allowed to download files and more. It wouldn't be hard for build_web_compilers to "cheat" by invoking the hooks build runner in the root directory, but I don't know how a proper hooks <-> build_runner integration would look like (especially in this case where builders would pass their own custom config). It might be worth discussing that first, should I open a dart-lang/build issue for that?

Yeah, we don't have any watcher mode for running build hooks. Supporting assets from hooks for webdev and jaspr means you cannot have a build_runner style where your app bundle stays 100% up to date.

dart build web-bundle is similar to dart build cli or flutter build bundle where you AOT compile a whole program and end up with a standalone app bundle ready for distribution/deployment or a standalone file bundle consumed by another bundling tool.

For data-assets Flutter run (JIT mode, development mode) re-runs the hooks for data assets when you do hot restart or hot reload. So that updates assets on-demand in the dev cycle. https://github.com/flutter/flutter/blob/master/packages/flutter_tools/test/integration.shard/isolated/dart_data_asset_flutter_run_test.dart

The way that webdev should handle assets on startup is to run the hooks (via dart build web-bundle --dart-target=none which omits compiling the Dart code to JS or WASM), and then copy all the asset files to the output dir and register the mapping from asset ID to the instantiate function.

I think we'll likely want to add a "rerun hooks" R to webdev so that we can rerun the hooks from there (rather than automatically running them on file changes). Similar to the flutter for web experience. And the R can run dart build web-bundle --dart-target=none again to get the new list of assets and JSON mapping from asset id to file path.

Sorry for the big blurp, lots of thoughts on lots of different aspects of the whole stack. Any aspects of this plan which you think might not work?

Will the package assets keep their names and relative layout OR is the expectation that hooks pre-bundle multi file output into a single file?

A non dart file 1 need to find file 2, resolve import "./file2.mjs" relative to file 1 URL, so asset lookup is not consulted. emscripten and generator output (js + wasm files) is what most hooks will emit for web.

Thanks for the details! The only thing that stands out to me is:

So, I'd much rather define the runtime API in dart:assets.

I assume we'd need more than one library for this though, right? webUriAssetUri wouldn't work on native, so I assume there would eventually be a dart:assets for data assets (cross-platform, loading them as bytes), a dart:web_assets (acquiring a URL for them, and linking JsCode assets in the future) and a dart:wasm_assets (for linking WasmCode assets)? ¹

At least initially, could we perhaps have a private dart:_assets and then only allow package:web_assets/runtime.dart to import that?

Rather, the Dart SDK can provide a bundle that contains (1) the compiled Dart code, (2) all the web assets, and (3) a JSON mapping asset-ids to web assets file paths.

I like that idea, a dart build web-bundle could also take care of some build_web_compilers features like compiling with both dart2js and dart2wasm and then running a runtime feature detection check to determine which output to use. I also like the idea for a --dart-target=none integration to webdev, I suppose we could run that once initially and then look into hot reloads later.

Will the package assets keep their names and relative layout OR is the expectation that hooks pre-bundle multi file output into a single file?

I think we should give embeders the freedom to modify file names. If the asset consists of multiple files that need to be able to resolve each other, I think a hook should represent that as multiple added assets (one per file).

I see how that can be problematic for Emscripten, but I suppose a hook would then need to post-process outputs to avoid the expectation that relative ESM imports work. E.g. instead of doing import './second.js', you'd have to webUriAssetUri('package:my_package/second.js') in Dart, turn that into an absolute URL and pass that as a parameter to the first script which would then do an await import(parameterResolvedFromDart) instead.

(maybe we could have a new type of asset that is essentially a bundle of files with a guarantee that relative imports between files of the bundle continue to work, but that also sounds like something worth only adding later)

Unrelated to the issue at hand: I'm already using hooks for a WASI target, they work really well for that! ↩

I assume we'd need more than one library for this though, right? webUriAssetUri wouldn't work on native, so I assume there would eventually be a dart:assets for data assets (cross-platform, loading them as bytes), a dart:web_assets (acquiring a URL for them, and linking JsCode assets in the future) and a dart:wasm_assets (for linking WasmCode assets)? 1

Yeah, we'd need multiple APIs. One for each asset type.

For actually linking wasm code assets, I'd expect some specific dart2wasm compiler feature that can deal with @Native() external functions or something similar. Not a lookup function. You'd want to write your dart code "statically linked" with the WASM assets.

I like that idea, a dart build web-bundle could also take care of some build_web_compilers features like compiling with both dart2js and dart2wasm and then running a runtime feature detection check to determine which output to use.

Maybe we should start with this. Just to have both js and wasm output, and design it in such a way that we would have space for JsAssets WasmAssets and DataAssets later.

I think we should give embedders the freedom to modify file names.

Possibly. This is exactly what we do with dylibs in the Flutter ios and macos embedders with the install_name_tool. And that's why it's important users interact with assets via an asset id!

However, we don't have an equivalent tool that rewrites install names and dynamic-linker dlopen calls for Linux, Android and Windows. So we cannot rewrite file names for code assets on those OSes. Hence, we've decided that conflicting dylib file-names is a hard error during Dart/Flutter compilation, and Dart packages much collaborate to avoid file name collisions.

I see how that can be problematic for Emscripten

It sounds like the format that Emscripten might output, might have the same issue that we can't rename files arbitrarily because the JS/WASM files might contain imports to other JS/WASM files that we can't automatically rewrite.

The locateFile Hook: Compilers like Emscripten provide runtime configuration hooks. When you initialize an Emscripten module, you can provide a locateFile function. Bundlers use this function to dynamically remap the requested .wasm filename to the final hashed URL at runtime.

However, if we see a WasmAsset, we don't know that it was an emscripten module. So I'm not entirely sure who would wire this logic up.

maybe we could have a new type of asset that is essentially a bundle of files with a guarantee that relative imports between files of the bundle continue to work, but that also sounds like something worth only adding later

I was googling a bit, and another strategy that I found is to use a sub-dir for each package (and no file renaming). That way relative imports work for wasm/js assets within a Dart package, but you can't import wasm/js assets that are bundled via another Dart package.

If we go that route, then that's the contract we specify what embedders must satisfy. (And then we only error on duplicate file names within a single package.)

Unrelated to the issue at hand: I'm already using hooks for a WASI target, they work really well for that! ↩

🤯

Bringing https://dart-review.googlesource.com/c/sdk/+/542800/comments/e5f4a4f6_01b2be0f?tab=comments and dart-lang/build#5141 back here.

Should "packages" wrapping the SDK low level commands run hooks?

dart run, dart test, dart build cli, flutter build, flutter run are all high-level commands that (1) define the final app bundle format, (2) run the build hooks, and (3) wire up the APIs to use assets (@Native() external functions, and Flutters' asset bundle).
dart compile (js, wasm, kernel shapshot, aot snapshot) are low level commands that do not define a bundle format for the final application bundle.
package:jaspr, package:webdev, package:build_web_compilers are "SDK-wrapping packages" that are high level: They define the final app bundle output.
- I think the one defining the bundle format should be the one running the hooks. So, yes then Jaspr/webdev/build_web_compilers should be running hooks.

Now, that leads to an interesting question. Assets are set up to be SDK (or SDK-wrapping-packages) agnostic. Assets are set up to be bundle-format agnostic. Where would an SDK-agnostic API live?

If it would live in the package defining it. Let's say we add an API for package:web_assets's UrlAsset asset in that package. Then that API needs to be somehow wired up.
What I could conceive is: The API is implemented by calling a registerImplementation top level method, and the SDK-wrapping-package generates code around the users' main function that register the implementation.
- But, generating code around main probably doesn't work so well with running the frontend server for incremental builds. (The user would have to explicitly import some kind of generated file.)

(This is ignoring the dependency tracking and incremental updates questions that arise from if the SDK-wrapping-package is using build_runner, and build_runner mandates knowing all precise dependencies up front and mandating not downloading from the internet. I'm ignoring that for simplicities' sake here. CodeAssets are not supported in Flutter hot restart, but DataAssets are. This works in Flutter because Flutters' build system also has the abstraction that you know your dependencies afterwards instead of upfront.)

Also, SDK-wrapping-packages should most likely run with record-use enabled and then feed in the recorded use file to the hooks_runner to enable tree-shaking.

I think the one defining the bundle format should be the one running the hooks. So, yes then Jaspr/webdev/build_web_compilers should be running hooks.

👍

What I could conceive is: The API is implemented by calling a registerImplementation top level method, and the SDK-wrapping-package generates code around the users' main function that register the implementation.

At least for the web, that package could also probe for say a $dartAssets symbol in the global JS object at runtime, which these tools (build web compilers and those wrapping them) then set before invoking the compiled JS app. They inject additional JS code to switch between dart2js/dart2wasm output already (and I believe also for reloads/restarts). That way they wouldn't have to mess with Dart sources.

My two cents: Maybe there is an opportunity here to finally ditch build_web_compilers altogether and implement a clean web compiler and bundler system, rather than trying to to shoehorn yet another functionality into build_web_compilers that has to work its way around the build system.

For Jaspr I much rather invoke all the pieces myself (compilers, asset bundling, hooks runner, whatever else) than having to work around even more build_web_compiler quirks.

Alright, I think I should finish setting up the package first, starting with something that definitely doesn't require Dart SDK changes. We can then see how annoying it is to integrate that into jaspr (or into a generic "you're using build_web_compilers and want build hooks? use this package" framework) to decide whether Dart SDK/ build_runner changes make sense or not.

Navigation Menu

Repository navigation

[hooks] `build.dart` / `link.dart` web backends and `WasmCode` and `JsCode` assets #988

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[hooks] build.dart / link.dart web backends and WasmCode and JsCode assets #988

Description

Activity

sigmundch commented on Mar 11, 2024

mkustermann commented on Mar 11, 2024

dcharkes commented on Mar 19, 2025

MichealReed commented on Mar 20, 2025

nikeokoronkwo commented on Nov 14, 2025

dcharkes commented on Nov 14, 2025

srujzs commented on Nov 18, 2025

simolus3 commented on Dec 6, 2025

nikeokoronkwo commented on Dec 6, 2025

simolus3 commented on Dec 6, 2025

nikeokoronkwo commented on Dec 12, 2025

dcharkes commented on Dec 15, 2025

simolus3 commented on Dec 15, 2025

dcharkes commented on Dec 15, 2025

nikeokoronkwo commented on Dec 15, 2025

dcharkes commented on Dec 16, 2025

simolus3 commented on Jul 12, 2026

dcharkes commented on Jul 13, 2026

chaudharydeepanshu commented on Jul 13, 2026

simolus3 commented on Jul 13, 2026

Footnotes

dcharkes commented on Jul 14, 2026

dcharkes commented on Sep 10, 2026

simolus3 commented on Sep 10, 2026

schultek commented on Sep 10, 2026

simolus3 commented on Sep 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[hooks] `build.dart` / `link.dart` web backends and `WasmCode` and `JsCode` assets #988