Skip to content

Support wasm32-unknown-unknown target#1643

Closed
Spxg wants to merge 29 commits into
rusqlite:masterfrom
Spxg:wasm-demo
Closed

Support wasm32-unknown-unknown target#1643
Spxg wants to merge 29 commits into
rusqlite:masterfrom
Spxg:wasm-demo

Conversation

@Spxg

@Spxg Spxg commented Feb 10, 2025

Copy link
Copy Markdown
Contributor

I don't know if it can be implemented this way.

@Spxg Spxg changed the title Support wasm32-unknown-unknown target Support wasm32-unknown-unknown target Feb 10, 2025
@Spxg Spxg marked this pull request as ready for review February 10, 2025 14:22
@gwenn

gwenn commented Feb 10, 2025

Copy link
Copy Markdown
Collaborator

I know nothing about wasm so we need to find someone kind enough to validate this PR.
And would you mind adding a dedicated job, something like https://github.com/rusqlite/rusqlite/pull/1010/files#diff-7829468e86c1cc5d5133195b5cb48e1ff6c75e3e9203777f6b2e379d9e4882b3R86
in your PR
Thanks.

@Spxg

Spxg commented Feb 11, 2025

Copy link
Copy Markdown
Contributor Author

I know nothing about wasm so we need to find someone kind enough to validate this PR. And would you mind adding a dedicated job, something like https://github.com/rusqlite/rusqlite/pull/1010/files#diff-7829468e86c1cc5d5133195b5cb48e1ff6c75e3e9203777f6b2e379d9e4882b3R86 in your PR Thanks.

Of course, I will add the tests runner to CI/CD later today, currently all the tests pass (except a couple that require std::fs)

@Spxg

Spxg commented Feb 11, 2025

Copy link
Copy Markdown
Contributor Author

(a) https://github.com/Spxg/sqlite-wasm-rs/blob/master/sqlite-wasm-rs/sqlite-wasm/sqlite3.wasm comes from https://sqlite.org/2025/sqlite-wasm-3490000.zip ? (b) You don't use https://github.com/sqlite/sqlite/blob/master/ext/wasm/api/sqlite3-wasm.c ?

(a) sqlite3.wasm is only used in wrapper feature, and the interface is exported through wasm-bindgen. The sqlite3.wasm in the project is not from the official product, see https://github.com/Spxg/sqlite-wasm-rs?tab=readme-ov-file#why-vendor-sqlite-wasm

(b) This file is used to compile sqlite3 into emscripten wasm, mainly for generating bindings in emscripten js. We don't need it, we have enabled -DSQLITE_OS_OTHER, and do not compile sqlite directly into .wasm, and we don't need emscripten js binding.

@Spxg

Spxg commented Feb 11, 2025

Copy link
Copy Markdown
Contributor Author

The tests have been added to the CI/CD (approval is required to verify whether they work). All tests have passed in local testing.
The rusqlite_test_helper library has been added to handle the differences between the wasm platform and the native platform.

@gwenn

gwenn commented Feb 11, 2025

Copy link
Copy Markdown
Collaborator

Thanks for your time and explanations.

@codecov

codecov Bot commented Feb 11, 2025

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.08%. Comparing base (44e0ef9) to head (8e53118).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1643      +/-   ##
==========================================
- Coverage   86.90%   82.08%   -4.82%     
==========================================
  Files          58       58              
  Lines       10638     7398    -3240     
==========================================
- Hits         9245     6073    -3172     
+ Misses       1393     1325      -68     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Spxg

Spxg commented Feb 11, 2025

Copy link
Copy Markdown
Contributor Author

Fixed CI for clippy and minimal versions. The reduced coverage is probably because I added two wasm-only tests and some comments.

By the way, why does the CI process require approval? It seems that it is not convenient to find problems and fix CI.

@gwenn

gwenn commented Feb 12, 2025

Copy link
Copy Markdown
Collaborator

Fixed CI for clippy and minimal versions. The reduced coverage is probably because I added two wasm-only tests and some comments.

By the way, why does the CI process require approval? It seems that it is not convenient to find problems and fix CI.

Maybe because a patched / malicious build.rs, you can do anything !

@Spxg

Spxg commented Feb 12, 2025

Copy link
Copy Markdown
Contributor Author

Maybe because a patched / malicious build.rs, you can do anything !

Reasonable. I need to rerun CI now to verify whether everything passes. Can you approve it?

@trevyn

trevyn commented Feb 12, 2025

Copy link
Copy Markdown
Contributor

By the way, why does the CI process require approval? It seems that it is not convenient to find problems and fix CI.

You can enable Actions on your fork and get CI to run inside your account instead of from the PR.

@Spxg

Spxg commented Feb 12, 2025

Copy link
Copy Markdown
Contributor Author

By the way, why does the CI process require approval? It seems that it is not convenient to find problems and fix CI.

You can enable Actions on your fork and get CI to run inside your account instead of from the PR.

Thanks!

@Spxg

Spxg commented Feb 12, 2025

Copy link
Copy Markdown
Contributor Author

CI have passed: https://github.com/Spxg/rusqlite/actions/runs/13284078817/job/37088729090, cov failed because I didn’t have a token to upload. (thanks to @trevyn )

I'm ready for the review.

@Spxg Spxg marked this pull request as draft February 13, 2025 14:26
@Spxg Spxg marked this pull request as ready for review February 14, 2025 15:32
@Spxg

Spxg commented Feb 14, 2025

Copy link
Copy Markdown
Contributor Author

Because of breaking changes, sqlite-wasm-rs is updated to 0.3

  1. Deleted wrapper feature
  2. Use bundled feature to compile sqlite by default
  3. Provided precompiled feature, see https://github.com/Spxg/sqlite-wasm-rs#why-provide-precompiled-library for details

@Spxg

Spxg commented Feb 21, 2025

Copy link
Copy Markdown
Contributor Author

we need to find someone kind enough to validate this PR.

Any suggestions?

@gwenn

gwenn commented Feb 21, 2025

Copy link
Copy Markdown
Collaborator

we need to find someone kind enough to validate this PR.

Any suggestions?

Maybe @thomcc or @trevyn ?

@lucyawrey

Copy link
Copy Markdown

This PR is exactly what I need for one of my projects, so I hope someone can validate it soon.

I can at least confirm @Spxg 's fork works without issue when used as a dependency in a project targeting wasm32-unknown-unknown. I only seem to be able to get it to work in-memory for wasm builds though, and am not sure how to get it to work with a persistent VFS.

@Spxg

Spxg commented Feb 22, 2025

Copy link
Copy Markdown
Contributor Author

and am not sure how to get it to work with a persistent VFS.

Currently, there is only one option for persistence: opfs-sahpool, which is ported from sqlite-wasm (https://sqlite.org/wasm/doc/trunk/persistence.md#vfs-opfs-sahpool) and needs to run on a dedicated worker. Other VFS are more difficult to implement, such as VFS based on IndexedDB, because they are asynchronous and require a lot of extra work: rhashimoto/wa-sqlite#154 (comment)

For more usages, see https://github.com/Spxg/sqlite-wasm-rs/blob/master/VFS.md#opfs-sahpool. This interface is now re-exported by rusqlite, and can be installed using rusqlite::ffi::install_opfs_sahpool.

@Spxg

Spxg commented Feb 22, 2025

Copy link
Copy Markdown
Contributor Author

am not sure how to get it to work with a persistent VFS.

If you are interested, here is an example I wrote for diesel, which can easily experience the difference between memory vfs and persistent vfs: https://github.com/diesel-rs/diesel/tree/master/examples/sqlite/wasm. (bundled feature is used by default, which requires the emscripten toolchain. If you don't have the emscripten toolchain, change feature to precompiled)

@Spxg

Spxg commented Feb 24, 2025

Copy link
Copy Markdown
Contributor Author

Precompiled products are not provided by default, you can specify the precompiled-wasm feature.
Using the bundled feature requires the emscripten toolchain.

See https://github.com/Spxg/sqlite-wasm-rs/?tab=readme-ov-file#why-provide-precompiled-library for details.

@lucyawrey

Copy link
Copy Markdown

@Spxg Thanks for the information!

@Spxg

Spxg commented Feb 26, 2025

Copy link
Copy Markdown
Contributor Author

Maybe @thomcc or @trevyn ?

It seems like they all don't have time to review it, so I thought of some questions you might be concerned about, I hope this can answer your questions.

@Spxg

Spxg commented Mar 5, 2025

Copy link
Copy Markdown
Contributor Author

we need to find someone kind enough to validate this PR.

Any suggestions?

Maybe @thomcc or @trevyn ?

It seems like they all don't have time to review it, so I thought of some questions you might be concerned about, I hope this can answer your questions.

@thomcc @gwenn @trevyn any other questions or suggestions?

@Spxg

Spxg commented Aug 1, 2025

Copy link
Copy Markdown
Contributor Author

@gwenn any suggestions? wasm-bindgen was unmaintained for a while, which might have been a concern for you, but the organization has changed and things are moving in a positive direction. wasm-bindgen/wasm-bindgen#4533

@Spxg Spxg closed this Aug 2, 2025
@gwenn

gwenn commented Aug 7, 2025

Copy link
Copy Markdown
Collaborator

@Spxg

Spxg commented Aug 7, 2025

Copy link
Copy Markdown
Contributor Author

Just for reference: https://blog.rust-lang.org/2025/04/04/c-abi-changes-for-wasm32-unknown-unknown/

Yes, wasm32-unknown-unknown is now C-ABI aligned, which means linking to emscripten or wasi will not be a problem.

@gwenn

gwenn commented Aug 7, 2025

Copy link
Copy Markdown
Collaborator

Just for reference: https://blog.rust-lang.org/2025/04/04/c-abi-changes-for-wasm32-unknown-unknown/

Yes, wasm32-unknown-unknown is now C-ABI aligned, which means linking to emscripten or wasi will not be a problem.

Which means the official sqlite3.wasm ?

@Spxg

Spxg commented Aug 7, 2025

Copy link
Copy Markdown
Contributor Author

Which means the official sqlite3.wasm ?

This means we can compile sqlite3 with emcc and link it into our rust wasm32-unknown-unknown program, just like sqlite-wasm-rs does.

However, this is only one step, more work is actually to implement a persistent VFS under wasm32-unknown-unknown, like opfs: https://github.com/Spxg/sqlite-wasm-rs/blob/master/sqlite-wasm-rs/src/vfs/sahpool.rs

If you are interested, you can try it out at https://sqlight.dev

@Spxg

Spxg commented Aug 7, 2025

Copy link
Copy Markdown
Contributor Author

@gwenn

gwenn commented Aug 7, 2025

Copy link
Copy Markdown
Collaborator

Ok, thanks for the details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants