Use `u64` range instead of `usize`, for better wasm32 support by XiangpengHao · Pull Request #6961 · apache/arrow-rs

XiangpengHao · 2025-01-10T15:55:26Z

Which issue does this PR close?

Closes Support 32-bit Architectures / Replace usize with u64 #5351

Rationale for this change

Ranges should be u64 so that 32 bit platforms can read files larger than 4GB. More details can be found in #5351

What changes are included in this PR?

This is a rather simple change. No functionality change for 64 bit platforms. But is a breaking change for trait implementors.
Given that we already break one in #6619, it's seems like a good timing to also include this change.

This PR added and removed a few casting. I have checked the casting is ok, but please help me check again.

My principle of using usize vs u64:

Anything already in memory should use usize, e.g., slicing a memory range
O.w. use u64

Are there any user-facing changes?

alamb · 2025-01-10T19:03:53Z

The CI failure https://github.com/apache/arrow-rs/actions/runs/12713696201/job/35442156669?pr=6961 should be fixed with this PR:

Fix WASM CI for Rust 1.84 release #6963

alamb

Thanks @XiangpengHao ❤️

Can you help me understand what currently doesn't work with 32 bit builds now?

We have an existing test for WASM32 that seems to work fine: https://github.com/apache/arrow-rs/actions/runs/12680096825/job/35341184364

I also locally tried checking with an i686 target and that worked fine too 🤔

rustup target add i686-unknown-linux-gnu
cd object_store
cargo check --target=i686-unknown-linux-gnu
...

🤔

alamb · 2025-01-10T19:22:08Z

🤔 maybe it is related to wanting to use the http features?

But it still seems to compile just fine for me 🤔

sudo apt-get install gcc-multilib
rustup target add i686-unknown-linux-gnu
cd object_store
cargo check --target=i686-unknown-linux-gnu --features=http
...

alamb · 2025-01-11T18:01:09Z

Can you help me understand what currently doesn't work with 32 bit builds now?

For anyone following along, the answer is here: #5351 (comment)

I still think we should try and add some sort of test that would fail on wasm32 before this change and not after the change. If we don't do that I feel like:

We are less sure we fixed all the places that require change
We may inadvertently introduce regressions but no test would fail

I'll see if I can find some time over the next day or two to help, if no on ebeats me to it

XiangpengHao · 2025-01-13T21:54:47Z

add some sort of test that would fail on wasm32

Thanks for the review @alamb , I agree with the rationale but unfortunately we don't yet have the infra to test wasm32 yet. We currently only build on wasm32, but there's no runner that can actually execute the webassembly

Xuanwo · 2025-01-14T15:00:31Z

Thanks for the review @alamb , I agree with the rationale but unfortunately we don't yet have the infra to test wasm32 yet. We currently only build on wasm32, but there's no runner that can actually execute the webassembly

OpenDAL has an edge test for this; it's worth taking a look.

https://github.com/apache/opendal/blob/781682cc05b31d1f12f91fb7568a868f038c4d3b/.github/workflows/test_edge.yml#L62-L107

The test code could be found here: https://github.com/apache/opendal/tree/main/core/edge/s3_read_on_wasm

alamb

Thank you @XiangpengHao -- I reviewed this PR carefully and I think the API changes are good but there are a few places where going from u64 to usize should be checked (as it could truncate on WASM).

I may be mistaken (my casting knowledge is mostly left over from C/C++ 😅 )

I also have a few other suggestions related to comments, but otherwise this is good to go from my perspectivel

Thank you again for helping push this along

object_store/src/azure/client.rs

object_store/src/chunked.rs

object_store/src/lib.rs

object_store/src/local.rs

object_store/src/memory.rs

object_store/src/util.rs

alamb · 2025-01-14T15:32:57Z

object_store/src/util.rs

                fetches.push(range.clone());
-                futures::future::ready(Ok(Bytes::from(src[range].to_vec())))
+                futures::future::ready(Ok(Bytes::from(
+                    src[range.start as usize..range.end as usize].to_vec(),


again, I think we need to check the conversion here rather than truncating

this function doesn't return error, I just unwrap here, as it's only a test

alamb · 2025-01-14T15:41:54Z

BTW I merged up from main. I also think the obejct_store emulator tests are unrelated to this PR. I have filed

Intermittent failure with object_store aws::tests::s3_test emulator arrow-rs-object-store#25

To track

alamb · 2025-01-14T16:57:17Z

Thanks for the review @alamb , I agree with the rationale but unfortunately we don't yet have the infra to test wasm32 yet. We currently only build on wasm32, but there's no runner that can actually execute the webassembly

OpenDAL has an edge test for this; it's worth taking a look.

https://github.com/apache/opendal/blob/781682cc05b31d1f12f91fb7568a868f038c4d3b/.github/workflows/test_edge.yml#L62-L107

The test code could be found here: https://github.com/apache/opendal/tree/main/core/edge/s3_read_on_wasm

Thank you @Xuanwo -- I didn't see this response before posting my update

alamb · 2025-01-14T17:16:18Z

I restarted the emulator test to see if it passes:
Old: https://github.com/apache/arrow-rs/actions/runs/12770295148/job/35594807418?pr=6961

alamb · 2025-01-14T19:27:55Z

Now the emulator test is passing after 2 retries: https://github.com/apache/arrow-rs/actions/runs/12770295148/job/35607985446?pr=6961

🤔

Co-authored-by: Andrew Lamb <[email protected]>

alamb

Thanks @XiangpengHao -- this looks great to me

alamb · 2025-01-17T21:52:06Z

Thanks again

…#6961) * u64 ranges * more u64 * make clippy happy * even more u64 * Update object_store/src/lib.rs Co-authored-by: Andrew Lamb <[email protected]> * Update object_store/src/lib.rs Co-authored-by: Andrew Lamb <[email protected]> * address comments --------- Co-authored-by: Andrew Lamb <[email protected]>

XiangpengHao added 3 commits January 9, 2025 16:46

u64 ranges

c1aab14

more u64

b114802

make clippy happy

450a268

github-actions bot added the object-store label Jan 10, 2025

even more u64

b6670cd

alamb added the api-change Changes to the arrow API label Jan 10, 2025

alamb reviewed Jan 10, 2025

View reviewed changes

alamb mentioned this pull request Jan 10, 2025

Support 32-bit Architectures / Replace usize with u64 #5351

Closed

XiangpengHao mentioned this pull request Jan 13, 2025

ObjectStore WASM32 Support #6818

Closed

3 tasks

Merge remote-tracking branch 'apache/main' into u64_range

ebcb38e

alamb mentioned this pull request Jan 14, 2025

Run object_store tests (on CI) with WASM build/runtime apache/arrow-rs-object-store#26

Open

alamb changed the title ~~Switch to use u64 range instead of usize, for better wasm32 support~~ Use u64 range instead of usize, for better wasm32 support Jan 14, 2025

alamb reviewed Jan 14, 2025

View reviewed changes

alamb mentioned this pull request Jan 14, 2025

Intermittent failure with object_store aws::tests::s3_test emulator apache/arrow-rs-object-store#25

Open

XiangpengHao and others added 3 commits January 15, 2025 16:49

Update object_store/src/lib.rs

af30a4c

Co-authored-by: Andrew Lamb <[email protected]>

Update object_store/src/lib.rs

5608f01

Co-authored-by: Andrew Lamb <[email protected]>

address comments

077e99b

alamb approved these changes Jan 16, 2025

View reviewed changes

alamb merged commit 9cd7018 into apache:main Jan 17, 2025
14 checks passed

waynr mentioned this pull request Feb 20, 2025

feat: implement GetOptions extensions #7160

Closed

tustvold mentioned this pull request Mar 5, 2025

Parquet Use U64 Instead of Usize (wasm support for files greater than 4GB) #7238

Closed

alamb mentioned this pull request Apr 7, 2025

Upgrade to arrow/parquet 55, and object_store to 0.12.0 and pyo3 to 0.24.0 apache/datafusion#15466

Merged

alamb mentioned this pull request Apr 30, 2025

feat: upgrade to DataFusion 47.0.0 delta-io/delta-rs#3378

Merged

Comments

Conversation

XiangpengHao commented Jan 10, 2025 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

alamb commented Jan 10, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 10, 2025

Uh oh!

alamb commented Jan 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XiangpengHao commented Jan 13, 2025

Uh oh!

Xuanwo commented Jan 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

XiangpengHao Jan 16, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 14, 2025

Uh oh!

alamb commented Jan 14, 2025

Uh oh!

alamb commented Jan 14, 2025

Uh oh!

alamb commented Jan 14, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb commented Jan 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

XiangpengHao commented Jan 10, 2025 •

edited by alamb

Loading

alamb commented Jan 11, 2025 •

edited

Loading

Xuanwo commented Jan 14, 2025 •

edited

Loading