chore!: update lance dependency to v9.0.0-rc.1#3673
Merged
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
github-actions
Bot
added
Rust
DataFusion 54 (pulled in by lance v9) replaced its hashing RandomState (ahash -> foldhash), which changes hash-based split output. Update the hardcoded expectations in test_hash_split, and hash a high-cardinality column in the Python test_split_hash_with_discard so its discard behavior is stable. The new lance file reader returns no batches for an empty file (the old read_record_batch returned a single empty batch), so concat_batches on batches[0] panicked when a shuffle output file was empty. Build the batch from the known schema instead, which handles the empty case. Bump the datafusion Python package to v54 to match the bundled DataFusion, avoiding an FFI ABI mismatch in register_table_provider. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
wjones127
force-pushed
the
codex/update-lance-9-0-0-rc-1
branch
from
July 16, 2026 15:46
830a67d to
6858a93
Compare
westonpace
approved these changes
Jul 16, 2026
| .collect::<Vec<_>>(); | ||
| let mut hashes = vec![0; batch.num_rows()]; | ||
| let random_state = ahash::RandomState::with_seeds(0, 0, 0, 0); | ||
| let random_state = RandomState::with_seed(0); |
Contributor
There was a problem hiding this comment.
I mentioned this elsewhere but it's not ideal we are beholden to DF being able to change the hash function. Stable is more important for data loading than performance or statistical properties.
Still, I think it is inevitable.
Can we document this as a breaking change?
…compat pylance>=7,<8 is built against datafusion-ffi 53 and segfaults when registering its FFILanceTableProvider with the datafusion 54 Python package now required by lance v9. Pin to the 9.0.0rc1 preview build from the lance-format fury.io index, which matches lance v9.0.0-rc.1 and declares datafusion>=54,<55 for its own tests.
wjones127
changed the title
github-actions
Bot
added
the
breaking-change
jackye1995
approved these changes
Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BREAKING CHANGE: splits generated by the permutation data loader will not be the same, due to a change in hash function.
Updates the Lance dependencies and Java lance-core to v9.0.0-rc.1. Includes the required DataFusion 54 and Lance file-reader compatibility updates.