Repository navigation
feat!: expose native lance scan descriptor for comet - #624
Draft
wirybeaver wants to merge 3 commits into
Draft
wirybeaver wants to merge 3 commits into
wirybeaver wants to merge 3 commits into
Conversation
Contributor
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
wirybeaver
marked this pull request as draft
June 12, 2026 15:50
yanghua
pushed a commit
that referenced
this pull request
Sep 2, 2026
## What Adds `LanceArrowStreamScanner`, which plans a Lance fragment scan and exports it as an Arrow C Data Interface stream (`ArrowArrayStream`) for native consumers such as Apache Gluten / Velox. ## Why This is the read-side building block for offloading Lance scans to a native engine. Only the `ArrowArrayStream` C-struct address crosses the JVM/native boundary, so the consumer's Arrow build and classloader do not need to match lance-spark's — Gluten builds Arrow 15 to match Spark 3.5 / Velox, while the Lance Java SDK is on Arrow 18. Passing the raw struct address sidesteps that mismatch entirely. ## How - `LanceArrowStreamScanner.export(fragmentId, inputPartition)` returns a `LanceArrowStream` handle exposing `stream()` / `streamAddress()`. - All scan planning (column projection, filter pushdown, limit/offset, row-id / row-address, batch size) is delegated to the existing `LanceFragmentScanner`, so the exported stream yields exactly the same rows in the same order as the Spark columnar reader. - The Lance native core populates the caller-owned stream directly via `LanceScanner#exportArrowStream(long)` (lance-format/lance#7259), so no Arrow data is materialized on the JVM heap on this path. - `LanceArrowStream` owns the exported stream and the scan behind it; closing it releases the native scan (through the stream's release callback) and then the scanner and dataset handles. ## Dependency `exportArrowStream(long)` first ships in lance-core `11.0.0-beta.21`, so this bumps `lance.version` from `11.0.0-beta.10` (isolated in its own commit). ## Testing - New `LanceArrowStreamScannerTest`: exports each fragment of the bundled test table, re-imports it on the JVM (standing in for a native consumer), and asserts the rows match the columnar reader — run under the leak-checking allocator to verify the export/scanner lifecycle releases cleanly. - Existing `LanceFragmentColumnarBatchScannerTest` still passes (regression on the touched `LanceFragmentScanner`). ## Related - Complements #715 (Gluten write path). - Distinct mechanism from #624 / #623 (serializable native scan descriptor for datafusion-comet): this exposes a live Arrow C stream handle rather than a re-plannable descriptor. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
wirybeaver
force-pushed
the
xuanyili/native-lance-read-descriptor
branch
from
September 15, 2026 14:10
0888e58 to
faca105
Compare
wirybeaver
force-pushed
the
xuanyili/native-lance-read-descriptor
branch
from
October 4, 2026 22:50
faca105 to
620523e
Compare
Keep a single package-private LanceScan constructor and use the existing public LanceScanBuilder for production scan creation. Update direct test callers to supply both schemas explicitly. BREAKING CHANGE: direct callers of the public LanceScan constructor must use LanceScanBuilder instead.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #623.
Summary
Adds
LanceScan.nativeScanPlan()so native engines can consume the scan already planned by Lance Spark: dataset URI and resolved version, full/projected schemas, filter SQL, limit/offset, batch size, storage options, and per-partition fragment IDs. Unsupported scan shapes expose fallback reasons.See the ASCII architecture and full roadmap for the Comet integration. This PR provides the descriptor, not Comet's Rust reader.
Construction and Compatibility
Spark retains the final
LanceScaninBatchScanExec, so the descriptor must capture its resolved planning state rather than ask a consumer to re-plan.Use the existing public
LanceScanBuilderfor production scan creation.LanceScannow has one package-private constructor; the delegating overload is removed. The builder passes both the full and projected schemas explicitly. No second builder or constructor compatibility overload is introduced.Scope
V1 targets ordinary reads. Search, TopN, pushed aggregation, and other unsupported scan semantics retain fallback behavior. Index/search extensions and namespace credential refresh are later roadmap phases.
Testing
./mvnw test -pl lance-spark-4.1_2.13 -am \ -Dtest=LanceScanTest,LanceScanBuilderTest \ -Dsurefire.failIfNoSpecifiedTests=false git diff --checkAll 49 tests pass (22 scan tests and 27 builder tests), with Maven Checkstyle/Spotless steps passing. This validates the Spark 4.1 profile, not the full version matrix or Comet native execution.
BREAKING CHANGE: the public
LanceScanconstructor is removed. External callers must migrate toLanceScanBuilder; existing binaries calling the removed signature must be rebuilt after migration. Maintainer agreement on this API change is requested.