Skip to content

feat!: expose native lance scan descriptor for comet - #624

Draft
wirybeaver wants to merge 3 commits into
lance-format:mainfrom
wirybeaver:xuanyili/native-lance-read-descriptor
Draft

wirybeaver wants to merge 3 commits into
lance-format:mainfrom
wirybeaver:xuanyili/native-lance-read-descriptor

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Jun 12, 2026 •

Copy link
Copy Markdown

Closes #623.

Summary

Adds LanceScan.nativeScanPlan() so native engines can consume the scan already planned by Lance Spark: dataset URI and resolved version, full/projected schemas, filter SQL, limit/offset, batch size, storage options, and per-partition fragment IDs. Unsupported scan shapes expose fallback reasons.

See the ASCII architecture and full roadmap for the Comet integration. This PR provides the descriptor, not Comet's Rust reader.

Construction and Compatibility

Spark retains the final LanceScan in BatchScanExec, so the descriptor must capture its resolved planning state rather than ask a consumer to re-plan.

Use the existing public LanceScanBuilder for production scan creation. LanceScan now has one package-private constructor; the delegating overload is removed. The builder passes both the full and projected schemas explicitly. No second builder or constructor compatibility overload is introduced.

Scope

V1 targets ordinary reads. Search, TopN, pushed aggregation, and other unsupported scan semantics retain fallback behavior. Index/search extensions and namespace credential refresh are later roadmap phases.

Testing

./mvnw test -pl lance-spark-4.1_2.13 -am \
  -Dtest=LanceScanTest,LanceScanBuilderTest \
  -Dsurefire.failIfNoSpecifiedTests=false
git diff --check

All 49 tests pass (22 scan tests and 27 builder tests), with Maven Checkstyle/Spotless steps passing. This validates the Spark 4.1 profile, not the full version matrix or Comet native execution.

BREAKING CHANGE: the public LanceScan constructor is removed. External callers must migrate to LanceScanBuilder; existing binaries calling the removed signature must be rebuilt after migration. Maintainer agreement on this API change is requested.

@github-actions

Copy link
Copy Markdown
Contributor

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@wirybeaver wirybeaver changed the title Expose native Lance scan descriptor Expose native Lance scan descriptor for datafusion-comet integration Jun 12, 2026
@wirybeaver
wirybeaver marked this pull request as draft June 12, 2026 15:50
yanghua pushed a commit that referenced this pull request Sep 2, 2026
## What

Adds `LanceArrowStreamScanner`, which plans a Lance fragment scan and
exports it as an Arrow C Data Interface stream (`ArrowArrayStream`) for
native consumers such as Apache Gluten / Velox.

## Why

This is the read-side building block for offloading Lance scans to a
native engine. Only the `ArrowArrayStream` C-struct address crosses the
JVM/native boundary, so the consumer's Arrow build and classloader do
not need to match lance-spark's — Gluten builds Arrow 15 to match Spark
3.5 / Velox, while the Lance Java SDK is on Arrow 18. Passing the raw
struct address sidesteps that mismatch entirely.

## How

- `LanceArrowStreamScanner.export(fragmentId, inputPartition)` returns a
`LanceArrowStream` handle exposing `stream()` / `streamAddress()`.
- All scan planning (column projection, filter pushdown, limit/offset,
row-id / row-address, batch size) is delegated to the existing
`LanceFragmentScanner`, so the exported stream yields exactly the same
rows in the same order as the Spark columnar reader.
- The Lance native core populates the caller-owned stream directly via
`LanceScanner#exportArrowStream(long)` (lance-format/lance#7259), so no
Arrow data is materialized on the JVM heap on this path.
- `LanceArrowStream` owns the exported stream and the scan behind it;
closing it releases the native scan (through the stream's release
callback) and then the scanner and dataset handles.

## Dependency

`exportArrowStream(long)` first ships in lance-core `11.0.0-beta.21`, so
this bumps `lance.version` from `11.0.0-beta.10` (isolated in its own
commit).

## Testing

- New `LanceArrowStreamScannerTest`: exports each fragment of the
bundled test table, re-imports it on the JVM (standing in for a native
consumer), and asserts the rows match the columnar reader — run under
the leak-checking allocator to verify the export/scanner lifecycle
releases cleanly.
- Existing `LanceFragmentColumnarBatchScannerTest` still passes
(regression on the touched `LanceFragmentScanner`).

## Related

- Complements #715 (Gluten write path).
- Distinct mechanism from #624 / #623 (serializable native scan
descriptor for datafusion-comet): this exposes a live Arrow C stream
handle rather than a re-plannable descriptor.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <[email protected]>
@wirybeaver
wirybeaver force-pushed the xuanyili/native-lance-read-descriptor branch from 0888e58 to faca105 Compare September 15, 2026 14:10
@wirybeaver wirybeaver changed the title Expose native Lance scan descriptor for datafusion-comet integration feat: Expose native Lance scan descriptor for datafusion-comet integration Sep 15, 2026
@github-actions github-actions Bot added the enhancement New feature or request label Sep 15, 2026
@wirybeaver
wirybeaver force-pushed the xuanyili/native-lance-read-descriptor branch from faca105 to 620523e Compare October 4, 2026 22:50
Keep a single package-private LanceScan constructor and use the existing public LanceScanBuilder for production scan creation. Update direct test callers to supply both schemas explicitly.

BREAKING CHANGE: direct callers of the public LanceScan constructor must use LanceScanBuilder instead.
@wirybeaver wirybeaver changed the title feat: Expose native Lance scan descriptor for datafusion-comet integration feat!: expose native lance scan descriptor for comet Oct 26, 2026
@github-actions github-actions Bot added the breaking-change Features that introduces a breaking API change label Oct 26, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change Features that introduces a breaking API change enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose a stable native scan descriptor for Lance Spark reads and datafusion-comet integration

1 participant