You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 616fa81
Browse filesBrowse the repository at this point in the historyBrowse files
feat(delta): native Delta write via delta-kernel-rs
Adds an experimental native Delta write path, gated by
`spark.comet.write.deltaNative.enabled` (default off) and the `delta` cargo feature.
Native engine:
- delta::write_batches / write_arrow_ipc: create the table from the Arrow IPC schema if it
does not exist, then write all batches in one transactional commit (commit/retry logic
factored into a shared helper). New JNI entry Native.writeDeltaTable(tableUri, ipcBytes).
The arrow `ipc` feature is enabled only with the `delta` feature.
Spark side:
- CometDeltaWriteRule (post-hoc resolution) rewrites an eligible Delta SaveIntoDataSourceCommand
into CometDeltaWriteCommand before Spark eagerly executes it. No-op unless the config is on.
- CometDeltaWriteCommand writes natively only for a brand-new, non-partitioned table with
kernel-supported primitive column types; any other case, or any native error, transparently
falls back to Delta's own write, so a correct table is always produced. The query result is
collected to the driver and serialized to a single Arrow IPC stream (CometDeltaWrite), so this
is for small writes.
Tests: native round-trip unit test (create + append + read back); CometDeltaNativeWriteSuite
e2e (kernel-written table read back by Spark with matching data + commit provenance; fallback
for partitioned writes and when disabled), registered in CI. Validated on Spark 4.1 + Delta
4.1.0: write 3/3, read 7/7, native delta 14/14; clippy clean (default + delta), check-suites=0.
Part of apache#174.
Co-authored-by: Copilot <[email protected]>
0 commit comments