You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dc60b71
Browse filesBrowse the repository at this point in the historyBrowse files
feat(delta): kernel-native CDF read mechanism via TableChanges (apache#76/apache#84, Rust foundation)
Add the executor-side Change Data Feed read using delta-kernel-rs's TableChanges API, the
kernel-native way to read change data (corrects the earlier mistaken "kernel can't do CDC").
- read_cdf_via_kernel: TableChanges::try_new(url, start, Option<end>) -> into_scan_builder().
build().execute() -> Arrow batches with data + _change_type / _commit_version /
_commit_timestamp (column mapping / partitions / DVs resolved by kernel). Single-task: kernel's
CDF per-file API (scan_metadata / CdfScanFile / get_cdf_transform_expr) is pub(crate), so only
the all-in-one execute() is usable -- one partition reads the whole bounded version range.
- DeltaKernelScanExec gains `cdf: Option<(start, end)>`; read_all routes to read_all_cdf, which
reuses the by-name assembler to place the CDF columns in scan.output order.
- proto: DeltaScanCommon.cdf_read / cdf_start_version / cdf_end_version. core maps them and
bypasses the kernel data-column schema requirement (CDF ships no per-file tasks/schemas).
Additive + dormant: the Scala side doesn't set cdf_read yet, so existing reads are unchanged
(contrib Rust tests 109/0). Remaining (apache#84): Scala CDF detection/routing -- intercept
DeltaCDFRelation (path readChangeFeed) + CdcAddFileIndex (streaming), extract start/end versions
+ the CDF output schema -- then CometDeltaCdcSuite, then retire DeltaSyntheticColumnsExec.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
0 commit comments