[Feature] Expose audit-log and incremental reads over externally planned splits through C FFI #1382

Search before asking

• I searched in the issues and found nothing similar.

Searched issues and PRs for incremental C, incremental FFI, audit_log, and audit C. #506 and #1042 concern the DataFusion table function, not an ordinary C FFI entry point. No directly matching C API issue was found as of November 12, 2026.

Motivation

Apache Doris plans Paimon scans in its Java frontend and assigns logical splits to backend workers. The backend already uses the C binding for native Paimon reads. To support table$audit_log over this path, each worker needs to execute its assigned splits with audit semantics and stream Arrow batches, without replanning the whole table.

The Rust core supports current-state audit reads through AuditLogRead, and snapshot-range planning with Delta, Changelog, Auto, and Diff modes. It exposes to_incremental_arrow and to_audit_log_arrow. Current-state audit reads and incremental audit reads are distinct APIs; exposing only the latter would leave ordinary table$audit_log queries uncovered.

These outputs have different contracts. Ordinary Diff returns after images and omits deleted keys. Audit-log Diff can return +I, -U, +U, and -D through its rowkind column. Consumers maintaining downstream state need that distinction.

At main f91fdb8ab034887547a5ee2f26c1979e30250558, the C ReadBuilder does not expose equivalent incremental-plan and audit-log read APIs. Its tests explicitly reject incremental-between passed as a scan option. Allowing that string alone would not expose the missing plan and output contracts.

Solution

Provide additive C entry points for current-state audit reading and for planning a bounded snapshot interval and reading its ordinary or audit-log output, reusing the existing core implementation. Define ownership of read/plan handles and the produced Arrow streams.

Distributed consumers also need to execute only externally assigned work units, without each worker replanning the complete table or resolving latest independently. An audit reader over imported DataSplits would cover current-state execution, provided those splits satisfy the audit planner's grouping requirements. DataFusion already demonstrates this execution pattern with AuditLogRead::new(read)?.to_arrow(splits).

Incremental import additionally needs a resolved mode and validated work-unit representation. Java 2.1.0 IncrementalSplit contains before/after files and deletion files and is not a DataSplit; the existing DataSplit byte import cannot be assumed to accept it. Define a compatible importer or an equivalent public representation mapping to IncrementalPlan::try_new, preserving each DiffPair as a complete work unit. Current-state binding and incremental transport can be separate contributions under the same API discussion.

The public contract should describe (start_exclusive, end_inclusive], equal endpoints, required retained snapshots, mode-specific output, rowkind/schema export, and error behavior for unsupported tables or missing history. Preserve the existing core restrictions rather than silently broadening support.

Coverage should include inserts, updates, deletes, repeated keys, deletion vectors, first-row split grouping where supported, relevant schema evolution, malformed intervals, missing endpoints, empty results, system-field-only projection, independent execution of assigned splits, and resource release after errors or cancellation. Ordinary reads followed by a synthetic +I column are not a substitute for AuditLogRead's retained-retract semantics.

The Diff contract needs explicit cross-version documentation: Java 2.1.0 IncrementalDiffSplitRead emits changed after images and retained deletes, whereas the Rust audit Diff API emits before/after update pairs. This is an interoperability question to resolve with shared fixtures, not a claim that the Rust implementation is necessarily incorrect.

Anything else?

References: core incremental-reading documentation and C binding.

Further references: DataFusion audit execution, Java IncrementalSplit, and Java Diff reader.

This issue does not add binlog pairing/array output, MV scheduling, progress publication, or a new changelog producer. Those are separate reader, consumer, or writer capabilities.

Willingness to contribute

• I'm willing to submit a PR!