[Feature] Align table-level system-table support with Java and expose native C APIs #1383

Search before asking

• I searched in the issues and found nothing similar.

Related issue #1382 already tracks audit-log and incremental reads through C FFI. This issue covers the remaining system-table coverage and shared native API contracts; it does not request a second implementation of #1382.

Motivation

Source comparison: paimon-rust main f91fdb8ab034887547a5ee2f26c1979e30250558 against Java Paimon 2.1.0 3a5ffe7870697e2c2605e63413fb5ac5800aae7e, checked on November 12, 2026. No directly matching issue or C API PR was found in the pre-submission search. Core SystemRead already exists through #1137/#1138; those implementations are reused here. Related #679 concerns referenced_files_size, which is outside this Java table-level scope.

Apache Doris exposes Java Paimon's table-level system tables and would like to execute their reads through paimon-rust. Its Java frontend can continue to bind relations and plan work, while native backend workers read their assigned inputs and return Arrow batches. This requires public APIs usable independently of DataFusion SQL.

Rust already implements many metadata tables through SystemRead, and Python exposes a thin binding. However, the C binding does not expose SystemRead. Some Java system-table relations are also missing from the public core API, and an existing reader does not always expose the complete Java output schema. As a result, native consumers cannot currently cover the same table-level system-table surface.

The compatibility baseline here is the 19 table-level tables registered by Java Paimon 2.1.0. It excludes catalog-wide global tables. The source comparison found the following gaps:

audit_log is covered by #1382. The two remaining table names, ro and row_tracking, should not be treated as entirely missing reader APIs: externally planned read-optimized DataSplits can reuse ordinary reading, and C projection can already select _ROW_ID and _SEQUENCE_NUMBER on appropriate tables. Their existing capabilities and limitations should be documented alongside the support matrix.

Solution

The missing public contracts are:

1. Metadata system-table reads through C. Expose the existing core readers, their logical output schema, projection, filtering, limits, Arrow results, and ownership/release rules. Predicate construction must use the target relation's complete schema, including filter-only columns, rather than the base table's field positions. Basic queries may leave unsupported filters in the consuming engine.
2. Shared readers for uncovered relations. Make aggregation_fields available outside DataFusion, and provide the Java-compatible file_indexes and statistics relations. File indexes must distinguish embedded and external index metadata; statistics must expose persisted table statistics rather than substitute scan counters.
3. Binlog output semantics. Preserve Java-compatible scalar system fields and array-valued user fields. Java 2.1.0 wraps non-streaming DataSplit values in one-element arrays; streaming updates are packed into before/after arrays. The paired output kind follows the second record, while an enabled sequence number comes from the first. Pair within the producer's logical stream, preserving repeated updates rather than grouping an entire interval by primary key. Pairing must work across Arrow batch boundaries with valid buffer ownership; incomplete-event behavior and supported split/mode combinations need a Java-compatible contract. This is distinct from returning audit rows with scalar user fields.
4. Output schema compatibility. Preserve field identities, names, order, types, and nullability, including nested index metadata. In particular, expose snapshots.operation from the corresponding snapshot metadata; commit_kind or a constant NULL is not an equivalent substitute. Compatibility should be stated against a specific Java baseline rather than assumed from matching table names.
5. Read-state compatibility. Allow native system reads to consume a resolved table environment with a selected branch and snapshot, including a captured empty view, while preserving the schema bound by the consuming engine. Document supported point-in-time and interval semantics per table. Ordinary time-travel options or timestamp-window support do not establish equivalence for every incremental system-table query, particularly partitions.
6. Incremental production of large metadata results. Provide a contract for producing batches without first collecting the complete output, with resource accounting and cancellation. Distributed execution needs non-overlapping assigned work units or an equivalent partitioning contract, so workers do not each enumerate the full table. Bucket/partition aggregates must remain correct when inputs are divided. Small metadata relations may continue to use a single execution unit.

These requirements reuse existing core capabilities where present. They do not require every system table to share the same physical reader or to become a DataSplit, nor do they require a Rust implementation of engine scheduling.

Anything else?

Useful cross-language coverage includes full and reordered projections, filter-only columns, empty and zero-column outputs, nullable and nested values, snapshot/branch selection, and supported interval modes. Additional cases include non-null snapshots.operation, file-index storage variants, absent persisted statistics, and binlog update pairs crossing batch boundaries. For large metadata reads, verify that batches are produced before full result materialization and that cancellation releases retained resources. These are validation requirements; this issue does not claim runtime reproductions or measured memory failures.

Related boundaries:

• Audit-log and incremental C reader work remains in [Feature] Expose audit-log and incremental reads over externally planned splits through C FFI #1382; this issue can reference that coverage.
• General memory integration is already discussed in #911; the gap here is applying an appropriate contract to metadata reads.
• Java-planned ro reads and basic row-tracking projections can reuse existing C APIs. A standalone Rust read-optimized planner is a separate requirement if consumers also migrate planning.
• Catalog-wide all_tables, all_partitions, all_table_options, and catalog_options are outside this table-level scope, as are changelog production, MV scheduling, and write operations.

Source references:

• Java table-level registry, Rust SystemRead, and C APIs.
• DataFusion aggregation_fields, Java FileIndexesTable, and Java StatisticTable.
• Java BinlogTable, Java snapshots schema, and Rust snapshots schema.

Willingness to contribute

• I'm willing to submit a PR!