[Feature] Support Native Read for Paimon System Tables (binlog, audit_log, ro) #60505

Background

Currently, Doris supports querying Paimon system tables (e.g., table$snapshots, table$partitions) through the Table-Valued Function (TVF) path. The execution flow is:

SQL: SELECT * FROM catalog.db.table$snapshots
  → MetadataScanNode → TVF → JNI Scanner → Paimon SDK

This approach works well for metadata-oriented system tables (snapshots, manifests, partitions, etc.) that return small result sets from metadata files.

However, for data-oriented system tables like binlog, audit_log, and ro (read-optimized), the current JNI-based approach has significant performance limitations:

1. Data Source Difference: Unlike metadata tables, binlog/audit_log/ro read actual data files (ORC/Parquet), not metadata files
2. JNI Overhead: Large data volumes suffer from JNI serialization/deserialization overhead
3. Missing Native Optimizations: Cannot leverage Doris's native vectorized ORC/Parquet readers and predicate pushdown optimizations

Paimon System Table Classification

Paimon binlog/audit_log Implementation Analysis

In Paimon's codebase, BinlogTable and AuditLogTable are special:

• They implement DataTable interface (not just ReadonlyTable)
• They wrap the underlying FileStoreTable and read actual data files
• They reuse DataSplit from the source table
• The underlying storage format is ORC/Parquet

Key implementation files in Paimon:

• paimon-core/.../table/system/BinlogTable.java - extends AuditLogTable
• paimon-core/.../table/system/AuditLogTable.java - wraps FileStoreTable

This makes them ideal candidates for native reading in Doris.

Proposal

Refactor the system table query path to support native reading for data-oriented Paimon system tables.

Phase 1: FE Refactoring

Goal: Route binlog/audit_log/ro queries through PaimonScanNode instead of MetadataScanNode.

1. Extend SysTable interface

   • Add useNativeTablePath() method to distinguish execution paths
   • Add getSchema() method for native path schema retrieval

2. Create PaimonSysExternalTable

   • New class extending PaimonExternalTable
   • Wraps source table with system table type
   • Returns Paimon BinlogTable/AuditLogTable instance from getPaimonTable()

3. Modify BindRelation

   • Check useNativeTablePath() before creating TVF relation
   • Create LogicalFileScan for native-path system tables

4. Adapt PaimonScanNode

   • Support PaimonSysExternalTable as scan source
   • Generate splits from system table's ReadBuilder
   • Pass system table type to BE via TPaimonFileDesc

Phase 2: BE Native Reader Implementation

Goal: Implement native readers for binlog/audit_log with row transformation logic.

1. Extend Thrift definitions

   • Add sys_table_type field to TPaimonFileDesc
   • Add force_keep_delete and is_streaming flags

2. Implement PaimonAuditLogReader

   • Wrap native ORC/Parquet reader
   • Add rowkind column based on delete vectors
   • Support forceKeepDelete semantics

3. Implement PaimonBinlogReader

   • Extend PaimonAuditLogReader
   • Convert columns to array types
   • Pack UPDATE_BEFORE/UPDATE_AFTER pairs (streaming mode)

Expected Architecture

SQL: SELECT * FROM catalog.db.table$binlog
                    ↓
    BindRelation (useNativeTablePath=true)
                    ↓
    PaimonSysExternalTable (wraps source table)
                    ↓
    PaimonScanNode (reuse existing logic)
      - Serialize BinlogTable
      - Get DataSplits from BinlogTable
      - Set sys_table_type="binlog"
                    ↓
    BE: PaimonBinlogReader
      - Native ORC/Parquet reading
      - Add rowkind column
      - Array conversion for binlog
      - Changelog packing (streaming)

Benefits

1. Performance: Leverage native vectorized readers, avoid JNI overhead
2. Predicate Pushdown: Native readers support efficient filtering
3. Resource Efficiency: Reduced memory copying between Java and C++
4. Consistency: Unified execution path with regular Paimon tables
5. Scalability: Better performance for large-scale CDC scenarios

Tasks

• Phase 1.1: Extend SysTable interface with useNativeTablePath()
• Phase 1.2: Implement PaimonSysExternalTable class
• Phase 1.3: Modify BindRelation to support native path routing
• Phase 1.4: Adapt PaimonScanNode for system tables
• Phase 2.1: Extend Thrift definitions for system table params
• Phase 2.2: Implement PaimonAuditLogReader
• Phase 2.3: Implement PaimonBinlogReader
• Phase 2.4: Add regression tests

Related

• Paimon BinlogTable: https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/table/system/BinlogTable.java
• Paimon AuditLogTable: https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/table/system/AuditLogTable.java

Use Case

This feature is particularly valuable for:

• Real-time CDC pipelines reading Paimon binlog
• Data auditing scenarios with large audit_log tables
• Read-optimized queries on Paimon tables with ro system table