Support parallel append-only writes in the DataFusion integration #1230

Search before asking

• I searched in the issues and found nothing similar.

Motivation

I noticed low CPU utilization when inserting data into append-only Paimon tables through DataFusion.

The current write path appears to limit parallelism: DataSinkExec requires a single input partition, and PaimonDataSink::write_all() creates one TableWrite and awaits each batch write sequentially. TableWrite::write_arrow_batch() also writes partition/bucket groups sequentially. Although file closing can run in background tasks, Parquet encoding runs synchronously within the writer task, limiting CPU parallelism during ingestion. Increasing target_partitions alone does not create additional Paimon writers.

An initial implementation maybe can focus on append-only INSERT INTO, leaving primary-key tables and other write modes unchanged. This should improve multicore utilization in data loading.

Solution

No response

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!