Skip to content

Commit 67c64ec

Browse files
author
Paul Mathew
committed
feat(2152): support pa.RecordBatchReader in Table.append/overwrite
Currently `Table.append(df)` and `Table.overwrite(df)` only accept a materialised `pa.Table`, which forces callers to load the entire dataset into memory before writing. This makes pyiceberg unusable for large or unbounded inputs and has been a recurring complaint (apache#1004, apache#2152, dlt-hub#3753). Allow `pa.RecordBatchReader` as an alternative input. When a reader is provided, batches are streamed and microbatched into target-sized Parquet files via the new `bin_pack_record_batches` helper, then committed in a single snapshot via the existing fast_append path. Memory is bounded by `write.target-file-size-bytes` (default 512 MiB) per worker rather than the full input size. Scope of this PR — unpartitioned tables only. Streaming into partitioned tables raises NotImplementedError pointing back to apache#2152; partitioned support needs additional design (high-cardinality partition handling, per-partition rolling writers) and is tracked as a follow-up. Mirrors iceberg-go#369's staging — that project shipped unpartitioned streaming first. Internal note: the implementation buffers up to `target_file_size` of in- memory RecordBatches before flushing to a Parquet file. A more memory- efficient rolling-ParquetWriter approach is a planned follow-up that will benefit from the `OutputStream.tell()` API added in apache#2998.
1 parent d008a04 commit 67c64ec

6 files changed

Lines changed: 497 additions & 29 deletions

File tree

‎mkdocs/docs/api.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,17 @@ for buf in tbl.scan().to_arrow_batch_reader():
365365
print(f"Buffer contains {len(buf)} rows")
366366
```
367367

368+
### Streaming writes from a `RecordBatchReader`
369+
370+
`tbl.append()` and `tbl.overwrite()` also accept a `pyarrow.RecordBatchReader` directly, which lets you write datasets that don't fit in memory without materialising them as a `pa.Table` first. PyIceberg consumes the reader once and microbatches it into Parquet files of approximately `write.target-file-size-bytes` (default 512 MiB), keeping memory usage bounded by the target size. All files are committed in a single snapshot.
371+
372+
```python
373+
reader = pa.RecordBatchReader.from_batches(schema, batch_iter)
374+
tbl.append(reader)
375+
```
376+
377+
Streaming writes are currently only supported on **unpartitioned** tables. For a partitioned table, materialise the reader as a `pa.Table` first, or follow [#2152](https://github.com/apache/iceberg-python/issues/2152) for the partitioned support tracked as a follow-up.
378+
368379
To avoid any type inconsistencies during writing, you can convert the Iceberg table schema to Arrow:
369380

370381
```python

‎pyiceberg/io/pyarrow.py‎

Lines changed: 76 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2666,6 +2666,18 @@ def write_parquet(task: WriteTask) -> DataFile:
26662666

26672667

26682668
def bin_pack_arrow_table(tbl: pa.Table, target_file_size: int) -> Iterator[list[pa.RecordBatch]]:
2669+
"""Bin-pack ``tbl`` into groups of RecordBatches, each ~``target_file_size``.
2670+
2671+
Note:
2672+
``target_file_size`` is measured in **uncompressed in-memory** Arrow bytes
2673+
(``Table.nbytes`` / ``RecordBatch.nbytes``), not compressed on-disk Parquet
2674+
bytes. The resulting Parquet file after compression (zstd by default,
2675+
plus dictionary/RLE encoding) is typically 3-10× smaller than
2676+
``target_file_size``. This is a coarse proxy for the spec-defined
2677+
``write.target-file-size-bytes`` and will be tightened to true on-disk
2678+
bytes once the writer is switched to a rolling-``ParquetWriter`` with
2679+
``OutputStream.tell()`` (#2998).
2680+
"""
26692681
from pyiceberg.utils.bin_packing import PackingIterator
26702682

26712683
avg_row_size_bytes = tbl.nbytes / tbl.num_rows
@@ -2681,6 +2693,41 @@ def bin_pack_arrow_table(tbl: pa.Table, target_file_size: int) -> Iterator[list[
26812693
return bin_packed_record_batches
26822694

26832695

2696+
def bin_pack_record_batches(batches: Iterable[pa.RecordBatch], target_file_size: int) -> Iterator[list[pa.RecordBatch]]:
2697+
"""Microbatch a single-pass stream of RecordBatches into target-sized groups.
2698+
2699+
Unlike :func:`bin_pack_arrow_table`, this consumes ``batches`` lazily and
2700+
holds at most one in-flight buffer in memory, bounded by ``target_file_size``.
2701+
Suitable for streaming inputs (``pa.RecordBatchReader``,
2702+
``Iterator[pa.RecordBatch]``) where the total size is unknown up front and
2703+
the caller cannot afford to materialise the full dataset.
2704+
2705+
Each yielded list of batches is intended to be written as a single Parquet
2706+
data file. Because this is single-pass FIFO accumulation (no lookback), the
2707+
last bin may be smaller than ``target_file_size``.
2708+
2709+
Note:
2710+
``target_file_size`` is measured in **uncompressed in-memory** Arrow
2711+
bytes (``RecordBatch.nbytes``), not compressed on-disk Parquet bytes.
2712+
The resulting Parquet file after compression is typically 3-10×
2713+
smaller than ``target_file_size``. Matches the existing
2714+
:func:`bin_pack_arrow_table` semantics; both will be tightened to true
2715+
on-disk bytes once the writer is switched to a rolling-
2716+
``ParquetWriter`` with ``OutputStream.tell()`` (#2998).
2717+
"""
2718+
buffer: list[pa.RecordBatch] = []
2719+
buffer_bytes = 0
2720+
for batch in batches:
2721+
buffer.append(batch)
2722+
buffer_bytes += batch.nbytes
2723+
if buffer_bytes >= target_file_size:
2724+
yield buffer
2725+
buffer = []
2726+
buffer_bytes = 0
2727+
if buffer:
2728+
yield buffer
2729+
2730+
26842731
def _check_pyarrow_schema_compatible(
26852732
requested_schema: Schema,
26862733
provided_schema: pa.Schema,
@@ -2800,15 +2847,24 @@ def _get_parquet_writer_kwargs(table_properties: Properties) -> dict[str, Any]:
28002847

28012848
def _dataframe_to_data_files(
28022849
table_metadata: TableMetadata,
2803-
df: pa.Table,
2850+
df: pa.Table | pa.RecordBatchReader,
28042851
io: FileIO,
28052852
write_uuid: uuid.UUID | None = None,
28062853
counter: itertools.count[int] | None = None,
28072854
) -> Iterable[DataFile]:
2808-
"""Convert a PyArrow table into a DataFile.
2855+
"""Convert a PyArrow Table or RecordBatchReader into DataFiles.
2856+
2857+
For a ``pa.Table`` the data is materialised in memory and bin-packed into
2858+
target-sized files (with partition splitting if the table is partitioned).
2859+
2860+
For a ``pa.RecordBatchReader`` batches are streamed and microbatched into
2861+
target-sized files using bounded memory (see :func:`bin_pack_record_batches`).
2862+
Streaming writes are currently only supported on unpartitioned tables;
2863+
partitioned support is tracked in
2864+
https://github.com/apache/iceberg-python/issues/2152.
28092865
28102866
Returns:
2811-
An iterable that supplies datafiles that represent the table.
2867+
An iterable that supplies datafiles that represent the input data.
28122868
"""
28132869
from pyiceberg.table import DOWNCAST_NS_TIMESTAMP_TO_US_ON_WRITE, TableProperties, WriteTask
28142870

@@ -2828,6 +2884,23 @@ def _dataframe_to_data_files(
28282884
format_version=table_metadata.format_version,
28292885
)
28302886

2887+
if isinstance(df, pa.RecordBatchReader):
2888+
if not table_metadata.spec().is_unpartitioned():
2889+
raise NotImplementedError(
2890+
"Writing a pa.RecordBatchReader to a partitioned table is not yet supported. "
2891+
"Materialise the reader as a pa.Table first, or follow "
2892+
"https://github.com/apache/iceberg-python/issues/2152 for partitioned streaming support."
2893+
)
2894+
yield from write_file(
2895+
io=io,
2896+
table_metadata=table_metadata,
2897+
tasks=(
2898+
WriteTask(write_uuid=write_uuid, task_id=next(counter), record_batches=batches, schema=task_schema)
2899+
for batches in bin_pack_record_batches(df, target_file_size)
2900+
),
2901+
)
2902+
return
2903+
28312904
if table_metadata.spec().is_unpartitioned():
28322905
yield from write_file(
28332906
io=io,

‎pyiceberg/table/__init__.py‎

Lines changed: 116 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -450,12 +450,53 @@ def update_statistics(self) -> UpdateStatistics:
450450
"""
451451
return UpdateStatistics(transaction=self)
452452

453-
def append(self, df: pa.Table, snapshot_properties: dict[str, str] = EMPTY_DICT, branch: str | None = MAIN_BRANCH) -> None:
453+
def append(
454+
self,
455+
df: pa.Table | pa.RecordBatchReader,
456+
snapshot_properties: dict[str, str] = EMPTY_DICT,
457+
branch: str | None = MAIN_BRANCH,
458+
) -> None:
454459
"""
455-
Shorthand API for appending a PyArrow table to a table transaction.
460+
Shorthand API for appending PyArrow data to a table transaction.
461+
462+
Accepts either a fully materialised ``pa.Table`` or a streaming
463+
``pa.RecordBatchReader``. Streaming is microbatched by
464+
``write.target-file-size-bytes`` so memory stays bounded; the reader is
465+
consumed once and cannot be reused.
466+
467+
Streaming writes are currently only supported on unpartitioned tables;
468+
passing a ``pa.RecordBatchReader`` for a partitioned table raises
469+
``NotImplementedError``. See
470+
https://github.com/apache/iceberg-python/issues/2152.
471+
472+
Note:
473+
When ``df`` is a ``pa.RecordBatchReader`` the reader is consumed
474+
once and cannot be replayed. If the catalog commit fails (e.g.
475+
``CommitFailedException`` from a concurrent writer) the reader is
476+
already drained and a naive retry will append zero rows. Callers
477+
that need at-least-once semantics should either:
478+
479+
- reconstruct the reader on each attempt via a factory callable,
480+
or
481+
- use a two-stage pattern — write Parquet files explicitly and
482+
then call :meth:`add_files` (whose input is a replayable list of
483+
paths) within a retry loop.
484+
485+
Failures during the write stage (mid-stream reader exception, S3
486+
errors) do not commit a snapshot, but may leave orphan data files
487+
in storage that are not referenced by any snapshot. Clean these
488+
up with expire/orphan-file maintenance jobs.
489+
490+
``write.target-file-size-bytes`` is currently interpreted as
491+
uncompressed in-memory Arrow bytes (the bin-packing weight) rather
492+
than compressed on-disk Parquet bytes. The resulting files are
493+
typically 3-10× smaller than the property suggests after
494+
compression. This matches the existing ``pa.Table`` write path and
495+
will be tightened once the writer is switched to a
496+
rolling-``ParquetWriter`` with ``OutputStream.tell()`` (#2998).
456497
457498
Args:
458-
df: The Arrow dataframe that will be appended to overwrite the table
499+
df: An Arrow Table or a RecordBatchReader of records to append.
459500
snapshot_properties: Custom properties to be added to the snapshot summary
460501
branch: Branch Reference to run the append operation
461502
"""
@@ -466,8 +507,8 @@ def append(self, df: pa.Table, snapshot_properties: dict[str, str] = EMPTY_DICT,
466507

467508
from pyiceberg.io.pyarrow import _check_pyarrow_schema_compatible, _dataframe_to_data_files
468509

469-
if not isinstance(df, pa.Table):
470-
raise ValueError(f"Expected PyArrow table, got: {df}")
510+
if not isinstance(df, (pa.Table, pa.RecordBatchReader)):
511+
raise ValueError(f"Expected pa.Table or pa.RecordBatchReader, got: {df}")
471512

472513
downcast_ns_timestamp_to_us = Config().get_bool(DOWNCAST_NS_TIMESTAMP_TO_US_ON_WRITE) or False
473514
_check_pyarrow_schema_compatible(
@@ -478,12 +519,14 @@ def append(self, df: pa.Table, snapshot_properties: dict[str, str] = EMPTY_DICT,
478519
)
479520

480521
with self._append_snapshot_producer(snapshot_properties, branch=branch) as append_files:
481-
# skip writing data files if the dataframe is empty
482-
if df.shape[0] > 0:
483-
data_files = list(
484-
_dataframe_to_data_files(
485-
table_metadata=self.table_metadata, write_uuid=append_files.commit_uuid, df=df, io=self._table.io
486-
)
522+
# For pa.Table we can short-circuit empty inputs cheaply. For a
523+
# RecordBatchReader the stream is consumed lazily by
524+
# _dataframe_to_data_files and an empty reader simply yields zero
525+
# data files (the snapshot is still committed for symmetry with the
526+
# pa.Table case where empty inputs also produce a snapshot).
527+
if isinstance(df, pa.RecordBatchReader) or df.shape[0] > 0:
528+
data_files = _dataframe_to_data_files(
529+
table_metadata=self.table_metadata, write_uuid=append_files.commit_uuid, df=df, io=self._table.io
487530
)
488531
for data_file in data_files:
489532
append_files.append_data_file(data_file)
@@ -555,14 +598,50 @@ def dynamic_partition_overwrite(
555598

556599
def overwrite(
557600
self,
558-
df: pa.Table,
601+
df: pa.Table | pa.RecordBatchReader,
559602
overwrite_filter: BooleanExpression | str = ALWAYS_TRUE,
560603
snapshot_properties: dict[str, str] = EMPTY_DICT,
561604
case_sensitive: bool = True,
562605
branch: str | None = MAIN_BRANCH,
563606
) -> None:
564607
"""
565-
Shorthand for adding a table overwrite with a PyArrow table to the transaction.
608+
Shorthand for adding a table overwrite with a PyArrow table or RecordBatchReader to the transaction.
609+
610+
Accepts either a fully materialised ``pa.Table`` or a streaming
611+
``pa.RecordBatchReader``. Streaming is microbatched by
612+
``write.target-file-size-bytes`` so memory stays bounded; the reader is
613+
consumed once and cannot be reused.
614+
615+
Streaming writes are currently only supported on unpartitioned tables;
616+
passing a ``pa.RecordBatchReader`` for a partitioned table raises
617+
``NotImplementedError``. See
618+
https://github.com/apache/iceberg-python/issues/2152.
619+
620+
Note:
621+
When ``df`` is a ``pa.RecordBatchReader`` the reader is consumed
622+
once and cannot be replayed. If the catalog commit fails (e.g.
623+
``CommitFailedException`` from a concurrent writer) the reader is
624+
already drained and a naive retry will write zero rows. Callers
625+
that need at-least-once semantics should either:
626+
627+
- reconstruct the reader on each attempt via a factory callable,
628+
or
629+
- use a two-stage pattern — write Parquet files explicitly and
630+
then call :meth:`add_files` (whose input is a replayable list
631+
of paths) within a retry loop.
632+
633+
Failures during the write stage (mid-stream reader exception, S3
634+
errors) do not commit a snapshot, but may leave orphan data files
635+
in storage that are not referenced by any snapshot. Clean these
636+
up with expire/orphan-file maintenance jobs.
637+
638+
``write.target-file-size-bytes`` is currently interpreted as
639+
uncompressed in-memory Arrow bytes (the bin-packing weight) rather
640+
than compressed on-disk Parquet bytes. The resulting files are
641+
typically 3-10× smaller than the property suggests after
642+
compression. This matches the existing ``pa.Table`` write path and
643+
will be tightened once the writer is switched to a
644+
rolling-``ParquetWriter`` with ``OutputStream.tell()`` (#2998).
566645
567646
An overwrite may produce zero or more snapshots based on the operation:
568647
@@ -571,7 +650,7 @@ def overwrite(
571650
- APPEND: In case new data is being inserted into the table.
572651
573652
Args:
574-
df: The Arrow dataframe that will be used to overwrite the table
653+
df: An Arrow Table or a RecordBatchReader of records to write.
575654
overwrite_filter: ALWAYS_TRUE when you overwrite all the data,
576655
or a boolean expression in case of a partial overwrite
577656
snapshot_properties: Custom properties to be added to the snapshot summary
@@ -585,8 +664,8 @@ def overwrite(
585664

586665
from pyiceberg.io.pyarrow import _check_pyarrow_schema_compatible, _dataframe_to_data_files
587666

588-
if not isinstance(df, pa.Table):
589-
raise ValueError(f"Expected PyArrow table, got: {df}")
667+
if not isinstance(df, (pa.Table, pa.RecordBatchReader)):
668+
raise ValueError(f"Expected pa.Table or pa.RecordBatchReader, got: {df}")
590669

591670
downcast_ns_timestamp_to_us = Config().get_bool(DOWNCAST_NS_TIMESTAMP_TO_US_ON_WRITE) or False
592671
_check_pyarrow_schema_compatible(
@@ -606,8 +685,8 @@ def overwrite(
606685
)
607686

608687
with self._append_snapshot_producer(snapshot_properties, branch=branch) as append_files:
609-
# skip writing data files if the dataframe is empty
610-
if df.shape[0] > 0:
688+
# See append() for the empty-input handling rationale.
689+
if isinstance(df, pa.RecordBatchReader) or df.shape[0] > 0:
611690
data_files = _dataframe_to_data_files(
612691
table_metadata=self.table_metadata, write_uuid=append_files.commit_uuid, df=df, io=self._table.io
613692
)
@@ -1373,12 +1452,21 @@ def upsert(
13731452
snapshot_properties=snapshot_properties,
13741453
)
13751454

1376-
def append(self, df: pa.Table, snapshot_properties: dict[str, str] = EMPTY_DICT, branch: str | None = MAIN_BRANCH) -> None:
1455+
def append(
1456+
self,
1457+
df: pa.Table | pa.RecordBatchReader,
1458+
snapshot_properties: dict[str, str] = EMPTY_DICT,
1459+
branch: str | None = MAIN_BRANCH,
1460+
) -> None:
13771461
"""
1378-
Shorthand API for appending a PyArrow table to the table.
1462+
Shorthand API for appending PyArrow data to the table.
1463+
1464+
Accepts either a ``pa.Table`` or a streaming ``pa.RecordBatchReader``.
1465+
See :meth:`Transaction.append` for streaming semantics and partition
1466+
limitations.
13791467
13801468
Args:
1381-
df: The Arrow dataframe that will be appended to overwrite the table
1469+
df: An Arrow Table or a RecordBatchReader of records to append.
13821470
snapshot_properties: Custom properties to be added to the snapshot summary
13831471
branch: Branch Reference to run the append operation
13841472
"""
@@ -1401,14 +1489,18 @@ def dynamic_partition_overwrite(
14011489

14021490
def overwrite(
14031491
self,
1404-
df: pa.Table,
1492+
df: pa.Table | pa.RecordBatchReader,
14051493
overwrite_filter: BooleanExpression | str = ALWAYS_TRUE,
14061494
snapshot_properties: dict[str, str] = EMPTY_DICT,
14071495
case_sensitive: bool = True,
14081496
branch: str | None = MAIN_BRANCH,
14091497
) -> None:
14101498
"""
1411-
Shorthand for overwriting the table with a PyArrow table.
1499+
Shorthand for overwriting the table with a PyArrow Table or RecordBatchReader.
1500+
1501+
Accepts either a ``pa.Table`` or a streaming ``pa.RecordBatchReader``.
1502+
See :meth:`Transaction.overwrite` for streaming semantics and partition
1503+
limitations.
14121504
14131505
An overwrite may produce zero or more snapshots based on the operation:
14141506
@@ -1417,7 +1509,7 @@ def overwrite(
14171509
- APPEND: In case new data is being inserted into the table.
14181510
14191511
Args:
1420-
df: The Arrow dataframe that will be used to overwrite the table
1512+
df: An Arrow Table or a RecordBatchReader of records to write.
14211513
overwrite_filter: ALWAYS_TRUE when you overwrite all the data,
14221514
or a boolean expression in case of a partial overwrite
14231515
snapshot_properties: Custom properties to be added to the snapshot summary

0 commit comments

Comments
 (0)