Skip to content

Commit c0c8fc0

Browse files
faysoucjdsellers
authored andcommitted
Bound Parquet delete memory and match NaN rows
Delete held the preserved rows of every split until it wrote them, so a large delete could exhaust memory. It now checks each split target with only that target's rows in memory, then writes the targets in a second pass that queries each again, keeping every check before the first write. An existing split target was matched with PartialEq, where NaN never equals itself, so a rerun of an interrupted delete with NaN rows could not complete. Targets are now matched on the Arrow row encoding. DataFusion reads also lost NaN: with file statistics it replaces a column whose min equals its max with that value, and Parquet float statistics skip NaN, so NaN rows read back as the other rows' value (apache/datafusion#26490). Catalog sessions no longer collect file statistics; row group pruning still reads the file metadata.
1 parent e0918ec commit c0c8fc0

4 files changed

Lines changed: 325 additions & 162 deletions

File tree

‎crates/persistence/src/backend/parquet/catalog/write.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -825,7 +825,7 @@ pub(crate) struct CustomDataWrite {
825825
directory: String,
826826
start_ts: UnixNanos,
827827
end_ts: UnixNanos,
828-
batches: Vec<RecordBatch>,
828+
pub(crate) batches: Vec<RecordBatch>,
829829
}
830830

831831
#[cfg(test)]

0 commit comments

Comments
 (0)