Skip to content

Commit 6fb7f3a

Browse files
committed
Bound Parquet delete memory and match NaN rows
Delete held the preserved rows of every split until it wrote them, so a large delete could exhaust memory. It now checks each split target with only that target's rows in memory, then writes the targets in a second pass that queries each again, keeping every check before the first write. An existing split target was matched with PartialEq, where NaN never equals itself, so a rerun of an interrupted delete with NaN rows could not complete. Targets are now matched on the Arrow row encoding. DataFusion reads also lost NaN: with file statistics it replaces a column whose min equals its max with that value, and Parquet float statistics skip NaN, so NaN rows read back as the other rows' value (apache/datafusion#26490). Catalog sessions no longer collect file statistics; row group pruning still reads the file metadata.
1 parent ed1a59e commit 6fb7f3a

4 files changed

Lines changed: 325 additions & 162 deletions

File tree

‎crates/persistence/src/backend/parquet/catalog/write.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -825,7 +825,7 @@ pub(crate) struct CustomDataWrite {
825825
directory: String,
826826
start_ts: UnixNanos,
827827
end_ts: UnixNanos,
828-
batches: Vec<RecordBatch>,
828+
pub(crate) batches: Vec<RecordBatch>,
829829
}
830830

831831
#[cfg(test)]

0 commit comments

Comments
 (0)