Repository navigation
Commit 6fb7f3a
committed
Bound Parquet delete memory and match NaN rows
Delete held the preserved rows of every split until it wrote them, so a
large delete could exhaust memory. It now checks each split target with
only that target's rows in memory, then writes the targets in a second
pass that queries each again, keeping every check before the first
write.
An existing split target was matched with PartialEq, where NaN never
equals itself, so a rerun of an interrupted delete with NaN rows could
not complete. Targets are now matched on the Arrow row encoding.
DataFusion reads also lost NaN: with file statistics it replaces a
column whose min equals its max with that value, and Parquet float
statistics skip NaN, so NaN rows read back as the other rows' value
(apache/datafusion#26490). Catalog sessions no longer collect file
statistics; row group pruning still reads the file metadata.1 parent ed1a59e commit 6fb7f3a
4 files changed
Lines changed: 325 additions & 162 deletions
File tree
- crates/persistence
- src
- backend/parquet
- catalog
- common
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
825 | 825 | | |
826 | 826 | | |
827 | 827 | | |
828 | | - | |
| 828 | + | |
829 | 829 | | |
830 | 830 | | |
831 | 831 | | |
| |||
0 commit comments