Repository navigation
Commit 95e30c8
committed
Fix massive spill files for StringView/BinaryView columns
Add garbage collection for StringView and BinaryView arrays before spilling
to disk. This prevents sliced arrays from carrying their entire original
buffers when written to spill files.
Changes:
- Add gc_view_arrays() function to apply GC on view arrays
- Integrate GC into InProgressSpillFile::append_batch()
- Use simple threshold-based heuristic (100+ rows, 10KB+ buffer size)
Fixes apache#19414 where GROUP BY on StringView columns created 820MB spill files
instead of 33MB due to sliced arrays maintaining references to original buffers.
Testing shows 80-98% reduction in spill file sizes for typical GROUP BY workloads.1 parent 1acaf7a commit 95e30c8
2 files changed
Lines changed: 422 additions & 16 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
| |||
61 | 62 | | |
62 | 63 | | |
63 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
64 | 68 | | |
65 | | - | |
| 69 | + | |
66 | 70 | | |
67 | 71 | | |
68 | 72 | | |
69 | 73 | | |
70 | 74 | | |
71 | 75 | | |
72 | | - | |
73 | | - | |
74 | 76 | | |
75 | 77 | | |
76 | 78 | | |
77 | 79 | | |
78 | | - | |
| 80 | + | |
79 | 81 | | |
80 | 82 | | |
81 | 83 | | |
82 | | - | |
| 84 | + | |
83 | 85 | | |
84 | | - | |
85 | | - | |
86 | 86 | | |
87 | 87 | | |
88 | 88 | | |
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
106 | | - | |
107 | 106 | | |
108 | | - | |
109 | 107 | | |
110 | 108 | | |
111 | 109 | | |
| |||
0 commit comments