Repository navigation
Commit ecf5dfe
committed
apacheGH-3817: Simplify content defined chunking and align it with Arrow
- Chunker state lives per file: InternalParquetRecordWriter creates
the chunkers of the file up front, one per leaf column, only when
chunking is enabled, as Arrow C++ and arrow-rs do, and passes them
to every row group's store through a new
ParquetProperties.newColumnWriteStore overload. With chunking
enabled, a store made without them fails fast rather than silently
restarting the chunking. CdcOptions are validated when built.
- Content defined chunking is configured through one setter,
withContentDefinedChunking(CdcOptions), where null disables it, and
named as Arrow names it: CdcOptions, ContentDefinedChunker.
- ChunkingColumnWriter wraps each column writer once, when the store is
built. It rolls every triplet into the column's chunker and, at record
starts, ends the page where needNewChunk() says, as Arrow C++ does, or
where a page limit is reached, counted from the page start; cached
nulls are flushed on the unchunked schedule's maximum.
- Binary values are hashed through one path.
- The tests fail when chunking is switched off or broken, cover the
cases the Arrow C++ and arrow-rs suites do (V2 pages, dictionary
encoding, edits across row groups, a chunker per column, the record
writer path), and follow the project's test guidelines. The benchmark
writes one mixed dataset.1 parent 65c3e08 commit ecf5dfe
33 files changed
Lines changed: 1811 additions & 2528 deletions
File tree
- parquet-benchmarks/src/main/java/org/apache/parquet/benchmarks
- parquet-column/src
- main/java/org/apache/parquet
- column
- impl
- values
- dictionary
- factory
- fallback
- internal/column/chunking
- test/java/org/apache/parquet
- column
- impl
- values/dictionary
- internal/column/chunking
- parquet-hadoop
- src
- main/java/org/apache/parquet/hadoop
- test/java/org/apache/parquet/hadoop
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
22 | | - | |
| 21 | + | |
23 | 22 | | |
24 | 23 | | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
31 | 27 | | |
32 | 28 | | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
| 29 | + | |
37 | 30 | | |
38 | 31 | | |
39 | 32 | | |
| |||
50 | 43 | | |
51 | 44 | | |
52 | 45 | | |
53 | | - | |
54 | | - | |
| 46 | + | |
55 | 47 | | |
56 | 48 | | |
57 | 49 | | |
| |||
97 | 89 | | |
98 | 90 | | |
99 | 91 | | |
100 | | - | |
101 | | - | |
102 | | - | |
| 92 | + | |
103 | 93 | | |
104 | 94 | | |
105 | 95 | | |
106 | 96 | | |
107 | 97 | | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | 98 | | |
113 | 99 | | |
114 | 100 | | |
115 | 101 | | |
116 | 102 | | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
| 103 | + | |
121 | 104 | | |
122 | 105 | | |
123 | 106 | | |
124 | 107 | | |
125 | 108 | | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | 109 | | |
131 | 110 | | |
132 | 111 | | |
133 | 112 | | |
134 | 113 | | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
| 114 | + | |
| 115 | + | |
139 | 116 | | |
140 | 117 | | |
141 | 118 | | |
| |||
147 | 124 | | |
148 | 125 | | |
149 | 126 | | |
150 | | - | |
151 | | - | |
| 127 | + | |
| 128 | + | |
152 | 129 | | |
153 | 130 | | |
154 | 131 | | |
| |||
0 commit comments