Repository navigation
Commit 51955ec
### Rationale for this change
Parquet null-count statistics can be incorrect for fixed-width leaf columns nested below a repeated ancestor, such as `list<struct<...>>`.
When a list is null or empty, its descendant leaf does not produce a value in the leaf values buffer. `DefLevelsToBitmap` intentionally excludes these repeated-ancestor nulls from its `null_count`. `MaybeCalculateValidityBits` was using that value directly for the Parquet column statistics, causing the null count to be undercounted.
For example, a `list<struct<string, int32>>` column can report an incorrect `null_count` for the `int32` leaf even though the encoded definition levels correctly represent the null and empty list entries.
### What changes are included in this PR?
* Update `MaybeCalculateValidityBits` to calculate the total null count from `batch_size - out_values_to_write`.
* This includes nulls represented by repeated ancestors, such as null or empty lists, while preserving the existing validity bitmap behavior.
* Add a regression test covering a fixed-width `int32` leaf under `list<struct<...>>`.
### Are these changes tested?
Yes.
Added a regression test that verifies the `int32` leaf reports:
* `null_count = 3`
* `num_values = 2`
The focused regression test passes:
`StatisticsTest.FixedWidthLeafUnderListStructNullCount`
The existing `parquet-writer-test` also passes.
### Are there any user-facing changes?
Yes. This fixes incorrect Parquet column statistics for affected nested fixed-width columns. The change does not alter the encoded data or public APIs.
### This PR contains a "Critical Fix".
This fixes a bug that produces incorrect Parquet statistics. The underlying data remains correct, but the reported `null_count` for affected fixed-width leaf columns can be incorrect.
### Was AI used for this PR?
**PR code and description written by:**
* [x] Human
* [ ] AI
**Reviewed before submission by:**
* [x] Human
* [ ] AI
* [ ] Not reviewed
* GitHub Issue: #51097
Lead-authored-by: AnuragRaut08 <[email protected]>
Co-authored-by: Gang Wu <[email protected]>
Signed-off-by: Gang Wu <[email protected]>
1 parent 3ad410b commit 51955ec
2 files changed
Lines changed: 111 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
160 | 161 | | |
161 | 162 | | |
162 | 163 | | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
163 | 266 | | |
164 | 267 | | |
165 | 268 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1397 | 1397 | | |
1398 | 1398 | | |
1399 | 1399 | | |
| 1400 | + | |
1400 | 1401 | | |
1401 | 1402 | | |
1402 | 1403 | | |
1403 | | - | |
| 1404 | + | |
1404 | 1405 | | |
1405 | 1406 | | |
1406 | 1407 | | |
1407 | 1408 | | |
1408 | | - | |
| 1409 | + | |
1409 | 1410 | | |
1410 | | - | |
1411 | | - | |
| 1411 | + | |
| 1412 | + | |
1412 | 1413 | | |
1413 | 1414 | | |
1414 | 1415 | | |
| |||
2051 | 2052 | | |
2052 | 2053 | | |
2053 | 2054 | | |
2054 | | - | |
| 2055 | + | |
| 2056 | + | |
| 2057 | + | |
2055 | 2058 | | |
2056 | 2059 | | |
2057 | 2060 | | |
| |||
0 commit comments