Repository navigation
Commit 61906ce
authored
[fix](be) Preserve signed decimal bounds in Iceberg ORC writes (#69888)
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
ORC `Int128` statistics already contain two's-complement bits, but the
Iceberg ORC writer negates negative values again before encoding them. A
file containing `[-100, -1]` consequently reports bounds `[100, 1]`, so
a negative predicate can prune a file that contains matching rows.
ORC footer statistics also trim trailing fractional zeros. For example,
`-1.00` is returned as `value=-1, scale=0`; a `DECIMAL(...,2)` manifest
needs the unscaled value `-100`. Preserving the sign alone therefore
leaves incorrect bounds.
Restore each bound to the declared column scale with the existing ORC
scaling utility, then serialize its original signed bits using minimal
big-endian encoding. Preserve necessary sign prefixes such as `128 -> 00
80` and `-129 -> FF 7F`.
The fix applies to newly collected bounds. Existing incorrect metrics
require recomputation or data-file rewriting.
### Release note
Correct decimal bounds emitted by Iceberg ORC writes to prevent
incorrect predicate pruning for negative values and values with trailing
fractional zeros.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Added BE tests cover 29 encoding cases, real ORC statistics at
precisions 1/9/18/38, scale restoration through scale 38, nullable
columns, and all-null columns. The external regression checks manifest
pruning for negative and mixed decimal data across decimal32/64/128.
Local validation:
- An isolated codec harness built from the production function passed
all 29 encoding cases.
- Five real ORC files passed independent Iceberg 1.12.0 manifest
write/read and predicate-pruning probes.
- clang-format 16, build hygiene, changed-line clang-tidy, and Groovy
4.0.19 syntax checks passed. Static analysis used real ASAN UT
compilation commands with PCH expanded as headers.
- The full local BE-UT build was stopped during the monolithic
dependency build before tests executed.
- Regenerated `test_iceberg_write_stats2.out` with the standard
regression runner after restoring decimal scales. Only `sql_4/5/7/9`
changed, and the decimal bounds match both stored rows and the Parquet
metrics.
- Re-ran `test_iceberg_write_stats2` and
`test_iceberg_orc_decimal_bounds` locally using this PR's CI FE/BE
artifacts: 2 suites passed, 0 failed, 0 skipped.
- Behavior changed:
- [ ] No.
- [x] Yes. Newly written ORC decimal bounds preserve the sign and the
column scale.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent deb0794 commit 61906ce
5 files changed
Lines changed: 275 additions & 26 deletions
File tree
- be
- src/format/transformer
- test/format/transformer
- regression-test
- data/external_table_p0/iceberg/write
- suites/external_table_p0/iceberg
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
677 | 677 | | |
678 | 678 | | |
679 | 679 | | |
680 | | - | |
681 | | - | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
682 | 683 | | |
683 | 684 | | |
684 | 685 | | |
685 | 686 | | |
686 | 687 | | |
687 | 688 | | |
688 | | - | |
689 | | - | |
690 | | - | |
691 | | - | |
692 | | - | |
693 | | - | |
694 | | - | |
695 | | - | |
696 | | - | |
697 | | - | |
698 | | - | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
699 | 696 | | |
700 | | - | |
701 | | - | |
702 | | - | |
703 | | - | |
704 | | - | |
705 | | - | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
706 | 701 | | |
707 | 702 | | |
708 | 703 | | |
| |||
711 | 706 | | |
712 | 707 | | |
713 | 708 | | |
714 | | - | |
| 709 | + | |
| 710 | + | |
715 | 711 | | |
716 | | - | |
| 712 | + | |
717 | 713 | | |
718 | 714 | | |
719 | 715 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
120 | | - | |
| 120 | + | |
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| 35 | + | |
34 | 36 | | |
35 | 37 | | |
36 | 38 | | |
| |||
48 | 50 | | |
49 | 51 | | |
50 | 52 | | |
| 53 | + | |
51 | 54 | | |
52 | 55 | | |
53 | 56 | | |
| |||
243 | 246 | | |
244 | 247 | | |
245 | 248 | | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
246 | 396 | | |
247 | 397 | | |
248 | 398 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
18 | 18 | | |
19 | 19 | | |
20 | | - | |
| 20 | + | |
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
| 32 | + | |
33 | 33 | | |
0 commit comments