Repository navigation
Commit 789106b
[SPARK-58949][CORE] Use unsigned UTF-8 byte ordering for Variant object fields
### What changes were proposed in this pull request?
This PR fixes [SPARK-58949](https://issues.apache.org/jira/browse/SPARK-58949) by:
- sorting newly written Variant object fields by unsigned lexicographic UTF-8 bytes, as required by the [Variant encoding specification](https://github.com/apache/parquet-format/blob/24102ed5c56e51b610a4897e5f79e76e43732d1d/VariantEncoding.md#L449-L463);
- using binary search for object lookup at every object size and comparing UTF-8-encoded keys using unsigned byte ordering;
- retrying lookup with Java UTF-16 order when needed so values written by older Spark versions remain readable; and
- accepting both canonical UTF-8 order and legacy UTF-16 order during schema validation, while preserving the established schema field order.
This follows the compatibility direction discussed in [apache/parquet-java#3736](apache/parquet-java#3736).
### Why are the changes needed?
The Variant specification orders object keys by unsigned UTF-8 bytes, but Spark used String.compareTo, which orders UTF-16 code units. These orders differ for some valid keys. For example, UTF-16 places U+10000 before U+FFFF, while unsigned UTF-8 places U+FFFF first.
As a result, Spark wrote non-canonical Variant objects and could miss fields when binary-searching canonical values produced by another implementation.
### Does this PR introduce _any_ user-facing change?
Yes. Newly written Variant objects use the specification's unsigned UTF-8 field order. Spark continues to read affected values written in the legacy UTF-16 order, and schema output keeps its existing field order.
### How was this patch tested?
Added regressions for canonical and legacy object lookup, nested objects, schema_of_variant, and Parquet shredding-schema inference:
build/sbt \
'catalyst/testOnly *VariantExpressionSuite -- -z "SPARK-58949"' \
'sql/testOnly *VariantInferShreddingSuite -- -z "SPARK-58949"'
Both suites passed (1 test each). The affected modules also passed Java checkstyle and main/test scalastyle.
I also ran a temporary lookup microbenchmark on an Apple M4 with Zulu OpenJDK 21.0.6, comparing upstream/master at 9da9f8d with this patch at b3a050b. Each result is the median of three alternating JVM fork medians. Each fork used a fixed 2 GiB heap, 5 seconds of warmup per case, and 9 measured rounds of 5,000,000 lookups. Object construction was excluded. Lower is better.
| Object / lookup | master (ns/op) | patch (ns/op) | Change |
| --- | ---: | ---: | ---: |
| 16 fields, ASCII present | 193.7 | 94.6 | -51.2% |
| 16 fields, ASCII absent | 180.3 | 87.2 | -51.6% |
| 256 fields, canonical present | 102.2 | 140.6 | +37.6% |
| 256 fields, canonical absent | 111.5 | 147.0 | +31.8% (approx.) |
| 256 fields, legacy fallback present | 196.5 | 366.3 | +86.4% |
| 256 fields, legacy fallback absent | 201.7 | 367.0 | +82.0% |
The correctness assertion for a canonical U+10000 lookup changes from false on master to true with this patch. Following parquet-java's implementation, each canonical binary-search probe decodes the metadata key and re-encodes it as UTF-8 instead of maintaining a separate raw-metadata comparator. This keeps the implementation simple but adds allocations: the measured 256-field canonical cases regress by 32-38%, and the legacy fallback cases regress by 82-86%. The 16-field cases improve by about 51% because they now use binary search instead of the previous linear scan. One canonical-absent fork was noisy, so that aggregate is marked approximate.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
Closes #58239 from peterxcli/SPARK-58949-variant-utf8-order.
Authored-by: peterxcli <[email protected]>
Signed-off-by: Chao Sun <[email protected]>
(cherry picked from commit 2a61a79)
Signed-off-by: Chao Sun <[email protected]>1 parent 800c803 commit 789106b
7 files changed
Lines changed: 156 additions & 28 deletions
File tree
- common/variant/src/main/java/org/apache/spark/types/variant
- sql
- catalyst/src
- main/scala/org/apache/spark/sql/catalyst/expressions/variant
- test/scala/org/apache/spark/sql/catalyst/expressions/variant
- core/src
- main/scala/org/apache/spark/sql/execution/datasources/parquet
- test/scala/org/apache/spark/sql/execution/datasources/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
144 | 144 | | |
145 | 145 | | |
146 | 146 | | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | | - | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
157 | 154 | | |
158 | | - | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
159 | 158 | | |
160 | 159 | | |
161 | 160 | | |
| |||
164 | 163 | | |
165 | 164 | | |
166 | 165 | | |
167 | | - | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
168 | 170 | | |
169 | 171 | | |
170 | 172 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
376 | 376 | | |
377 | 377 | | |
378 | 378 | | |
379 | | - | |
| 379 | + | |
380 | 380 | | |
381 | 381 | | |
382 | 382 | | |
| |||
994 | 994 | | |
995 | 995 | | |
996 | 996 | | |
| 997 | + | |
997 | 998 | | |
998 | 999 | | |
999 | 1000 | | |
| |||
1005 | 1006 | | |
1006 | 1007 | | |
1007 | 1008 | | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
1008 | 1016 | | |
1009 | 1017 | | |
1010 | | - | |
| 1018 | + | |
1011 | 1019 | | |
1012 | 1020 | | |
1013 | 1021 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
688 | 688 | | |
689 | 689 | | |
690 | 690 | | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
691 | 701 | | |
692 | 702 | | |
693 | 703 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1933 | 1933 | | |
1934 | 1934 | | |
1935 | 1935 | | |
1936 | | - | |
1937 | | - | |
| 1936 | + | |
| 1937 | + | |
| 1938 | + | |
| 1939 | + | |
1938 | 1940 | | |
1939 | | - | |
1940 | | - | |
1941 | | - | |
| 1941 | + | |
| 1942 | + | |
| 1943 | + | |
| 1944 | + | |
| 1945 | + | |
| 1946 | + | |
| 1947 | + | |
| 1948 | + | |
| 1949 | + | |
1942 | 1950 | | |
| 1951 | + | |
| 1952 | + | |
| 1953 | + | |
1943 | 1954 | | |
1944 | 1955 | | |
1945 | 1956 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
548 | 548 | | |
549 | 549 | | |
550 | 550 | | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
551 | 602 | | |
552 | 603 | | |
553 | 604 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
365 | 365 | | |
366 | 366 | | |
367 | 367 | | |
368 | | - | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
369 | 372 | | |
370 | | - | |
371 | | - | |
372 | | - | |
373 | | - | |
374 | | - | |
375 | | - | |
376 | | - | |
377 | | - | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
378 | 385 | | |
379 | 386 | | |
380 | 387 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
764 | 764 | | |
765 | 765 | | |
766 | 766 | | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
767 | 806 | | |
768 | 807 | | |
769 | 808 | | |
| |||
0 commit comments