Repository navigation
Commit 11e2e6c
[SPARK-60179][SQL] Treat a cast to STRING as time-zone dependent when the source type contains a nested TIMESTAMP_LTZ
### What changes were proposed in this pull request?
`Cast.needsTimeZone` incorrectly treats complex-to-STRING casts containing nested LTZ timestamps as time-zone independent, even though `castToString` uses the session time zone.
This PR fixes the classification by recursively checking the source type, including Python UDTs. TIMESTAMP_NTZ, DATE and TIME remain time-zone independent.
### Why are the changes needed?
The incorrect classification can cause unresolved time-zone access, incorrect expression equivalence, and wrong results when cached queries are reused across session time zones.
### Does this PR introduce any user-facing change?
Yes. Complex-to-STRING casts now correctly preserve time-zone semantics, preventing incorrect cross-time-zone cache reuse.
### How was this patch tested?
- Added tests for nested types, Python UDTs, expression resolution, semantic equality, and cross-time-zone cache reuse.
- Mutation testing confirmed that the affected cache tests fail without the fix, while the scalar TIMESTAMP control still passes.
- All 276 Cast tests and 107 CachedTableSuite tests pass.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Co-authored-by: Claude Code (Claude Opus 5)
Closes #59394 from hhr293/SPARK-60179.
Authored-by: hu hengrui <[email protected]>
Signed-off-by: Chao Sun <[email protected]>1 parent 4bc916c commit 11e2e6c
3 files changed
Lines changed: 106 additions & 0 deletions
File tree
- sql
- catalyst/src
- main/scala/org/apache/spark/sql/catalyst/expressions
- test/scala/org/apache/spark/sql/catalyst/expressions
- core/src/test/scala/org/apache/spark/sql
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
380 | 380 | | |
381 | 381 | | |
382 | 382 | | |
| 383 | + | |
| 384 | + | |
383 | 385 | | |
384 | 386 | | |
385 | 387 | | |
| |||
429 | 431 | | |
430 | 432 | | |
431 | 433 | | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
432 | 454 | | |
433 | 455 | | |
434 | 456 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2698 | 2698 | | |
2699 | 2699 | | |
2700 | 2700 | | |
| 2701 | + | |
| 2702 | + | |
| 2703 | + | |
| 2704 | + | |
| 2705 | + | |
| 2706 | + | |
| 2707 | + | |
| 2708 | + | |
| 2709 | + | |
| 2710 | + | |
| 2711 | + | |
| 2712 | + | |
| 2713 | + | |
| 2714 | + | |
| 2715 | + | |
| 2716 | + | |
| 2717 | + | |
| 2718 | + | |
| 2719 | + | |
| 2720 | + | |
| 2721 | + | |
| 2722 | + | |
| 2723 | + | |
| 2724 | + | |
| 2725 | + | |
| 2726 | + | |
| 2727 | + | |
| 2728 | + | |
| 2729 | + | |
| 2730 | + | |
| 2731 | + | |
| 2732 | + | |
| 2733 | + | |
| 2734 | + | |
| 2735 | + | |
| 2736 | + | |
| 2737 | + | |
| 2738 | + | |
| 2739 | + | |
| 2740 | + | |
| 2741 | + | |
| 2742 | + | |
| 2743 | + | |
| 2744 | + | |
| 2745 | + | |
| 2746 | + | |
| 2747 | + | |
| 2748 | + | |
| 2749 | + | |
| 2750 | + | |
| 2751 | + | |
| 2752 | + | |
| 2753 | + | |
2701 | 2754 | | |
2702 | 2755 | | |
2703 | 2756 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2853 | 2853 | | |
2854 | 2854 | | |
2855 | 2855 | | |
| 2856 | + | |
| 2857 | + | |
| 2858 | + | |
| 2859 | + | |
| 2860 | + | |
| 2861 | + | |
| 2862 | + | |
| 2863 | + | |
| 2864 | + | |
| 2865 | + | |
| 2866 | + | |
| 2867 | + | |
| 2868 | + | |
| 2869 | + | |
| 2870 | + | |
| 2871 | + | |
| 2872 | + | |
| 2873 | + | |
| 2874 | + | |
| 2875 | + | |
| 2876 | + | |
| 2877 | + | |
| 2878 | + | |
| 2879 | + | |
| 2880 | + | |
| 2881 | + | |
| 2882 | + | |
| 2883 | + | |
| 2884 | + | |
| 2885 | + | |
| 2886 | + | |
2856 | 2887 | | |
2857 | 2888 | | |
2858 | 2889 | | |
| |||
0 commit comments