Repository navigation
Commit c25c7b3
committed
fix: label cached timestamps UTC on the row write path too
The row write path passed conf.sessionLocalTimeZone into rowToArrowBatchIter
while the columnar path already encoded with CometArrowStream.NATIVE_TIMEZONE, so
the same logical cache had two physical formats depending on which path filled
it, and the row one persisted the writing session's timezone into cached data.
Unlike Spark's Arrow cache, whose RecordBatch is deliberately schema-less and
reconstructs the timezone on read, CometCachedBatch stores a full IPC stream
including the schema, so that label really is written down. Standardise on
NATIVE_TIMEZONE ("UTC") for TimestampType, per the analysis on the review thread.
This is a label only. Spark's internal timestamp representation is micros since
the Unix epoch regardless of session timezone, so no values are converted, and
matching Comet's native schema also avoids a cast at the native boundary.
TimestampNTZType already had no timezone: Utils.toArrowType maps it to
Timestamp(MICROSECOND, null) whatever is passed in.
The closure no longer needs the session timezone at all, so it no longer captures
anything derived from `conf`.
Test: "stores timestamps with a UTC schema label" caches a row-based (local Seq)
plan under two non-UTC session timezones, decodes the cached batches back through
the serializer, and asserts the Arrow field carries "UTC" -- then checks the
values and their string rendering still match Spark. Reverting the one-line change
fails it with `got [America/Los_Angeles]`, so it pins the format rather than
passing vacuously.
All 20 tests in CometInMemoryCacheSuite pass on spark-3.5.1 parent 1367665 commit c25c7b3
2 files changed
Lines changed: 92 additions & 2 deletions
File tree
- spark/src
- main/scala/org/apache/spark/sql/comet/execution/arrow
- test/scala/org/apache/comet/exec
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
363 | 363 | | |
364 | 364 | | |
365 | 365 | | |
366 | | - | |
367 | 366 | | |
368 | 367 | | |
369 | 368 | | |
370 | 369 | | |
371 | 370 | | |
372 | 371 | | |
373 | | - | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
374 | 381 | | |
375 | 382 | | |
376 | 383 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| 36 | + | |
35 | 37 | | |
36 | 38 | | |
37 | 39 | | |
| 40 | + | |
| 41 | + | |
38 | 42 | | |
39 | 43 | | |
40 | 44 | | |
| |||
521 | 525 | | |
522 | 526 | | |
523 | 527 | | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
524 | 607 | | |
525 | 608 | | |
526 | 609 | | |
| |||
0 commit comments