Cast.needsTimeZone misses complex-to-STRING casts containing a timestamp leaf

XMLWordPrintableJSON

      Summary

      Cast.needsTimeZone classifies a cast from a complex type to STRING as time-zone independent,
      although Cast.castToString recurses into ARRAY/MAP/STRUCT and renders a nested TIMESTAMP with
      the session zoneId. The function is not self-consistent:

      needsTimeZone(TIMESTAMP, STRING)                  = true
      needsTimeZone(ARRAY<TIMESTAMP>, ARRAY<STRING>)    = true
      needsTimeZone(ARRAY<TIMESTAMP>, STRING)           = false   <-- renders with zoneId
      needsTimeZone(MAP<INT,TIMESTAMP>, STRING)         = false   <-- renders with zoneId
      needsTimeZone(STRUCT<t:TIMESTAMP>, STRING)        = false   <-- renders with zoneId
      needsTimeZone(ARRAY<ARRAY<TIMESTAMP>>, ARRAY<STRING>) = false
      

      The last line follows from the first: the complex-to-complex branch recurses to
      (ARRAY<TIMESTAMP>, STRING) and stops there.

      That the conversion really is zone dependent:

      val lit = Literal.create(Seq(0L), ArrayType(TimestampType))
      Cast(lit, StringType, Some("UTC")).eval(null)                 // [1970-01-01 00:00:00]
      Cast(lit, StringType, Some("America/Los_Angeles")).eval(null) // [1969-12-31 16:00:00]
      

      Consequences

      1. A zone-dependent cast is considered resolved without a time zone, and crashes on eval.

      val c = Cast(Literal.create(Seq(0L), ArrayType(TimestampType)), StringType)  // no timeZoneId
      c.resolved                      // true
      c.contextIndependentFoldable    // true
      c.eval(null)                    // java.util.NoSuchElementException: None.get
      

      Cast.resolved only requires timeZoneId.isDefined when needsTimeZone is true.
      ResolveTimeZone assigns the session zone unconditionally, so plans built by the analyzer are
      covered; a Cast constructed after that point is reported as resolved and then fails in
      TimeZoneAwareExpression.zoneId.

      2. Cast.canonicalized drops the time zone, so two casts with different zones compare equal.

      canonicalized calls basic.withTimeZone(null) when !needsTimeZone. With a shared Parquet
      table holding ts TIMESTAMP and ats ARRAY<TIMESTAMP>, caching under one session time zone and
      querying under another returns the stale rendering, because the cache lookup goes through
      sameResult:

      // q = select cast(<col> as string) as v from t
      spark.conf.set("spark.sql.session.timeZone", "UTC");  d1 = spark.sql(q); d1.cache(); d1.collect()
      spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles"); spark.sql(q).collect()
      
      col = ts   (needsTimeZone = true, control)
        plan under LA: *(1) Project [cast(ts#0 as string) AS v#31] +- FileScan parquet
        value under LA: [1969-12-31 16:00:00]                       correct, cache not reused
      
      col = ats  (needsTimeZone = false)
        plan under LA: InMemoryTableScan [v#64] +- InMemoryRelation ...
        value under LA: [[1970-01-01 00:00:00]]                     WRONG, UTC rendering reused
        value under LA after unpersist: [[1969-12-31 16:00:00]]      correct
      

      Measured on current master (5.0.0-SNAPSHOT), local[4], AQE disabled.

      3. Downstream users of the predicate inherit the misclassification, e.g.
      contextIndependentFoldable (used by V2ExpressionBuilder), the partition-predicate pushdown
      check in HiveShim, and ResolveDefaultColumnsUtil.

      Proposed fix

      Add one case to Cast.needsTimeZone: a cast whose target is STRING needs the session time zone if
      the source type recursively contains a time-zone-sensitive timestamp leaf (TIMESTAMP /
      TIMESTAMP_LTZ(p)), e.g. via DataType.existsRecursively. The existing complex-to-complex
      recursion then covers the nested ARRAY<ARRAY<TIMESTAMP>> -> ARRAY<STRING> shape as well, since
      it bottoms out in the new case. TIMESTAMP_NTZ stays zone independent, consistent with the scalar
      rule.

      Since the predicate becomes stricter, casts to STRING over complex types now require a
      timeZoneId to be resolved. Call sites that construct such a cast outside the analyzer need a
      zone, and plan-string golden files may change.

      Suggested coverage: extend the needsTimeZone assertions in CastSuiteBase with the
      complex-to-STRING pairs above (with TIMESTAMP_NTZ as the negative control), and add the
      cross-time-zone cache case.

      Notes

      Not a regression. Cast.needsTimeZone has had no complex-to-STRING case since branch-3.5, so
      every maintained branch is affected; the numbers above were measured on master.

      Found while reviewing https://github.com/apache/spark/pull/58424, where an optimizer rule uses
      Cast.needsTimeZone to decide whether a projection is repeatable across two evaluations. The gap
      is in the shared classification, not in that rule.

            Assignee:
            Hengrui Hu
            Reporter:
            Hengrui Hu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: