Skip to content

Native string-to-timestamp cast rejects zone suffixes Spark accepts (PST, CET, Zulu, Etc/GMT+8, +08:00:00, no space) #7209

Closed

Description

@andygrove

Describe the bug

The native string-to-timestamp cast rejects several zone suffixes that Spark accepts. Spark parses the zone with ZoneId.of (after its own getZoneId rewrites), so any id Java knows works. For these inputs Comet returns NULL with ANSI off and from try_cast. Under ANSI, the default on Spark 4, it raises CAST_INVALID_INPUT. Spark returns a value:

Input Spark (UTC session)
2020-01-01 12:34:56 PST 2020-01-01 20:34:56
2020-01-01 12:34:56 CET 2020-01-01 11:34:56
2020-01-01 12:34:56Zulu 2020-01-01 12:34:56
2020-01-01 12:34:56America/Los_Angeles (no space) 2020-01-01 20:34:56
2020-01-01 12:34:56 +08:00:00 2020-01-01 04:34:56
2020-01-01 12:34:56 Etc/GMT+8 2020-01-01 20:34:56

The same happens for TIMESTAMP_NTZ and in an America/Los_Angeles session. Etc/GMT+8 appears to be caught by the GMT prefix rule in extract_offset_suffix. Spark resolves PST through ZoneId.SHORT_IDS, which the native parser does not consult. The other forms (Z, UTC, UTC+8, GMT-08:00, +08, -0800, EST, IANA names after a space) match Spark.

The 1.1.0 release has the same gaps by code: extract_offset_suffix and chrono-tz 0.10.4 are unchanged since then.

Steps to reproduce

Spark 4.1, default configs, UTC session:

Seq("2020-01-01 12:34:56 PST", "2020-01-01 12:34:56Zulu").toDF("s").write.parquet("/tmp/tz")
spark.read.parquet("/tmp/tz").selectExpr("s", "CAST(s AS TIMESTAMP) t").show()
  • Spark: both rows parse.
  • Comet: the query fails with [CAST_INVALID_INPUT] The value '2020-01-01 12:34:56 PST' of the type "STRING" cannot be cast to "TIMESTAMP". With spark.sql.ansi.enabled=false, or with try_cast, t is NULL.

Note that unix_micros(CAST(s AS TIMESTAMP)) matches Spark, because unix_micros goes through the codegen dispatcher, which evaluates the cast in Spark's code.

Expected behavior

The same timestamps as Spark, or a fallback to Spark for zone suffixes the native parser does not resolve.

Additional context

Found in the branch-1.2 release audit (#7204). It is not a 1.2 regression. Related but different: #5674 and #6329 (both closed).

Activity

  1. added this to the 1.3.0 milestone on Oct 26, 2026
  2. added a commit that references this issue on Nov 10, 2026
    567dea8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions