Skip to content

Snowflake Offline to Parquet export failure #3548

Closed

Description

@adamschmidt

Expected Behavior

Snowflake offline export to Parquet in S3 should successfully export the materialization data set, including timestamp fields.

Current Behavior

Export to S3 integration in Parquet format fails with the following message:

Error encountered when unloading to PARQUET: TIMESTAMP_TZ and LTZ types are not supported for unloading to Parquet. value get: TIMESTAMP_LTZ

This is a known issue with recommended guidance.

Steps to reproduce

  • Configure offline store in Snowflake
  • Attempt to run materialization to external destination (e.g. DynamoDB)
  • Confirm "temporary_" table is created in the offline schema and an attempt was made to write to the storage integration

Specifications

  • Version: 0.29.0+
  • Platform: LInux + EKS
  • Subsystem: Python, Snowflake, DynamoDB

Possible Solution

Two things will assist in resolving this issue:

  • update the data extraction to cast timestamp fields to string or convert to a UTC timestamp integer
  • add an optional configuration option to enable timestamp field treatment, default to None/False
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions