[VL] Enable Spark legacy date formatter if spark.sql.legacy.timeParserPolicy is set to 'LEGACY'#7375
Conversation
…cy is set to 'LEGACY'.
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? Then could you also rename commit message and pull request title in the following format? See also: |
|
Run Gluten Clickhouse CI |
|
@rui-mo Could you help to take a look please? Thanks. |
|
|
||
| // Enable Spark legacy date formatter if spark.sql.legacy.timeParserPolicy is set to 'LEGACY'. | ||
| if (veloxCfg_->get<std::string>(kSparkLegacyTimeParserPolicy, "") == "LEGACY") { | ||
| configs[velox::core::QueryConfig::kSparkLegacyDateFormatter] = "true"; |
There was a problem hiding this comment.
Do we need to set this config as false if 'kSparkLegacyTimeParserPolicy' is not 'LEGACY'?
There was a problem hiding this comment.
Thanks for your feedback. It is necessary to do this. Updated.
|
Run Gluten Clickhouse CI |
|
@rui-mo Could you help to merge this PR? Thanks. |
|
@NEUpanning does this work for spark properties |
|
@surnaik No, this PR only works for |
3 participants
What changes were proposed in this pull request?
Velox#10966 introduced simple date time formatter that aligns with Spark legacy date parsing/formatting behavior. This PR enables Velox simple date time formatter if
spark.sql.legacy.timeParserPolicyis set toLEGACY.Relates Velox issue: #10354
How was this patch tested?
Integration tests with Spark