[GLUTEN-8581][VL] Fix Spark legacy date formatter under case insensitive configuration by weixiuli · Pull Request #8583 · apache/gluten

weixiuli · 2025-01-21T07:28:16Z

What changes were proposed in this pull request?

The #7375 has enabled Velox simple date time formatter if the spark.sql.legacy.timeParserPolicy is set to LEGACY. But, it doesn't work when the spark.sql.legacy.timeParserPolicy is set to legacy.

(Fixes: #8581)

How was this patch tested?

Add unittests.

github-actions · 2025-01-21T07:28:34Z

#8581

github-actions · 2025-01-21T07:28:48Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-01-21T07:56:24Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-01-21T08:51:12Z

Run Gluten Clickhouse CI on x86

…rk.sql.legacy.timeParserPolicy is set to LEGACY or legacy.

github-actions · 2025-01-21T11:15:46Z

Run Gluten Clickhouse CI on x86

cpp/velox/compute/WholeStageResultIterator.cc

github-actions · 2025-01-22T01:30:14Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-01-22T01:30:34Z

Run Gluten Clickhouse CI on x86

cpp/velox/compute/WholeStageResultIterator.cc

philo-he

Two comments. Thank you!

cpp/velox/compute/WholeStageResultIterator.cc

gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDateFunctionsSuite.scala

github-actions · 2025-01-23T08:55:01Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-01-23T08:55:59Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-01-24T01:42:49Z

Run Gluten Clickhouse CI on x86

gluten-ut/spark32/src/test/scala/org/apache/spark/sql/GlutenDateFunctionsSuite.scala

gluten-ut/spark33/src/test/scala/org/apache/spark/sql/GlutenDateFunctionsSuite.scala

gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDateFunctionsSuite.scala

gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDateFunctionsSuite.scala

github-actions · 2025-01-24T03:32:37Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-01-24T06:38:25Z

Run Gluten Clickhouse CI on x86

weixiuli · 2025-01-24T09:09:04Z

@philo-he @zhztheplayer @zml1206 @jackylee-ch @FelixYBW

zml1206

Thank you.

philo-he

Thank you!

GlutenPerfBot · 2025-01-25T17:55:23Z

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query	log/native_master_01_25_2025_time.csv	log/native_master_01_24_2025_beadc1a777_time.csv	difference	percentage
q1	16.11	15.58	-0.533	96.69%
q2	15.59	15.11	-0.475	96.95%
q3	3.56	3.68	0.125	103.51%
q4	85.74	85.32	-0.421	99.51%
q5	11.43	12.79	1.352	111.83%
q6	5.32	4.96	-0.351	93.39%
q7	7.94	8.64	0.698	108.79%
q8	4.94	4.85	-0.096	98.05%
q9	30.77	30.10	-0.671	97.82%
q10	12.29	11.92	-0.366	97.02%
q11	42.83	42.32	-0.506	98.82%
q12	2.18	2.05	-0.131	94.01%
q13	10.69	10.61	-0.088	99.18%
q14a	70.85	63.47	-7.379	89.58%
q14b	57.19	56.62	-0.566	99.01%
q15	3.82	4.04	0.222	105.82%
q16	29.25	28.93	-0.316	98.92%
q17	8.19	7.53	-0.655	92.00%
q18	10.02	10.77	0.757	107.56%
q19	3.74	5.25	1.506	140.22%
q20	2.75	2.01	-0.745	72.95%
q21	1.64	1.88	0.236	114.33%
q22	9.73	10.09	0.366	103.76%
q23a	139.84	139.86	0.023	100.02%
q23b	166.51	162.78	-3.731	97.76%
q24a	109.14	110.17	1.023	100.94%
q24b	97.46	96.69	-0.770	99.21%
q25	7.83	8.70	0.863	111.01%
q26	4.04	4.70	0.665	116.48%
q27	5.79	5.57	-0.216	96.26%
q28	38.07	36.80	-1.269	96.67%
q29	18.93	18.30	-0.630	96.67%
q30	6.45	6.58	0.122	101.89%
q31	11.92	12.03	0.110	100.92%
q32	2.29	2.36	0.071	103.11%
q33	6.96	7.45	0.488	107.01%
q34	4.45	4.48	0.036	100.81%
q35	11.21	9.97	-1.240	88.94%
q36	5.93	5.60	-0.327	94.49%
q37	5.27	5.37	0.097	101.83%
q38	17.45	17.41	-0.041	99.77%
q39a	4.71	4.31	-0.398	91.54%
q39b	4.94	5.05	0.117	102.37%
q40	5.24	5.41	0.171	103.27%
q41	0.89	1.19	0.306	134.47%
q42	1.14	1.31	0.172	115.10%
q43	4.45	4.52	0.068	101.52%
q44	11.60	11.49	-0.110	99.05%
q45	4.80	4.12	-0.682	85.78%
q46	5.59	5.42	-0.169	96.98%
q47	24.12	19.56	-4.560	81.10%
q48	7.13	6.65	-0.488	93.16%
q49	9.96	10.67	0.710	107.13%
q50	38.76	38.54	-0.217	99.44%
q51	14.55	14.50	-0.047	99.68%
q52	1.27	1.18	-0.085	93.29%
q53	3.15	2.98	-0.169	94.62%
q54	6.74	6.77	0.030	100.45%
q55	1.47	1.48	0.008	100.56%
q56	7.38	6.98	-0.399	94.60%
q57	13.16	13.31	0.145	101.10%
q58	3.40	3.46	0.055	101.63%
q59	6.66	7.27	0.611	109.18%
q60	8.38	10.74	2.362	128.18%
q61	8.56	9.60	1.037	112.12%
q62	5.31	4.92	-0.395	92.56%
q63	3.53	3.07	-0.463	86.90%
q64	66.35	60.51	-5.841	91.20%
q65	29.79	29.78	-0.013	99.96%
q66	4.60	4.53	-0.075	98.37%
q67	227.98	226.87	-1.112	99.51%
q68	4.60	4.09	-0.516	88.80%
q69	6.94	7.01	0.071	101.03%
q70	12.50	14.72	2.218	117.74%
q71	6.20	4.26	-1.934	68.79%
q72	39.70	38.09	-1.612	95.94%
q73	3.40	3.19	-0.210	93.83%
q74	28.15	26.97	-1.182	95.80%
q75	42.73	43.84	1.107	102.59%
q76	15.07	14.35	-0.714	95.26%
q77	3.61	3.29	-0.311	91.38%
q78	84.06	84.25	0.189	100.23%
q79	5.23	5.22	-0.002	99.96%
q80	17.28	17.40	0.127	100.74%
q81	8.88	8.02	-0.863	90.28%
q82	10.57	10.01	-0.566	94.64%
q83	2.91	2.72	-0.196	93.28%
q84	4.00	4.23	0.234	105.85%
q85	9.64	10.15	0.516	105.36%
q86	4.57	4.48	-0.096	97.89%
q87	18.84	17.83	-1.019	94.59%
q88	23.47	24.08	0.608	102.59%
q89	4.76	4.64	-0.122	97.44%
q90	3.29	3.47	0.181	105.50%
q91	5.34	6.44	1.097	120.54%
q92	2.27	2.51	0.234	110.31%
q93	54.95	52.07	-2.888	94.74%
q94	17.00	17.68	0.679	103.99%
q9	96.44	101.04	4.603	104.77%
q5	3.84	3.11	-0.725	81.09%
q96	27.88	28.63	0.744	102.67%
q97	2.67	2.61	-0.051	98.08%
q98	10.02	10.48	0.461	104.60%
q99	10.02	10.48	0.461	104.60%
total	2222.54	2200.41	-22.129	99.00%

…ive configuration (apache#8583)

github-actions bot added CORE works for Gluten Core VELOX labels Jan 21, 2025

weixiuli force-pushed the timeParserPolicy branch from 1f05a5a to c4c1e99 Compare January 21, 2025 07:55

weixiuli force-pushed the timeParserPolicy branch from c4c1e99 to e6e1039 Compare January 21, 2025 08:50

[GLUTEN-8581][VL] Support Spark legacy date formatter whether the spa…

a84c127

…rk.sql.legacy.timeParserPolicy is set to LEGACY or legacy.

weixiuli force-pushed the timeParserPolicy branch from e6e1039 to a84c127 Compare January 21, 2025 11:15

FelixYBW reviewed Jan 21, 2025

View reviewed changes

cpp/velox/compute/WholeStageResultIterator.cc Outdated Show resolved Hide resolved

Address comments

daff06f

weixiuli force-pushed the timeParserPolicy branch from e7079e6 to daff06f Compare January 22, 2025 01:30

philo-he changed the title ~~[GLUTEN-8581][VL] Support Spark legacy date formatter whether the spark.sql.legacy.timeParserPolicy is set to LEGACY or legacy.~~ [GLUTEN-8581][VL] Fix Spark legacy date formatter under case insensitive configuration Jan 22, 2025

jackylee-ch reviewed Jan 22, 2025

View reviewed changes

cpp/velox/compute/WholeStageResultIterator.cc Outdated Show resolved Hide resolved

philo-he reviewed Jan 22, 2025

View reviewed changes

cpp/velox/compute/WholeStageResultIterator.cc Outdated Show resolved Hide resolved

gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDateFunctionsSuite.scala Outdated Show resolved Hide resolved

weixiuli force-pushed the timeParserPolicy branch from 02ff850 to 5500591 Compare January 23, 2025 08:55

Address comments

7e1d7cb

weixiuli force-pushed the timeParserPolicy branch from 5500591 to 7e1d7cb Compare January 24, 2025 01:42

zml1206 reviewed Jan 24, 2025

View reviewed changes

Address comments

50fc3c4

weixiuli force-pushed the timeParserPolicy branch from 7266254 to 50fc3c4 Compare January 24, 2025 06:37

FelixYBW approved these changes Jan 24, 2025

View reviewed changes

zml1206 approved these changes Jan 25, 2025

View reviewed changes

philo-he approved these changes Jan 25, 2025

View reviewed changes

philo-he merged commit 70c3259 into apache:main Jan 25, 2025

baibaichen pushed a commit to baibaichen/gluten that referenced this pull request Feb 1, 2025

[GLUTEN-8581][VL] Fix Spark legacy date formatter under case insensit…

7a04e0f

…ive configuration (apache#8583)

jinchengchenghh mentioned this pull request Apr 1, 2025

[CORE] Fix duplicate setting for config LEGACY_TIME_PARSER_POLICY #9201

Merged

Conversation

weixiuli commented Jan 21, 2025

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

github-actions bot commented Jan 21, 2025

Uh oh!

github-actions bot commented Jan 21, 2025

Uh oh!

github-actions bot commented Jan 21, 2025

Uh oh!

github-actions bot commented Jan 21, 2025

Uh oh!

github-actions bot commented Jan 21, 2025

Uh oh!

Uh oh!

github-actions bot commented Jan 22, 2025

Uh oh!

github-actions bot commented Jan 22, 2025

Uh oh!

Uh oh!

philo-he left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 23, 2025

Uh oh!

github-actions bot commented Jan 23, 2025

Uh oh!

github-actions bot commented Jan 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 24, 2025

Uh oh!

github-actions bot commented Jan 24, 2025

Uh oh!

weixiuli commented Jan 24, 2025

Uh oh!

zml1206 left a comment

Choose a reason for hiding this comment

Uh oh!

philo-he left a comment

Choose a reason for hiding this comment

Uh oh!

GlutenPerfBot commented Jan 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants