Skip to content

[GLUTEN-6067][CH][Minor] Compile Spark-3.5 ut with backends-clickhouse#6114

Merged
baibaichen merged 2 commits intoapache:mainfrom
baibaichen:feature/spark_35_build
Jun 17, 2024
Merged

[GLUTEN-6067][CH][Minor] Compile Spark-3.5 ut with backends-clickhouse#6114
baibaichen merged 2 commits intoapache:mainfrom
baibaichen:feature/spark_35_build

Conversation

@baibaichen
Copy link
Copy Markdown
Contributor

@baibaichen baibaichen commented Jun 17, 2024

What changes were proposed in this pull request?

GlutenV1WriteCommandSuite and GlutenInsertSuite failed to compile with backends-clickhouse profiler. I refactor them in this PR, so we can compile spark-ut with spark-3.5 and backends-clickhouse profile. Once merged, we can add compile step in CICD asap to avoid regression.

How to do

I introduce GlutenColumnarWriteTestSupport trait, and use build-helper-maven-plugin to add diffierent implementation per backends

(Fixes: #6067)

How was this patch tested?

Existed UTs

@github-actions
Copy link
Copy Markdown

#6067

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

@zhouyuan
Copy link
Copy Markdown
Member

CC @JkSelf, please help to take a look

@baibaichen baibaichen force-pushed the feature/spark_35_build branch from 5834910 to 5ca17fc Compare June 17, 2024 09:58
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI

Copy link
Copy Markdown
Contributor

@JkSelf JkSelf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks.

@baibaichen baibaichen merged commit 5b87efa into apache:main Jun 17, 2024
@baibaichen baibaichen deleted the feature/spark_35_build branch June 17, 2024 13:39
@GlutenPerfBot
Copy link
Copy Markdown
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_06_17_2024_time.csv log/native_master_06_16_2024_a08a57c61_time.csv difference percentage
q1 34.50 37.13 2.635 107.64%
q2 23.69 24.00 0.302 101.27%
q3 39.94 37.23 -2.716 93.20%
q4 32.62 32.75 0.124 100.38%
q5 67.62 71.56 3.945 105.83%
q6 6.63 8.16 1.528 123.04%
q7 81.08 80.98 -0.094 99.88%
q8 85.44 86.78 1.338 101.57%
q9 120.83 121.57 0.744 100.62%
q10 46.34 44.17 -2.174 95.31%
q11 19.96 21.42 1.455 107.29%
q12 25.57 25.32 -0.250 99.02%
q13 38.36 39.84 1.478 103.85%
q14 19.32 19.23 -0.091 99.53%
q15 32.79 33.05 0.263 100.80%
q16 13.93 14.54 0.604 104.33%
q17 103.20 102.23 -0.963 99.07%
q18 147.38 142.93 -4.448 96.98%
q19 14.12 15.44 1.320 109.35%
q20 28.78 27.99 -0.786 97.27%
q21 258.28 261.41 3.128 101.21%
q22 12.23 12.41 0.188 101.53%
total 1252.62 1260.15 7.530 100.60%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CH] Support CH backend with Spark 3.5.x

4 participants