Skip to content

[GLUTEN-8616] [VL] Make filescan limit for encrypted fallback as configurable#8621

Merged
Yohahaha merged 1 commit intoapache:mainfrom
ArnavBalyan:arnavb/configurable-limit
Feb 7, 2025
Merged

[GLUTEN-8616] [VL] Make filescan limit for encrypted fallback as configurable#8621
Yohahaha merged 1 commit intoapache:mainfrom
ArnavBalyan:arnavb/configurable-limit

Conversation

@ArnavBalyan
Copy link
Copy Markdown
Member

@ArnavBalyan ArnavBalyan commented Jan 26, 2025

What changes were proposed in this pull request?

  • Making parquet file scan limit as configurable.
  • Currently it's hardcoded to default 10, now available as a user config can be changed externally.
  • Default remains the same as before of 10.

How was this patch tested?

  • Existing UTs

@github-actions github-actions bot added CORE works for Gluten Core VELOX labels Jan 26, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 26, 2025

#8616

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 26, 2025

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/configurable-limit branch from acbcc3f to fc66b86 Compare January 26, 2025 09:48
@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 26, 2025

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/configurable-limit branch from fc66b86 to ce47d23 Compare January 26, 2025 12:06
@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 26, 2025

Run Gluten Clickhouse CI on x86

@ArnavBalyan
Copy link
Copy Markdown
Member Author

ArnavBalyan commented Jan 28, 2025

cc @zhztheplayer could you please review thanks!

jackylee-ch
jackylee-ch reviewed Jan 28, 2025
View reviewed changes
shims/common/src/main/scala/org/apache/gluten/config/GlutenConfig.scala
.doc("If supplied, `limit` number of files will be checked to determine encryption " +
"and falling back java scan")
.intConf
.createWithDefault(10)
Copy link
Copy Markdown
Contributor

@jackylee-ch jackylee-ch Jan 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: checkValue is required here to ensure the data is non-negative.

Copy link
Copy Markdown
Member Author

@ArnavBalyan ArnavBalyan Jan 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@ArnavBalyan ArnavBalyan force-pushed the arnavb/configurable-limit branch from ce47d23 to 9a27d24 Compare January 28, 2025 17:02
@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 28, 2025

Run Gluten Clickhouse CI on x86

@ArnavBalyan
Copy link
Copy Markdown
Member Author

ArnavBalyan commented Jan 30, 2025

cc @jackylee-ch fixed, can you please take a look

@Yohahaha Yohahaha merged commit 91800b4 into apache:main Feb 7, 2025
@GlutenPerfBot
Copy link
Copy Markdown
Contributor

GlutenPerfBot commented Feb 7, 2025

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_02_07_2025_time.csv log/native_master_02_06_2025_9aa21250fc_time.csv difference percentage
q1 16.34 16.43 0.092 100.56%
q2 16.42 14.47 -1.956 88.09%
q3 5.08 2.83 -2.254 55.66%
q4 86.63 85.55 -1.075 98.76%
q5 13.26 11.71 -1.545 88.35%
q6 5.39 3.72 -1.672 68.98%
q7 6.70 6.36 -0.345 94.85%
q8 4.66 6.62 1.968 142.28%
q9 28.40 29.47 1.070 103.77%
q10 13.29 12.20 -1.086 91.82%
q11 43.80 42.99 -0.806 98.16%
q12 2.31 2.13 -0.184 92.06%
q13 10.43 10.07 -0.365 96.50%
q14a 73.57 67.05 -6.519 91.14%
q14b 58.30 57.34 -0.960 98.35%
q15 3.31 3.73 0.423 112.79%
q16 29.50 29.96 0.466 101.58%
q17 8.49 7.37 -1.127 86.73%
q18 9.90 10.08 0.181 101.83%
q19 3.88 3.46 -0.419 89.19%
q20 2.34 4.35 2.007 185.70%
q21 1.54 1.50 -0.049 96.82%
q22 9.47 9.65 0.177 101.86%
q23a 137.99 136.87 -1.120 99.19%
q23b 163.33 162.14 -1.195 99.27%
q24a 109.31 101.26 -8.048 92.64%
q24b 92.81 96.16 3.352 103.61%
q25 6.58 6.32 -0.256 96.11%
q26 4.26 4.35 0.090 102.12%
q27 5.25 4.88 -0.377 92.82%
q28 35.57 38.90 3.329 109.36%
q29 18.06 18.64 0.576 103.19%
q30 8.74 6.85 -1.893 78.34%
q31 10.44 10.30 -0.142 98.64%
q32 2.62 2.01 -0.612 76.67%
q33 7.54 7.36 -0.179 97.63%
q34 4.65 4.67 0.020 100.43%
q35 11.38 10.69 -0.695 93.90%
q36 5.63 5.92 0.293 105.21%
q37 5.25 6.66 1.404 126.73%
q38 22.94 17.29 -5.646 75.38%
q39a 4.42 4.56 0.135 103.05%
q39b 5.12 5.16 0.045 100.87%
q40 6.57 5.35 -1.221 81.42%
q41 1.29 1.19 -0.101 92.19%
q42 1.18 1.19 0.012 101.04%
q43 4.57 4.49 -0.086 98.13%
q44 14.28 12.86 -1.418 90.07%
q45 4.52 4.74 0.224 104.95%
q46 5.79 5.51 -0.280 95.16%
q47 20.07 20.22 0.155 100.77%
q48 6.55 6.39 -0.161 97.55%
q49 10.12 10.35 0.230 102.27%
q50 39.28 38.54 -0.737 98.12%
q51 14.64 14.20 -0.436 97.02%
q52 1.16 1.48 0.327 128.24%
q53 3.03 2.86 -0.174 94.26%
q54 7.11 6.55 -0.566 92.05%
q55 1.39 3.51 2.118 252.51%
q56 6.95 7.43 0.485 106.98%
q57 13.69 13.09 -0.602 95.61%
q58 3.40 3.33 -0.070 97.94%
q59 6.83 6.63 -0.202 97.04%
q60 10.75 8.55 -2.205 79.50%
q61 10.08 7.88 -2.194 78.22%
q62 5.01 5.22 0.212 104.23%
q63 3.23 2.99 -0.236 92.70%
q64 64.96 64.08 -0.885 98.64%
q65 31.41 29.65 -1.754 94.42%
q66 4.98 4.93 -0.050 99.00%
q67 227.52 225.15 -2.376 98.96%
q68 5.16 4.29 -0.863 83.25%
q69 6.92 6.91 -0.007 99.90%
q70 12.79 12.74 -0.046 99.64%
q71 4.28 4.54 0.256 105.99%
q72 40.24 42.41 2.167 105.39%
q73 3.38 3.68 0.299 108.85%
q74 27.91 27.57 -0.340 98.78%
q75 43.10 44.85 1.748 104.06%
q76 15.04 14.10 -0.939 93.76%
q77 3.91 3.55 -0.355 90.91%
q78 85.34 84.79 -0.554 99.35%
q79 4.87 4.78 -0.094 98.08%
q80 17.08 17.25 0.172 101.01%
q81 8.91 9.87 0.955 110.72%
q82 10.48 10.33 -0.152 98.55%
q83 2.72 3.19 0.478 117.62%
q84 3.84 3.79 -0.040 98.94%
q85 9.66 10.26 0.599 106.20%
q86 4.62 4.52 -0.097 97.90%
q87 18.69 18.34 -0.342 98.17%
q88 23.39 24.05 0.662 102.83%
q89 4.28 4.86 0.583 113.62%
q90 3.83 3.37 -0.461 87.98%
q91 5.68 6.41 0.725 112.76%
q92 2.63 2.06 -0.565 78.50%
q93 54.69 55.27 0.581 101.06%
q94 18.06 17.27 -0.785 95.66%
q9 95.83 98.97 3.138 103.27%
q5 3.02 3.13 0.112 103.70%
q96 29.58 28.46 -1.118 96.22%
q97 3.40 2.77 -0.630 81.47%
q98 10.54 10.45 -0.082 99.22%
q99 10.54 10.45 -0.082 99.22%
total 2229.11 2197.23 -31.879 98.57%

@GlutenPerfBot
Copy link
Copy Markdown
Contributor

GlutenPerfBot commented Feb 7, 2025

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_02_07_2025_time.csv log/native_master_02_06_2025_9aa21250fc_time.csv difference percentage
q1 42.37 43.76 1.396 103.29%
q2 43.16 45.60 2.441 105.66%
q3 93.39 92.21 -1.184 98.73%
q4 69.99 71.59 1.596 102.28%
q5 180.58 181.10 0.518 100.29%
q6 19.22 20.05 0.830 104.32%
q7 105.61 106.27 0.664 100.63%
q8 186.19 185.66 -0.523 99.72%
q9 282.10 280.98 -1.116 99.60%
q10 103.10 100.90 -2.198 97.87%
q11 34.18 33.85 -0.326 99.05%
q12 42.68 43.24 0.561 101.31%
q13 77.43 75.97 -1.458 98.12%
q14 35.55 37.72 2.170 106.10%
q15 67.19 66.87 -0.314 99.53%
q16 28.37 28.07 -0.300 98.94%
q17 232.96 233.46 0.498 100.21%
q18 338.82 359.75 20.929 106.18%
q19 36.32 38.15 1.829 105.04%
q20 62.00 64.46 2.465 103.98%
q21 530.75 535.62 4.864 100.92%
q22 24.84 24.71 -0.130 99.48%
total 2636.78 2669.99 33.212 101.26%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jackylee-ch jackylee-ch jackylee-ch approved these changes

Assignees

No one assigned

Labels

CORE works for Gluten Core VELOX

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ArnavBalyan @GlutenPerfBot @jackylee-ch @Yohahaha