You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bdbc460
Browse filesBrowse the repository at this point in the historyBrowse files
feat: give each default Spark-to-Arrow conversion its own spark.comet.convert config (#6602)
Add spark.comet.convert.range.enabled, spark.comet.convert.inMemoryCache.enabled,
spark.comet.convert.rdd.enabled and spark.comet.convert.oneRowRelation.enabled for the four
operators that spark.comet.sparkToColumnar.supportedOperatorList named by default. All are off
by default.
The list now defaults to empty and stays the way to convert other leaf operators. Using
spark.comet.sparkToColumnar.enabled and the list for the four operators above is deprecated: it
keeps working as before and logs a one-time warning naming the new config. The OneRowRelation
config matches both the RDDScanExec Spark plans before 4.1 and the OneRowRelationExec it plans
from 4.1.
CometTestBase and the suites that set the old keys move to the new ones.
0 commit comments