Skip to content

Commit 05869bb

Browse files
HyukjinKwondongjoon-hyun
HyukjinKwon
authored and
dongjoon-hyun
committed
[SPARK-39979][SQL][FOLLOW-UP] Make spark.sql.execution.arrow.useLargeVarTypes as an internal configuration
### What changes were proposed in this pull request? This PR is a followup of #39572 that hides the `spark.sql.execution.arrow.useLargeVarTypes` configuration as an internal configuration. ### Why are the changes needed? As described in #41569, this feature only works for `mapInArrow`, and other cases cannot be completely supported because of Arrow side limitation, see apache/arrow#35289. Therefore, this PR hides this configuration as an internal one for now. ### Does this PR introduce _any_ user-facing change? No, this configuration was not released out yet. ### How was this patch tested? Ran the Scala linter. Closes #41584 from HyukjinKwon/SPARK-39979-followup2. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent fff0b71 commit 05869bb

1 file changed

Lines changed: 2 additions & 1 deletion

File tree

  • sql/catalyst/src/main/scala/org/apache/spark/sql/internal

‎sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2841,8 +2841,9 @@ object SQLConf {
28412841
.doc("When using Apache Arrow, use large variable width vectors for string and binary " +
28422842
"types. Regular string and binary types have a 2GiB limit for a column in a single " +
28432843
"record batch. Large variable types remove this limitation at the cost of higher memory " +
2844-
"usage per value.")
2844+
"usage per value. Note that this only works for DataFrame.mapInArrow.")
28452845
.version("3.5.0")
2846+
.internal()
28462847
.booleanConf
28472848
.createWithDefault(false)
28482849

0 commit comments

Comments
 (0)