Skip to content

Commit 75d80c7

Browse files
haiyangsun-dbHyukjinKwon
authored andcommitted
[SPARK-51079] Added constructor for ArrowSerializer for backward compatibility
### What changes were proposed in this pull request? Previously, [PR](#49790) extends ArrowSerializer class to support large variable types in pandas UDF. This PR makes it more backward-compatible by adding a new constructor for cases where the flag for large variable is irrelevant (default to false). ### Why are the changes needed? To achieve versionless in spark connect, we need to be careful about changing a client API in Scala. For example, the change of an interface like this could affect Scala UDF referring to this class (as it is part of the spark connect client package) but running with an older spark connect client. ### How was this patch tested? N/A. It's a new constructor. Closes #50513 from haiyangsun-db/SPARK-51079. Authored-by: Haiyang Sun <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
1 parent 24f600f commit 75d80c7

1 file changed

Lines changed: 6 additions & 0 deletions

File tree

‎sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,12 @@ class ArrowSerializer[T](
5252
private[this] val allocator: BufferAllocator,
5353
private[this] val timeZoneId: String,
5454
private[this] val largeVarTypes: Boolean) {
55+
56+
// SPARK-51079: keep the old constructor for backward-compatibility.
57+
def this(enc: AgnosticEncoder[T], allocator: BufferAllocator, timeZoneId: String) = {
58+
this(enc, allocator, timeZoneId, false)
59+
}
60+
5561
private val (root, serializer) =
5662
ArrowSerializer.serializerFor(enc, allocator, timeZoneId, largeVarTypes)
5763
private val vectors = root.getFieldVectors.asScala

0 commit comments

Comments
 (0)