Skip to content

[ARROW] Update arguments of ArrowUtils#toArrowSchema function - #4923

Closed
cfmcgrady wants to merge 1 commit into
apache:masterfrom
cfmcgrady:arrow-toArrowSchema
Closed

cfmcgrady wants to merge 1 commit into
apache:masterfrom
cfmcgrady:arrow-toArrowSchema

Conversation

@cfmcgrady

Copy link
Copy Markdown
Contributor

Why are the changes needed?

to adapt Spark 3.5, the new conf spark.sql.execution.arrow.useLargeVarType was introduced in apache/spark#39572

the signature of function ArrowUtils#toArrowSchema before

   def toArrowSchema(
       schema: StructType,
       timeZoneId: String,
       errorOnDuplicatedFieldNames: Boolean): Schema

after

  def toArrowSchema(
      schema: StructType,
      timeZoneId: String,
      errorOnDuplicatedFieldNames: Boolean,
      largeVarTypes: Boolean = false): Schema

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before make a pull request

@turboFei turboFei closed this in f040d7c Jun 5, 2023
@turboFei turboFei added this to the v1.8.0 milestone Jun 5, 2023
@turboFei

turboFei commented Jun 5, 2023

Copy link
Copy Markdown
Member

thanks, merged to master

@cfmcgrady
cfmcgrady deleted the arrow-toArrowSchema branch June 5, 2023 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants