Skip to content

Bump Netty 4.1.93.Final#4992

Closed
pan3793 wants to merge 2 commits intoapache:masterfrom
pan3793:netty
Closed

Bump Netty 4.1.93.Final#4992
pan3793 wants to merge 2 commits intoapache:masterfrom
pan3793:netty

Conversation

@pan3793
Copy link
Member

@pan3793 pan3793 commented Jun 25, 2023

Why are the changes needed?

Upgrade Netty to the latest Arrow-compatible version: 4.1.93.Final

Currently, we can not upgrade to 4.1.94.Final or above because of netty/netty#13408

java.lang.NoSuchMethodError: 'io.netty.buffer.PoolThreadCache io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.threadCache()'
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:164)
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:214)
	at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:58)
	at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:77)
	at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:84)
	at org.apache.arrow.memory.NettyAllocationManager$1.create(NettyAllocationManager.java:34)
	at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:354)
	at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:349)
	at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:337)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:315)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:279)
	at org.apache.arrow.vector.BaseVariableWidthVector.allocateBytes(BaseVariableWidthVector.java:462)
	at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:420)
	at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:380)
	at org.apache.spark.sql.execution.arrow.ArrowWriter$.$anonfun$create$1(ArrowWriter.scala:42)

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before make a pull request

@pan3793 pan3793 requested a review from cfmcgrady June 25, 2023 02:53
@codecov-commenter
Copy link

codecov-commenter commented Jun 25, 2023

Codecov Report

Merging #4992 (9dd8f9a) into master (c2e861b) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #4992   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         561     561           
  Lines       30953   30953           
  Branches     4063    4063           
======================================
  Misses      30953   30953

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

cxzl25
cxzl25 reviewed Jun 26, 2023
View reviewed changes
Copy link
Contributor

@cxzl25 cxzl25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lower version of netty-handler has a CVE, and the failure level is set to moderate in PR #4266, causing GA to fail.

GHSA-6mjq-h674-j845

#4266 (comment)

https://github.com/actions/dependency-review-action

@pan3793
Copy link
Member Author

pan3793 commented Jun 26, 2023

@cxzl25 I see the CVE, looks acceptable. Also cc @bowenliang123

bowenliang123
bowenliang123 approved these changes Jun 26, 2023
View reviewed changes
Copy link
Contributor

@bowenliang123 bowenliang123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dependency review action only scans the changes and will not block the GA for the future PRs. The CVE-2023-34462 in Netty 4.1.93 is in moderate level and it seems acceptable.

@pan3793 pan3793 self-assigned this Jun 26, 2023
@pan3793 pan3793 added this to the v1.8.0 milestone Jun 26, 2023
@pan3793 pan3793 closed this in c8905bf Jun 26, 2023
@pan3793
Copy link
Member Author

pan3793 commented Jun 26, 2023

Thanks, merged to master

@pan3793 pan3793 deleted the netty branch October 7, 2023 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cxzl25 cxzl25 cxzl25 left review comments

@bowenliang123 bowenliang123 bowenliang123 approved these changes

@cfmcgrady cfmcgrady cfmcgrady approved these changes

Assignees

@pan3793 pan3793

Labels

kind:build

Projects

None yet

Milestone

v1.8.0

Development

Successfully merging this pull request may close these issues.

5 participants

@pan3793 @codecov-commenter @bowenliang123 @cxzl25 @cfmcgrady