Avoid Spark driver IP binding failure due to VPN setting by flyrain · Pull Request #866 · apache/polaris

flyrain · 2025-01-24T02:11:37Z

Fixed Spark local run issue like this:

24/09/01 23:00:54 ERROR Inbox: Ignoring error
java.lang.NullPointerException: **Cannot invoke "org.apache.spark.storage.BlockManagerId.executorId()" because "idWithoutTopologyInfo"** is null
        at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:677)    
        at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:133)
        at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
        at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
        at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
        at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
        at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

flyrain · 2025-01-24T02:21:41Z

regtests/run_spark_sql.sh

Looks like this change is optional. It passed without the change for me locally.

flyrain · 2025-01-24T04:05:23Z

The test failed due to it use the test profile in application.properties. There are two ways to fix this

Using a different profile or even different application.properties file for the tests.
Creating a new profile for local run.

jbonofre · 2025-01-24T08:12:12Z

When possible, we should use the same application.properties between the app and the tests.
Imho, it's fine to create a dedicated profile for test/local run.

adutra · 2025-01-24T09:27:07Z

quarkus/defaults/src/main/resources/application.properties

My idea was to do exactly the opposite: make all tests (automated + regression) use the default authenticator. Cf #804.

That makes sense to me.

flyrain · 2025-01-24T22:03:36Z

Repurpose this PR as #804 will resolve the credential issue.

regtests/run_spark_sql.sh

adutra · 2025-01-25T11:01:28Z

regtests/run_spark_sql.sh

This change may not be needed anymore since I removed the default values for these environment variables in #804.

Tested again based on #804. This change is still needed for my local run. I think this is due a recent change of my local VPN setting. However I couldn't touch VPN settings by myself. Given this script is mainly used for local test, I think this change should be good.

Here is the error message without this change:

Exception in thread "main" java.io.IOException: Failed to send RPC /jars/software.amazon.awssdk_http-client-spi-2.23.19.jar to mac.lan/192.168.86.27:55366: io.netty.channel.StacklessClosedChannelException at org.apache.spark.network.client.TransportClient$2.handleFailure(TransportClient.java:164) at org.apache.spark.network.client.TransportClient$StdChannelListener.operationComplete(TransportClient.java:372) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:999) at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:860) at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:877) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856) at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:113) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:881) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856) at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:302) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879) at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:940) at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1247) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: io.netty.channel.StacklessClosedChannelException at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)

eric-maynard · 2025-01-27T18:28:39Z

regtests/run.sh

  export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
 fi
 export PYTHONPATH="${SPARK_HOME}/python/:${SPARK_HOME}/python/lib/py4j-0.10.9.7-src.zip:$PYTHONPATH"
+export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messing up driver local IP address binding


nit: space after #

eric-maynard · 2025-01-27T18:28:47Z

regtests/run_spark_sql.sh


 export SPARK_VERSION=spark-3.5.3
 export SPARK_DISTRIBUTION=${SPARK_VERSION}-bin-hadoop3
+export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messing up driver local IP address binding


nit: space after #

adutra · 2025-01-28T11:25:53Z

I also have a VPN on but wasn't having issues. I checked that this change does not introduce any regressions on my end, so LGTM.

flyrain · 2025-01-28T16:33:48Z

Thanks @eric-maynard and @adutra for the review

flyrain requested review from RussellSpitzer, adutra, ashvina, collado-mike, dennishuo, dimas-b, ebyhr, eric-maynard, jackye1995, jbonofre, snazy, takidau and vvcephei as code owners January 24, 2025 02:11

flyrain commented Jan 24, 2025

View reviewed changes

regtests/run_spark_sql.sh Outdated

Copy link

Contributor Author

flyrain Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this change is optional. It passed without the change for me locally.

flyrain mentioned this pull request Jan 24, 2025

Failed to run run_spark_sql.sh #865

Closed

adutra reviewed Jan 24, 2025

View reviewed changes

adutra mentioned this pull request Jan 24, 2025

Make regression tests use default authentication #804

Merged

flyrain force-pushed the test-token-fix branch from f14cc67 to 79930c7 Compare January 24, 2025 22:00

flyrain changed the title ~~Fix the Polaris test credentials~~ Avoid Jan 24, 2025

flyrain changed the title ~~Avoid~~ Avoid Spark driver IP binding failure due to VPN setting Jan 24, 2025

adutra reviewed Jan 25, 2025

View reviewed changes

regtests/run_spark_sql.sh Outdated Show resolved Hide resolved

adutra reviewed Jan 25, 2025

View reviewed changes

Yufei Gu added 2 commits January 25, 2025 09:50

fix the same issue in run.sh

25d9c5c

fix the same issue in run.sh

7ff9a37

flyrain force-pushed the test-token-fix branch from 63cb810 to 7ff9a37 Compare January 25, 2025 17:52

eric-maynard reviewed Jan 27, 2025

View reviewed changes

eric-maynard approved these changes Jan 27, 2025

View reviewed changes

Resolve comments

122ad65

flyrain requested a review from MonkeyCanCode as a code owner January 27, 2025 19:30

adutra approved these changes Jan 28, 2025

View reviewed changes

flyrain merged commit 2f664a3 into apache:main Jan 28, 2025
5 checks passed

Conversation

flyrain commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain commented Jan 24, 2025

Uh oh!

jbonofre commented Jan 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain commented Jan 24, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adutra commented Jan 28, 2025

Uh oh!

Uh oh!

flyrain commented Jan 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flyrain commented Jan 24, 2025 •

edited

Loading