I'm trying to send messages from Spark Google Dataproc to Google PubSub. But I'm stuck with the following errors when trying to initialize the PubSub client:
17/01/10 15:12:28 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, fca-cluster-dev-w-1.c.fcms-sandbox-gce.internal): java.lang.NoSuchMethodError: io.netty.util.AttributeKey.valueOf(Ljava/lang/Class;Ljava/lang/String;)Lio/netty/util/AttributeKey;
at io.grpc.netty.Utils.<clinit>(Utils.java:85)
at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:311)
at io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(NettyChannelBuilder.java:280)
at io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:230)
at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:239)
at com.google.api.gax.grpc.InstantiatingChannelProvider.createChannel(InstantiatingChannelProvider.java:119)
at com.google.api.gax.grpc.InstantiatingChannelProvider.getChannel(InstantiatingChannelProvider.java:106)
at com.google.api.gax.grpc.ProviderManager.getChannel(ProviderManager.java:106)
at com.google.api.gax.grpc.ChannelAndExecutor.create(ChannelAndExecutor.java:67)
at com.google.api.gax.grpc.ClientSettings.getChannelAndExecutor(ClientSettings.java:78)
at com.google.cloud.pubsub.spi.v1.PublisherClient.<init>(PublisherClient.java:182)
at com.google.cloud.pubsub.spi.v1.PublisherClient.create(PublisherClient.java:173)
at com.google.cloud.pubsub.spi.DefaultPubSubRpc.<init>(DefaultPubSubRpc.java:168)
at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubRpcFactory.create(PubSubOptions.java:69)
at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubRpcFactory.create(PubSubOptions.java:63)
at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:482)
at com.google.cloud.pubsub.PubSubImpl.<init>(PubSubImpl.java:115)
at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubFactory.create(PubSubOptions.java:44)
at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubFactory.create(PubSubOptions.java:39)
at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:469)
at services.PubSubService$.<init>(PubSubService.scala:14)
at services.PubSubService$.<clinit>(PubSubService.scala)
at Main$$anonfun$createContext$5$$anonfun$apply$1.apply(Main.scala:93)
at Main$$anonfun$createContext$5$$anonfun$apply$1.apply(Main.scala:92)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$27.apply(RDD.scala:894)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$27.apply(RDD.scala:894)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
The client initialization is as follow:
val pubsub: PubSub = PubSubOptions.getDefaultInstance().getService()
I look around and it's an issue between the netty version of Spark Google Dataproc netty-all-4.0.29.Final vs the one on google-cloud-pubsub (com.google.cloud) library in netty-all-4.1.3.Final. Did anyone manage to resolve this ?
Same issue as #1331 but with Google Dataproc (Spark).
Regards,
I'm trying to send messages from Spark Google Dataproc to Google PubSub. But I'm stuck with the following errors when trying to initialize the PubSub client:
The client initialization is as follow:
I look around and it's an issue between the netty version of Spark Google Dataproc netty-all-4.0.29.Final vs the one on google-cloud-pubsub (com.google.cloud) library in netty-all-4.1.3.Final. Did anyone manage to resolve this ?
Same issue as #1331 but with Google Dataproc (Spark).
Regards,