Bug in netty 4.1.43 prevents from using Linux native Epoll transport #6330
Description
Describe the bug
netty 4.1.43 has a bug preventing it from using Linux native Epoll transport - netty/netty#9758
This results in pulsar brokers failing over to NioEventLoopGroup even when running on Linux.
The bug seems to be fixed in netty releases 4.1.44+
To Reproduce
- Run pulsar docker container:
docker run -it apachepulsar/pulsar:latest bin/pulsar standalone
- Observe the logs. Following errors appears on startup
23:56:05.428 [main] WARN org.apache.distributedlog.impl.BKNamespaceDriver - Could not use Netty Epoll event loop for bookie server:
java.lang.NoClassDefFoundError: Could not initialize class io.netty.channel.epoll.EpollEventLoop
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:142) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68) ~[io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at org.apache.distributedlog.impl.BKNamespaceDriver.getDefaultEventLoopGroup(BKNamespaceDriver.java:257) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
at org.apache.distributedlog.impl.BKNamespaceDriver.initializeBookKeeperClients(BKNamespaceDriver.java:268) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
at org.apache.distributedlog.impl.BKNamespaceDriver.initialize(BKNamespaceDriver.java:206) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
at org.apache.distributedlog.api.namespace.NamespaceBuilder.build(NamespaceBuilder.java:239) ~[org.apache.distributedlog-distributedlog-core-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.stream.server.service.DLNamespaceProviderService.doStart(DLNamespaceProviderService.java:120) ~[org.apache.bookkeeper-stream-storage-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) ~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$2(LifecycleComponentStack.java:113) ~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:407) [com.google.guava-guava-25.1-jre.jar:?]
at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:113) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent.doStart(StreamStorageLifecycleComponent.java:51) [org.apache.bookkeeper-stream-storage-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runStreamStorage(LocalBookkeeperEnsemble.java:323) [org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar:2.5.0]
at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone(LocalBookkeeperEnsemble.java:405) [org.apache.pulsar-pulsar-zookeeper-utils-2.5.0.jar:2.5.0]
at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:260) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:119) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
With DEBUG logging enabled following error appears as well:
22:31:13.747 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_transport_native_epoll_x86_64', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860) ~[?:1.8.0_232]
at java.lang.Runtime.loadLibrary0(Runtime.java:870) ~[?:1.8.0_232]
at java.lang.System.loadLibrary(System.java:1122) ~[?:1.8.0_232]
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_232]
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361) ~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:228) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.Native.<clinit>(Native.java:57) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:52) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:142) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68) [io.netty-netty-transport-native-epoll-4.1.43.Final.jar:4.1.43.Final]
at org.apache.bookkeeper.util.EventLoopUtil.getEventLoopGroup(EventLoopUtil.java:59) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.util.EventLoopUtil.getClientEventLoopGroup(EventLoopUtil.java:43) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:458) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:296) [org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:70) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.<init>(ManagedLedgerClientFactory.java:61) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:394) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:264) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:329) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0]
Expected behavior
Netty EpollEventLoopGroup is initialized correctly.