Skip to content

[BUG] When Admin is in cluster mode, the restart causes the gateway node to keep reconnecting by websocket #6461

New issue
New issue
Closed
#7661
Closed
[BUG] When Admin is in cluster mode, the restart causes the gateway node to keep reconnecting by websocket#6461
#7661
Labels
type: bugSomething isn't working

Description

@next-evolve-x
next-evolve-x
opened on Jul 29, 2026

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

  1. admin和网关都是k8s部署
  2. admin配置集群模式
  3. 网关使用k8s本地域名进行ws连接
  4. 使用websocket同步数据
  5. 重启admin

网关日志：2026-07-29 14:29:31.078 INFO [] ,,,WebSocketConnectReadThread-121,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending sync msg
2026-07-29 14:29:31.078 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is successful.....
2026-07-29 14:30:31.079 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,shenyu bootstrap websocket namespaceId: 403dfe7c-e94c-4582-8e71-994e8356f961
2026-07-29 14:30:31.082 INFO [] ,,,WebSocketConnectReadThread-124,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending sync msg
2026-07-29 14:30:31.082 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is successful.....
2026-07-29 14:31:31.082 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,shenyu bootstrap websocket namespaceId: 403dfe7c-e94c-4582-8e71-994e8356f961
2026-07-29 14:31:31.085 INFO [] ,,,WebSocketConnectReadThread-127,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending sync msg
2026-07-29 14:31:31.085 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is successful.....
2026-07-29 14:32:31.085 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,shenyu bootstrap websocket namespaceId: 403dfe7c-e94c-4582-8e71-994e8356f961
2026-07-29 14:32:31.089 INFO [] ,,,WebSocketConnectReadThread-130,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending sync msg
2026-07-29 14:32:31.089 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is successful.....
2026-07-29 14:33:31.089 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,shenyu bootstrap websocket namespaceId: 403dfe7c-e94c-4582-8e71-994e8356f961
2026-07-29 14:33:31.092 INFO [] ,,,WebSocketConnectReadThread-133,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending sync msg
2026-07-29 14:33:31.093 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is successful.....
2026-07-29 14:34:31.094 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,shenyu bootstrap websocket namespaceId: 403dfe7c-e94c-4582-8e71-994e8356f961
2026-07-29 14:34:31.097 INFO [] ,,,WebSocketConnectReadThread-136,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending syn

Admin日志

2026-07-29 15:08:31.249 ERROR [] ,,,http-nio-0.0.0.0-80-exec-10,o.a.s.a.l.w.WebsocketCollector,websocket send result is exception: java.io.IOException: java.io.IOException: Broken pipe
1785308911249 2026-07-29T07:08:31.249Z 2026-07-29 15:08:31.249 WARN [] ,,,http-nio-0.0.0.0-80-exec-10,o.a.s.a.l.w.WebsocketCollector,websocket close on client[10.10.16.173]
1785308911249 2026-07-29T07:08:31.249Z
1785308911249 2026-07-29T07:08:31.249Z ... 37 common frames omitted
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlockInternal(WsRemoteEndpointImplBase.java:313)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:521)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:171)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:1337)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:1366)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1440)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.SocketWrapperBase$OperationState.start(SocketWrapperBase.java:1051)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1632)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:134)
1785308911249 2026-07-29T07:08:31.249Z at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at java.base/sun.nio.ch.SocketDispatcher.writev(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at java.base/sun.nio.ch.SocketDispatcher.writev0(Native Method)
1785308911249 2026-07-29T07:08:31.249Z Caused by: java.io.IOException: Broken pipe
1785308911249 2026-07-29T07:08:31.249Z at java.base/java.lang.Thread.run(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:57)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:60)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:152)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsFrameServer.notifyDataAvailable(WsFrameServer.java:164)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsFrameServer.doOnDataAvailable(WsFrameServer.java:184)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:85)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsFrameBase.processInputBuffer(WsFrameBase.java:130)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsFrameBase.processData(WsFrameBase.java:284)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsFrameBase.processDataText(WsFrameBase.java:484)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsFrameServer.sendMessageText(WsFrameServer.java:130)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsFrameBase.sendMessageText(WsFrameBase.java:390)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.pojo.PojoMessageHandlerWholeBase.onMessage(PojoMessageHandlerWholeBase.java:102)
1785308911249 2026-07-29T07:08:31.249Z at java.base/java.lang.reflect.Method.invoke(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.websocket.WebsocketCollector.onMessage(WebsocketCollector.java:203)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.service.impl.SyncDataServiceImpl.syncAllByNamespaceId(SyncDataServiceImpl.java:153)
1785308911249 2026-07-29T07:08:31.249Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384)
1785308911249 2026-07-29T07:08:31.249Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451)
1785308911249 2026-07-29T07:08:31.249Z at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
1785308911249 2026-07-29T07:08:31.249Z at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
1785308911249 2026-07-29T07:08:31.249Z at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.DataChangedEventDispatcher.onApplicationEvent(DataChangedEventDispatcher.java:52)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.DataChangedEventDispatcher.onApplicationEvent(DataChangedEventDispatcher.java:99)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.websocket.WebsocketDataChangedListener.onSelectorChanged(WebsocketDataChangedListener.java:84)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.websocket.WebsocketCollector.send(WebsocketCollector.java:279)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.websocket.WebsocketCollector.sendMessageBySession(WebsocketCollector.java:293)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendText(WsRemoteEndpointBasic.java:36)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendString(WsRemoteEndpointImplBase.java:192)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:250)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:266)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlockInternal(WsRemoteEndpointImplBase.java:324)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:589)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSession.java:789)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:266)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlockInternal(WsRemoteEndpointImplBase.java:326)
1785308911249 2026-07-29T07:08:31.249Z 2026-07-29 15:08:31.249 ERROR [] ,,,http-nio-0.0.0.0-80-exec-10,o.a.s.a.l.w.WebsocketCollector,websocket collection on client[10.10.16.173] error: java.io.IOException: java.io.IOException: Broken pipe

每次Admin准备写回数据，ws连接就关闭了

Expected Behavior

正常连接到master

Steps To Reproduce

  1. admin和网关都是k8s部署
  2. admin配置集群模式
  3. 网关使用k8s本地域名进行ws连接
  4. 使用websocket同步数据
  5. 重启admin

Environment

ShenYu version(s): 2.7.0.3

Debug logs

No response

Anything else?

No response

Activity

  1. next-evolve-x
    added
    type: bugSomething isn't working
    on Jul 29, 2026

  2. github-actions commented on Jul 29, 2026

    @github-actions
    github-actionsBot
    on Jul 29, 2026 – with GitHub Actions
    Contributor

    👋 Hello @Aias00

    A new issue has been opened by @next-evolve-x:

    Please review when you have time. Thank you! 🙏

  3. next-evolve-x commented on Jul 29, 2026

    @next-evolve-x
    next-evolve-x
    on Jul 29, 2026
    Author

    ` private void masterCheck() {
    if (LOG.isDebugEnabled()) {
    LOG.debug("master checking task start...");
    }
    if (CollectionUtils.isEmpty(clients)) {
    List urls = websocketConfig.getUrls();
    for (String url : urls) {
    if (StringUtils.isNotEmpty(websocketConfig.getAllowOrigin())) {
    Map<String, String> headers =
    ImmutableMap.of(ORIGIN_HEADER_NAME, websocketConfig.getAllowOrigin());
    clients.add(
    new ShenyuWebsocketClient(
    URI.create(url),
    headers,
    Objects.requireNonNull(pluginDataSubscriber),
    metaDataSubscribers,
    authDataSubscribers,
    proxySelectorDataSubscribers,
    discoveryUpstreamDataSubscribers,
    this.aiProxyApiKeyDataSubscribers,
    namespaceId, serverProperties.getPort()));
    } else {
    clients.add(
    new ShenyuWebsocketClient(
    URI.create(url),
    Objects.requireNonNull(pluginDataSubscriber),
    metaDataSubscribers,
    authDataSubscribers,
    proxySelectorDataSubscribers,
    discoveryUpstreamDataSubscribers,
    this.aiProxyApiKeyDataSubscribers,
    namespaceId, serverProperties.getPort()));
    }
    }
    }
    Iterator iterator = clients.iterator();
    while (iterator.hasNext()) {
    ShenyuWebsocketClient websocketClient = iterator.next();
    if (!websocketClient.isOpen()) {
    iterator.remove();
    continue;
    }
    String runningMode = websocketClient.getRunningMode();
    // check running mode
    if (Objects.equals(runningMode, RunningModeEnum.STANDALONE.name())) {
    LOG.info("admin running in standalone mode...");
    timerTask.cancel();
    return;
    }

            if (!websocketClient.isConnectedToMaster()) {
            websocketClient.nowClose();
            iterator.remove();
        }
    }
}`

    应该是这里面的runningMode 可能检查的时候，Admin还没有返回导致runningMode是null,且isConnectedToMaster是false，导致关闭掉
    等到下一次定时任务，不断重复这个逻辑

  4. sunnysabor commented on Oct 1, 2026

    @sunnysabor
    sunnysabor
    on Oct 1, 2026
    Contributor

    I traced the current upstream/master WebSocket path at 09c6a528. The client records the masterUrl and isMaster values from the RUNNING_MODE reply in ShenyuWebsocketClient#onMessage. However, WebsocketSyncDataService#masterCheck recreates clients from the original websocketConfig.urls and closes/removes cluster-mode clients that are not connected to the master. The stored masterUrl is not used to redirect/reconnect.

    This suggests a loop if the Kubernetes Service URL repeatedly routes a gateway to a non-master admin pod: the connection is dropped and the same load-balanced URL is retried. This is distinct from the runningMode == null timing hypothesis in the earlier comment. Could you confirm whether the Service can repeatedly select a non-master pod in your deployment, and whether the intended behavior is for the gateway to reconnect directly to the returned masterUrl? That will clarify the right fix and regression test.

  5. Aias00
    added a commit that references this issue on Oct 19, 2026

    [ISSUE #6461] Follow advertised WebSocket master addresses (#7661)

    8b3b0d3
  6. Aias00
    closed this as completedin #7661on Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions