Skip to content

[BUG] Cluster master election scheduler can stop after one renewal exception #6495

Closed

Description

@Aias00

Current Behavior

Cluster master election/renewal can stop permanently after one exception escapes the scheduled task.

ShenyuClusterService.startSelectMasterTask() schedules master selection with scheduleAtFixedRate(...):

executorService.scheduleAtFixedRate(() -> doSelectMaster(host, port, contextPath), ...);

doSelectMaster() catches exceptions, closes the check services, and then throws a new ShenyuException:

} catch (Exception e) {
    LOG.error("select master error", e);
    upstreamCheckService.close();
    instanceCheckService.close();
    throw new ShenyuException(...);
}

For ScheduledThreadPoolExecutor, an exception escaping a fixed-rate task suppresses later executions of that periodic task. A transient lock/renewal failure can therefore stop future master selection attempts in that process.

Expected Behavior

Master election/renewal errors should be logged and the next scheduled election attempt should still run. Exceptions should not escape the periodic scheduler task.

Impact

After a transient cluster backend failure, the admin node may stop participating in master election until it is restarted.

Code Location

  • shenyu-admin/src/main/java/org/apache/shenyu/admin/mode/cluster/service/ShenyuClusterService.java
    • startSelectMasterTask() schedules the periodic task.
    • doSelectMaster() rethrows from the catch block.

Activity

  1. github-actions commented on Jul 31, 2026

    @github-actions
    Contributor

    👋 Hello @Aias00

    A new issue has been opened by @Aias00:

    Please review when you have time. Thank you! 🙏

  2. sunnysabor commented on Oct 1, 2026

    @sunnysabor
    Contributor

    Confirmed this still exists on current upstream/master: startSelectMasterTask() schedules doSelectMaster directly, and its exception path rethrows ShenyuException, which cancels later fixed-rate executions. I plan a focused scheduler-boundary guard for the expected ShenyuException path (leaving fatal Errors visible), preserving the existing service-close and lock-release behavior, with a regression test proving a later scheduled election still runs after one failed attempt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions