[Task] Harden periodic tasks so a single exception no longer cancels the schedule #6608

Background

A 2026-08-02 source audit confirmed that 8 open issues share the same root cause: a periodic task scheduled via ScheduledThreadPoolExecutor (either scheduleAtFixedRate or scheduleWithFixedDelay) is permanently cancelled after a single uncaught exception, because the JDK contract states "if any execution of the task encounters an exception, subsequent executions are suppressed." Confirmed that org.apache.shenyu.common.exception.ShenyuException extends RuntimeException, so rethrows escape the task and kill the schedule.

This one defect class silently stops cluster election, instance health sync, registry watches, token refresh, heartbeat, and the upstream health-check loop — a broad class of "the gateway/admin silently stops reconciling" symptoms.

Affected issues (all verified VALID)

• [BUG] Client heartbeat scheduler can stop permanently after one full failure #6487 Client heartbeat scheduler can stop permanently after one full failure
• [BUG] Cluster master election scheduler can stop after one renewal exception #6495 Cluster master election scheduler can stop after one renewal exception
• [BUG] Instance health DB sync scheduler can stop after one persistence error #6496 Instance health DB sync scheduler can stop after one persistence error
• [BUG] Consul TTL heartbeat task can stop after one transient Consul error #6498 Consul TTL heartbeat task can stop after one transient Consul error
• [BUG] Consul instance watch task can stop after one polling error #6499 Consul instance watch task can stop after one polling error
• [BUG] Eureka instance watch task can stop after one polling error #6500 Eureka instance watch task can stop after one polling error
• [BUG] Upstream health check can get stuck after one async check failure #6502 Upstream health check can get stuck after one async check failure
• [BUG] HTTP sync token refresh can stop after a login response parsing error #6504 HTTP sync token refresh can stop after a login response parsing error

Variants observed

• No try/catch at all (task body throws directly): [BUG] Client heartbeat scheduler can stop permanently after one full failure #6487, [BUG] Consul TTL heartbeat task can stop after one transient Consul error #6498, [BUG] Consul instance watch task can stop after one polling error #6499.
• Catch-and-rethrow as ShenyuException/RuntimeException (still escapes and cancels): [BUG] Cluster master election scheduler can stop after one renewal exception #6495, [BUG] Eureka instance watch task can stop after one polling error #6500.
• Sibling task is guarded but this one isn't: [BUG] Instance health DB sync scheduler can stop after one persistence error #6496 (scheduled() is wrapped, syncDB() is not).
• Aggregation/list bookkeeping fails before clear(): [BUG] Upstream health check can get stuck after one async check failure #6502 (CompletableFuture.allOf(...).join() re-throws before futures.clear()).

Suggested fix pattern (single mechanism closes all 8)

Introduce a common resilient wrapper for periodic tasks in shenyu-common (or a small helper in each module) that:

1. Wraps the task body in try { ... } catch (Throwable t) { LOG.error(...); } so an exception never escapes and cancels the schedule.
2. For tasks that must surface fatal errors, swallow-and-log for transient errors and escalate (metric/alert) rather than killing the schedule.
3. Ensures any shared mutable state (e.g. the futures list in [BUG] Upstream health check can get stuck after one async check failure #6502) is cleared in a finally, independent of whether join() succeeded.

Apply it uniformly to ShenyuClusterService, InstanceCheckService, TtlScheduler, ConsulInstanceRegisterRepository, EurekaInstanceRegisterRepository, ShenyuClientURIExecutorSubscriber, AccessTokenManager, and UpstreamCheckService.

Goal

Track the shared fix so the 8 issues can be closed by one well-scoped PR (or a small series) rather than eight independent patches that risk re-implementing the same guard inconsistently.

Audit ref: docs/issue-candidates-2026-08-02.md