Skip to content

[ISSUE #7743] Isolate destination heartbeats and enforce per-destinat… - #7762

Merged
Aias00 merged 1 commit into
apache:masterfrom
harshkha3417:fix/7743-isolate-admin-heartbeats
Oct 22, 2026
Merged

Aias00 merged 1 commit into
apache:masterfrom
harshkha3417:fix/7743-isolate-admin-heartbeats

Conversation

@harshkha3417

@harshkha3417 harshkha3417 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #7743

Changes

  • Isolate destination heartbeat dispatching to a bounded worker executor (instance-beat-worker) using AbortPolicy so a slow or stalled Admin does not delay reports to healthy Admins or stall the periodic scheduler.
  • Enforce an end-to-end deadline per destination (heartbeatTimeoutMs, default 4000ms) covering login, network transfer, and response processing. deadlineMs <= 0 disables the deadline. Note: the deadline relies on OkHttp/Okio honoring Thread.interrupted(); custom BeatSender implementations must honor thread interrupts as well.
  • Defensively parse heartbeatTimeoutMs falling back to default 4000ms with a warning on invalid number formats.
  • Guard against overlapping dispatches via per-destination in-flight tracking (AtomicBoolean), with guaranteed flag reset on task completion, deadline cancellation, or executor rejection.
  • Maintain per-destination token ownership and isolated caching in Caffeine.
  • Ensure graceful shutdown of both scheduler and worker executor on context closure.
  • Add unit tests verifying isolation with slow Admins, token loading stalls, deadline cancellation, in-flight suppression, malformed configuration fallback, and queue/rejection flag recovery.

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the isolation direction is right (bounded pool, per-destination in-flight guard, per-destination deadline, injectable sender). But the in-flight bookkeeping can permanently disable a destination, and I would rather fix that here than in a follow-up.

1. The in-flight flag is only cleared inside the task body, so anything that stops the task from running blocks that admin forever

serverInFlight is reset in the finally of the submitted runnable (:178-186). Two paths never reach it:

  • Queued task cancelled at the deadline. poolSize is capped at MAX_BEAT_THREADS (8), so with more than eight destinations some submits sit in the queue. The deadline task (:187-195) calls future.cancel(true), and cancel on a FutureTask that has not started prevents it from ever running. Its finally never executes, the flag stays true, and every later beat to that admin is skipped with "still in flight" until the process restarts.
  • Discarded submit. DiscardPolicy (:137) silently drops the command, yet AbstractExecutorService.submit still returns a FutureTask, so the caller gets a future that never completes and never clears the flag.

Concretely: ten admins, two of them slow, and those two stop receiving heartbeats permanently — the admin then marks those gateways offline. That is the opposite of the isolation this PR is about.

Minimal fix: also clear the flag in the timeout branch (AtomicBoolean.set(false) is idempotent, so double-clearing is harmless), or submit via CompletableFuture.runAsync(...).whenComplete((r, e) -> flag.set(false)), which runs on cancellation as well. And please replace DiscardPolicy with CallerRunsPolicy, or handle rejection explicitly — "silently dropped heartbeat + stuck flag" is the worst possible combination.

2. A malformed heartbeatTimeoutMs fails application start

:124 calls Long.parseLong(timeoutProp) in the constructor, and HeartbeatListenerConfiguration:47 builds this bean during startup. shenyu.bootstrap.heartBeat.props.heartbeatTimeoutMs=4s — an easy typo — throws NumberFormatException out of the constructor and takes the gateway down. Please parse defensively and fall back to DEFAULT_DEADLINE_MS with a warn, the same way the neighbouring blank check does.

3. Smaller things

  • setBeatExecutor, setBeatSender, getBeatExecutor, getBeatSender are unused by main code and by the tests (which inject through the new constructor) — dead code, please drop them.
  • Please add a test for the cancellation/rejection path. Every current test uses a generously sized pool, which is precisely why (1) is invisible to them.
  • Worth stating in the PR body: deadlineMs <= 0 silently disables the deadline, and the 4 s deadline is only enforceable because RegisterUtils → OkHttpTools uses Okio timeouts that honour Thread.interrupted(). A custom BeatSender (the new extension point) must honour interrupts too, or the deadline becomes advisory.
  • The branch has no CI run (no checks reported), so none of the new concurrency tests have executed in CI yet.

The overall shape is good; it is the "who clears the flag" question that needs to be closed before this goes in.

@harshkha3417
harshkha3417 force-pushed the fix/7743-isolate-admin-heartbeats branch from a754a30 to 2067420 Compare October 18, 2026 02:48
@harshkha3417

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review and catch regarding the stuck in-flight flag @Aias00!

I've addressed all points and rebased onto the latest master:

  1. In-flight flag lifecycle:
    • Added finalInFlight.set(false) to the scheduled timeout branch so tasks cancelled while sitting in the queue immediately clear their in-flight guard.
    • Switched from DiscardPolicy to AbortPolicy and wrapped beatExecutor.submit(...) in a try/catch (RejectedExecutionException) that resets the in-flight flag and logs the rejection.
  2. Defensive deadline parsing: Added safe parsing for heartbeatTimeoutMs, falling back to DEFAULT_DEADLINE_MS (4000 ms) with a warning on NumberFormatException.
  3. Dead code cleanup: Removed setBeatExecutor, setBeatSender, getBeatExecutor, and getBeatSender.
  4. Tests & notes:
    • Added unit tests covering queued task cancellation flag recovery, task rejection flag recovery, and malformed timeout fallback.
    • Updated the PR description noting deadlineMs <= 0 semantics and BeatSender interrupt requirements.
    • All tests pass.

@harshkha3417
harshkha3417 requested a review from Aias00 October 18, 2026 04:11
Aias00
Aias00 previously approved these changes Oct 21, 2026

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the code. 2067420e addresses the three things I raised, and the third one is the interesting fix:

  1. DiscardPolicy → AbortPolicy + catch. This was the real defect behind "heartbeats stop forever": with DiscardPolicy the task was silently dropped after the in-flight flag had been set, so finalInFlight stayed true and every later tick skipped that Admin. Resetting the flag in the rejection handler (and continue-ing to the next server) makes a saturated queue a dropped beat rather than a dead destination. Catching RejectedExecutionException is only possible because you switched the policy — with DiscardPolicy nothing is thrown, which is exactly why it was invisible.
  2. Deadline cancellation resets the flag too. future.cancel(true) in the timeout task previously left the flag set until the worker's finally ran, and if the worker was stuck in a socket write that finally may be a long time coming. Resetting in the scheduled task as well closes that window; the double reset is harmless (AtomicBoolean.set(false) is idempotent).
  3. parseDeadlineMs. A malformed heartbeatTimeoutMs used to throw out of the constructor and take down the whole listener; warn-and-default is the right behaviour for an optional tuning knob.

Also good: onShutdown now shuts down beatExecutor as well (it only handled the scheduler before, so the worker pool leaked a non-daemon-ish thread per listener on context close), and the test-only setters/getters are gone in favour of reflection in the test.

Non-blocking observations:

  1. testMalformedHeartbeatTimeoutMsFallback feeds "4s" and asserts 4000L — which happens to equal DEFAULT_DEADLINE_MS, so at a glance the test reads as "4s is parsed to 4000 ms". It does prove the fallback (a NumberFormatException path is the only way to reach 4000 here), but "abc" or asserting the WARN would remove the ambiguity for the next reader.
  2. future.cancel(true) interrupts the worker, but RegisterUtils.doHeartBeat runs over OkHttp, which does not abort an in-flight socket write on interrupt. So a "timed out" beat can still complete on the wire while the next tick has already been allowed to start — two concurrent heartbeats to the same Admin. That is still better than the previous no-deadline behaviour, but the isolation is not absolute; worth a comment so nobody assumes cancel(true) is a guarantee.
  3. LOG.warn("Previous heartbeat to admin url :{} is still in flight, skipping.") fires on every tick while an Admin is slow. With a 3 s beat period and a hung Admin that is a lot of log volume; debug or a rate-limited warn would be kinder.
  4. deadlineMs > 0 disables the deadline entirely, and parseDeadlineMs accepts negative values, so heartbeatTimeoutMs=-1 is a valid "off switch". That is a nice escape hatch — please mention it in the property documentation rather than leaving it to be discovered.

Two merge blockers, neither of them in the change itself:

  • mergeStateStatus is DIRTY / mergeable: CONFLICTING. The branch conflicts with master (the heartbeat client was touched by other merged work). Please rebase.
  • No CI has run on 2067420e. gh pr checks reports no checks on this head and actions/runs?head_sha=2067420e is empty, so the force-push did not start a workflow. After the rebase the push should trigger the normal set; if it does not, ask a maintainer to approve the run.

@Aias00

Aias00 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Heads-up: this branch now has merge conflicts with master (mergeStateStatus: DIRTY), so CI cannot validate it against the current base.

The conflict is in shenyu-register-client-beat/.../HeartbeatListener.java and its test — that area has seen a lot of churn lately (instance db-sync isolation, heartbeat scheduler hardening, heartbeat recovery).

Could you rebase onto master and make sure your destination-scoped isolation still holds against the current heartbeat listener flow? I will re-review and merge once the build is green.

@harshkha3417

Copy link
Copy Markdown
Contributor Author

Thanks @Aias00 for the detailed review!

I have rebased the branch onto latest master to resolve the merge conflict with the boundPort refactoring, and addressed all non-blocking observations:

  1. Rebase & Constructor Alignment: Resolved the merge conflict with boundPort by removing ServerProperties across the listener and tests while maintaining the worker isolation and deadline tracking.
  2. Ambiguity Removal in Tests: Updated testMalformedHeartbeatTimeoutMsFallback to feed "invalid" instead of "4s".
  3. Cancellation Semantics: Added an explanatory comment on future.cancel(true) regarding OkHttp socket write interrupt semantics.
  4. Log Noise: Downgraded the overlapping in-flight skip message to LOG.debug to prevent log saturation during prolonged Admin stalls.
  5. Escape Hatch Javadoc: Documented that heartbeatTimeoutMs <= 0 (e.g. -1) disables the deadline entirely.

@harshkha3417
harshkha3417 requested a review from Aias00 October 22, 2026 01:17

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review: #7762 [ISSUE #7743] Isolate destination heartbeats and enforce per-destination deadlines

Verdict: approved — both merge blockers are resolved, and the follow-up commit adopted three of my four non-blocking notes, including the one about cancel(true) not being a real guarantee.

Merge blockers resolved

  • The branch no longer conflicts. mergeable_state went from DIRTY/CONFLICTING to BLOCKED, i.e. the rebase onto current master (including the heartbeat-client work that had landed) is clean.
  • CI has now run on the rebased head. 22 checks pass, 9 are path-filtered skips, nothing failing. The previous head had no workflow run at all after the force-push, which is why I would not approve it then.

Non-blocking notes adopted

  • cancel(true) is now documented as best-effort right where it matters:
    // Note: RegisterUtils.doHeartBeat relies on OkHttp. Thread interruption does not immediately
    // abort active socket writes on the wire, so cancel(true) is best-effort cancellation.
    That is the note I most wanted recorded. Without it, the next person reads future.cancel(true) and assumes the in-flight heartbeat is dead, when a "timed out" beat can still complete on the wire while the next tick starts — two concurrent beats to one Admin is now a bounded, understood state rather than a hidden one.
  • The "still in flight" message moved from warn to debug. With a 5 s tick and a hung Admin, the previous level produced a warning per second per destination; debug keeps it available for diagnosis without flooding the log of a gateway that is otherwise working as designed.
  • The <= 0 off-switch is documented rather than discoverable. parseDeadlineMs has a Javadoc stating "Values <= 0 (e.g. -1) disable the per-destination deadline entirely", so heartbeatTimeoutMs=-1 is a supported escape hatch instead of a curiosity.

The remaining note (the "4s" → 4000L assertion coinciding with DEFAULT_DEADLINE_MS) still stands as a readability nit; the "abc" case would make the fallback unambiguous.

What I re-verified in the current diff

  • AbortPolicy plus the rejection handler is still the core fix, and still the right one: with the previous DiscardPolicy the task was dropped after inFlight had been set, so the destination was dead forever with no signal. Now RejectedExecutionException is caught, the flag is reset, and the loop continues to the next server — a saturated queue becomes a dropped beat, not a dead Admin.
  • The flag is reset on all three paths: the worker's finally, the rejection handler, and the timeout task. The double reset is harmless because AtomicBoolean.set(false) is idempotent, and covering the timeout path matters because a worker stuck in a socket write may not reach its finally for a long time.
  • inFlight is keyed per destination (Map<String, AtomicBoolean> via computeIfAbsent), so a slow Admin no longer blocks heartbeats to a healthy one. That is the "isolate destination heartbeats" half of the issue and it is done with the minimum state.
  • poolSize = min(max(1, serverList.size()), 8) with a queue of serverList.size() bounds both concurrency and backlog by the actual destination count, so a large Admin list cannot spawn an unbounded pool. The MAX_BEAT_THREADS cap is the right guard.
  • BeatSender is a genuine improvement, not just test plumbing. Extracting login/doHeartBeat behind an interface and defaulting to DefaultBeatSender removes the need for MockedStatic on RegisterUtils, which is what the old test did. A mockable seam on a network call is exactly where one belongs.
  • onShutdown now tears down beatExecutor as well, with null guards on both executors and shutdownNow() only after the grace period. The previous version only shut the scheduler, so the worker pool leaked a thread per listener on context close.
  • HeartbeatBoundPortTest sets heartbeatDirectExecution=true, so the port-binding test does not start a worker pool it never uses — that is the right use of the new escape hatch and shows it works as intended.

Non-blocking

  • heartbeatDirectExecution reverts to inline execution on the scheduler thread, which also disables the per-destination inFlight isolation by setting serverInFlight = null. That is a deliberate coupling (an inline beat cannot overlap itself), but it means the property silently changes two things at once. Worth a sentence in the property documentation alongside the heartbeatTimeoutMs note.
  • inFlight entries are never removed, including in onShutdown. Bounded by the destination count so it is not a leak, but inFlight.clear() in onShutdown would make the intent explicit.

CI is fully green on this head (22 pass, 0 fail), so this is mergeable as soon as a second reviewer agrees.

@Aias00
Aias00 merged commit 862f5e3 into apache:master Oct 22, 2026
31 checks passed
@Aias00 Aias00 added this to the 2.7.2 milestone Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] A slow Admin delays independent heartbeats to all subsequent Admins

3 participants