Repository navigation
[ISSUE #7743] Isolate destination heartbeats and enforce per-destinat… - #7762
Conversation
Aias00
left a comment
There was a problem hiding this comment.
Thanks — the isolation direction is right (bounded pool, per-destination in-flight guard, per-destination deadline, injectable sender). But the in-flight bookkeeping can permanently disable a destination, and I would rather fix that here than in a follow-up.
1. The in-flight flag is only cleared inside the task body, so anything that stops the task from running blocks that admin forever
serverInFlight is reset in the finally of the submitted runnable (:178-186). Two paths never reach it:
- Queued task cancelled at the deadline.
poolSizeis capped atMAX_BEAT_THREADS(8), so with more than eight destinations some submits sit in the queue. The deadline task (:187-195) callsfuture.cancel(true), andcancelon aFutureTaskthat has not started prevents it from ever running. Itsfinallynever executes, the flag staystrue, and every later beat to that admin is skipped with "still in flight" until the process restarts. - Discarded submit.
DiscardPolicy(:137) silently drops the command, yetAbstractExecutorService.submitstill returns aFutureTask, so the caller gets a future that never completes and never clears the flag.
Concretely: ten admins, two of them slow, and those two stop receiving heartbeats permanently — the admin then marks those gateways offline. That is the opposite of the isolation this PR is about.
Minimal fix: also clear the flag in the timeout branch (AtomicBoolean.set(false) is idempotent, so double-clearing is harmless), or submit via CompletableFuture.runAsync(...).whenComplete((r, e) -> flag.set(false)), which runs on cancellation as well. And please replace DiscardPolicy with CallerRunsPolicy, or handle rejection explicitly — "silently dropped heartbeat + stuck flag" is the worst possible combination.
2. A malformed heartbeatTimeoutMs fails application start
:124 calls Long.parseLong(timeoutProp) in the constructor, and HeartbeatListenerConfiguration:47 builds this bean during startup. shenyu.bootstrap.heartBeat.props.heartbeatTimeoutMs=4s — an easy typo — throws NumberFormatException out of the constructor and takes the gateway down. Please parse defensively and fall back to DEFAULT_DEADLINE_MS with a warn, the same way the neighbouring blank check does.
3. Smaller things
setBeatExecutor,setBeatSender,getBeatExecutor,getBeatSenderare unused by main code and by the tests (which inject through the new constructor) — dead code, please drop them.- Please add a test for the cancellation/rejection path. Every current test uses a generously sized pool, which is precisely why (1) is invisible to them.
- Worth stating in the PR body:
deadlineMs <= 0silently disables the deadline, and the 4 s deadline is only enforceable becauseRegisterUtils→OkHttpToolsuses Okio timeouts that honourThread.interrupted(). A customBeatSender(the new extension point) must honour interrupts too, or the deadline becomes advisory. - The branch has no CI run (
no checks reported), so none of the new concurrency tests have executed in CI yet.
The overall shape is good; it is the "who clears the flag" question that needs to be closed before this goes in.
a754a30 to
2067420
Compare
|
Thanks for the thorough review and catch regarding the stuck in-flight flag @Aias00! I've addressed all points and rebased onto the latest
|
Aias00
left a comment
There was a problem hiding this comment.
Approving the code. 2067420e addresses the three things I raised, and the third one is the interesting fix:
DiscardPolicy→AbortPolicy+ catch. This was the real defect behind "heartbeats stop forever": withDiscardPolicythe task was silently dropped after the in-flight flag had been set, sofinalInFlightstayedtrueand every later tick skipped that Admin. Resetting the flag in the rejection handler (andcontinue-ing to the next server) makes a saturated queue a dropped beat rather than a dead destination. CatchingRejectedExecutionExceptionis only possible because you switched the policy — withDiscardPolicynothing is thrown, which is exactly why it was invisible.- Deadline cancellation resets the flag too.
future.cancel(true)in the timeout task previously left the flag set until the worker'sfinallyran, and if the worker was stuck in a socket write thatfinallymay be a long time coming. Resetting in the scheduled task as well closes that window; the double reset is harmless (AtomicBoolean.set(false)is idempotent). parseDeadlineMs. A malformedheartbeatTimeoutMsused to throw out of the constructor and take down the whole listener; warn-and-default is the right behaviour for an optional tuning knob.
Also good: onShutdown now shuts down beatExecutor as well (it only handled the scheduler before, so the worker pool leaked a non-daemon-ish thread per listener on context close), and the test-only setters/getters are gone in favour of reflection in the test.
Non-blocking observations:
testMalformedHeartbeatTimeoutMsFallbackfeeds"4s"and asserts4000L— which happens to equalDEFAULT_DEADLINE_MS, so at a glance the test reads as "4sis parsed to 4000 ms". It does prove the fallback (aNumberFormatExceptionpath is the only way to reach 4000 here), but"abc"or asserting theWARNwould remove the ambiguity for the next reader.future.cancel(true)interrupts the worker, butRegisterUtils.doHeartBeatruns over OkHttp, which does not abort an in-flight socket write on interrupt. So a "timed out" beat can still complete on the wire while the next tick has already been allowed to start — two concurrent heartbeats to the same Admin. That is still better than the previous no-deadline behaviour, but the isolation is not absolute; worth a comment so nobody assumescancel(true)is a guarantee.LOG.warn("Previous heartbeat to admin url :{} is still in flight, skipping.")fires on every tick while an Admin is slow. With a 3 s beat period and a hung Admin that is a lot of log volume;debugor a rate-limited warn would be kinder.deadlineMs > 0disables the deadline entirely, andparseDeadlineMsaccepts negative values, soheartbeatTimeoutMs=-1is a valid "off switch". That is a nice escape hatch — please mention it in the property documentation rather than leaving it to be discovered.
Two merge blockers, neither of them in the change itself:
mergeStateStatusisDIRTY/mergeable: CONFLICTING. The branch conflicts withmaster(the heartbeat client was touched by other merged work). Please rebase.- No CI has run on
2067420e.gh pr checksreports no checks on this head andactions/runs?head_sha=2067420eis empty, so the force-push did not start a workflow. After the rebase the push should trigger the normal set; if it does not, ask a maintainer to approve the run.
|
Heads-up: this branch now has merge conflicts with The conflict is in Could you rebase onto |
… and resolve rebase conflicts
2067420 to
895431e
Compare
|
Thanks @Aias00 for the detailed review! I have rebased the branch onto latest
|
Aias00
left a comment
There was a problem hiding this comment.
Re-review: #7762 [ISSUE #7743] Isolate destination heartbeats and enforce per-destination deadlines
Verdict: approved — both merge blockers are resolved, and the follow-up commit adopted three of my four non-blocking notes, including the one about cancel(true) not being a real guarantee.
Merge blockers resolved
- The branch no longer conflicts.
mergeable_statewent fromDIRTY/CONFLICTINGtoBLOCKED, i.e. the rebase onto currentmaster(including the heartbeat-client work that had landed) is clean. - CI has now run on the rebased head. 22 checks pass, 9 are path-filtered skips, nothing failing. The previous head had no workflow run at all after the force-push, which is why I would not approve it then.
Non-blocking notes adopted
cancel(true)is now documented as best-effort right where it matters:That is the note I most wanted recorded. Without it, the next person reads// Note: RegisterUtils.doHeartBeat relies on OkHttp. Thread interruption does not immediately // abort active socket writes on the wire, so cancel(true) is best-effort cancellation.
future.cancel(true)and assumes the in-flight heartbeat is dead, when a "timed out" beat can still complete on the wire while the next tick starts — two concurrent beats to one Admin is now a bounded, understood state rather than a hidden one.- The "still in flight" message moved from
warntodebug. With a 5 s tick and a hung Admin, the previous level produced a warning per second per destination;debugkeeps it available for diagnosis without flooding the log of a gateway that is otherwise working as designed. - The
<= 0off-switch is documented rather than discoverable.parseDeadlineMshas a Javadoc stating "Values <= 0 (e.g. -1) disable the per-destination deadline entirely", soheartbeatTimeoutMs=-1is a supported escape hatch instead of a curiosity.
The remaining note (the "4s" → 4000L assertion coinciding with DEFAULT_DEADLINE_MS) still stands as a readability nit; the "abc" case would make the fallback unambiguous.
What I re-verified in the current diff
AbortPolicyplus the rejection handler is still the core fix, and still the right one: with the previousDiscardPolicythe task was dropped afterinFlighthad been set, so the destination was dead forever with no signal. NowRejectedExecutionExceptionis caught, the flag is reset, and the loopcontinues to the next server — a saturated queue becomes a dropped beat, not a dead Admin.- The flag is reset on all three paths: the worker's
finally, the rejection handler, and the timeout task. The double reset is harmless becauseAtomicBoolean.set(false)is idempotent, and covering the timeout path matters because a worker stuck in a socket write may not reach itsfinallyfor a long time. inFlightis keyed per destination (Map<String, AtomicBoolean>viacomputeIfAbsent), so a slow Admin no longer blocks heartbeats to a healthy one. That is the "isolate destination heartbeats" half of the issue and it is done with the minimum state.poolSize = min(max(1, serverList.size()), 8)with a queue ofserverList.size()bounds both concurrency and backlog by the actual destination count, so a large Admin list cannot spawn an unbounded pool. TheMAX_BEAT_THREADScap is the right guard.BeatSenderis a genuine improvement, not just test plumbing. Extractinglogin/doHeartBeatbehind an interface and defaulting toDefaultBeatSenderremoves the need forMockedStaticonRegisterUtils, which is what the old test did. A mockable seam on a network call is exactly where one belongs.onShutdownnow tears downbeatExecutoras well, with null guards on both executors andshutdownNow()only after the grace period. The previous version only shut the scheduler, so the worker pool leaked a thread per listener on context close.HeartbeatBoundPortTestsetsheartbeatDirectExecution=true, so the port-binding test does not start a worker pool it never uses — that is the right use of the new escape hatch and shows it works as intended.
Non-blocking
heartbeatDirectExecutionreverts to inline execution on the scheduler thread, which also disables the per-destinationinFlightisolation by settingserverInFlight = null. That is a deliberate coupling (an inline beat cannot overlap itself), but it means the property silently changes two things at once. Worth a sentence in the property documentation alongside theheartbeatTimeoutMsnote.inFlightentries are never removed, including inonShutdown. Bounded by the destination count so it is not a leak, butinFlight.clear()inonShutdownwould make the intent explicit.
CI is fully green on this head (22 pass, 0 fail), so this is mergeable as soon as a second reviewer agrees.
Fixes #7743
Changes
instance-beat-worker) usingAbortPolicyso a slow or stalled Admin does not delay reports to healthy Admins or stall the periodic scheduler.heartbeatTimeoutMs, default 4000ms) covering login, network transfer, and response processing.deadlineMs <= 0disables the deadline. Note: the deadline relies on OkHttp/Okio honoringThread.interrupted(); customBeatSenderimplementations must honor thread interrupts as well.heartbeatTimeoutMsfalling back to default 4000ms with a warning on invalid number formats.AtomicBoolean), with guaranteed flag reset on task completion, deadline cancellation, or executor rejection.