Repository navigation
Fix event loop deadlock in MultiTargetRoutingHandler - #2718
Merged
AndreKurait merged 1 commit intoApr 15, 2026
Merged
Conversation
AndreKurait
requested review from
gregschohn,
jugal-chauhan and
sumobrian
as code owners
April 14, 2026 18:01
AndreKurait
had a problem deploying
to
migrations-cicd
April 14, 2026 18:01 — with
GitHub Actions
Error
AndreKurait
temporarily deployed
to
migrations-cicd
April 14, 2026 18:01 — with
GitHub Actions
Inactive
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2718 +/- ##
============================================
- Coverage 73.43% 73.43% -0.01%
Complexity 106 106
============================================
Files 721 721
Lines 33368 33372 +4
Branches 2910 2907 -3
============================================
+ Hits 24504 24506 +2
- Misses 7533 7535 +2
Partials 1331 1331
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
AndreKurait
force-pushed
the
fix-shim-proxy-test-flaky
branch
from
April 14, 2026 18:26
5220a65 to
895aa7a
Compare
AndreKurait
had a problem deploying
to
migrations-cicd
April 14, 2026 18:26 — with
GitHub Actions
Error
AndreKurait
had a problem deploying
to
migrations-cicd
April 14, 2026 18:26 — with
GitHub Actions
Failure
AndreKurait
force-pushed
the
fix-shim-proxy-test-flaky
branch
from
April 14, 2026 18:52
895aa7a to
a47e58e
Compare
AndreKurait
temporarily deployed
to
migrations-cicd
April 14, 2026 18:52 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
April 14, 2026 18:52 — with
GitHub Actions
Inactive
Move collectResponses() (which blocks on secondary future.join()) off the Netty event loop thread via CompletableFuture.supplyAsync(). The secondary FixedChannelPools share the same NioEventLoopGroup, so calling join() on the event loop can deadlock when a pooled channel's response handler needs that same thread to fire channelRead0. The fix preserves the original behavior: response triggers as soon as the primary completes, secondaries are collected with their existing timeout on a ForkJoinPool worker thread, then the response is written back on the event loop thread via thenAcceptAsync. Signed-off-by: Andre Kurait <[email protected]>
AndreKurait
force-pushed
the
fix-shim-proxy-test-flaky
branch
from
April 14, 2026 19:48
a47e58e to
82d85de
Compare
AndreKurait
temporarily deployed
to
migrations-cicd
April 14, 2026 19:49 — with
GitHub Actions
Inactive
AndreKurait
temporarily deployed
to
migrations-cicd
April 14, 2026 19:49 — with
GitHub Actions
Inactive
AndreKurait
enabled auto-merge
April 14, 2026 19:49
jugal-chauhan
approved these changes
Apr 15, 2026
jugal-chauhan
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for brining in these changes
AndreKurait
added a commit
to AndreKurait/opensearch-migrations
that referenced
this pull request
Apr 15, 2026
…proxy-test-flaky Fix flaky ShimProxyTest.dualTarget_validationPass()
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes flaky
ShimProxyTestdual-target tests (dualTarget_validationPass,dualTarget_cursorMarkMerge) which intermittently fail withTimeoutExceptionon the secondary target.Root Cause
Event loop deadlock in
MultiTargetRoutingHandler.handlePrimaryCompletion()attached awhenCompletecallback to the primary future, then ran on the inbound channel's event loop thread viactx.channel().eventLoop().execute(...). Inside that callback,collectSingleResponse()calledfuture.join()— a blocking call on the event loop thread.The backend connection pool (
FixedChannelPool) shares the sameNioEventLoopGroupas the proxy's inbound channels. When the secondary target's pooled channel happened to be assigned to the same event loop thread as the inbound channel, itsPooledTargetResponseHandler.channelRead0()could never fire because the thread was blocked injoin()— a classic Netty deadlock that manifested as aTimeoutException.Fix
Move the blocking
collectResponses()call off the Netty event loop thread viaCompletableFuture.supplyAsync(), then write the response back on the event loop viathenAcceptAsync(_, ctx.channel().eventLoop()).This preserves the original behavior: response triggers as soon as the primary completes, secondaries are collected with their existing timeout on a ForkJoinPool worker thread, and the response is written on the event loop thread for thread safety.
Changes
MultiTargetRoutingHandler.java— 1 file changed, +15/-6 lines. No test changes, no new methods, no removed methods.Signed-off-by: Andre Kurait [email protected]