Skip to content

Fix event loop deadlock in MultiTargetRoutingHandler - #2718

Merged
AndreKurait merged 1 commit into
opensearch-project:mainfrom
AndreKurait:fix-shim-proxy-test-flaky
Apr 15, 2026
Merged

AndreKurait merged 1 commit into
opensearch-project:mainfrom
AndreKurait:fix-shim-proxy-test-flaky

Conversation

@AndreKurait

@AndreKurait AndreKurait commented Apr 14, 2026 •

Copy link
Copy Markdown
Member

Description

Fixes flaky ShimProxyTest dual-target tests (dualTarget_validationPass, dualTarget_cursorMarkMerge) which intermittently fail with TimeoutException on the secondary target.

Root Cause

Event loop deadlock in MultiTargetRoutingHandler.

handlePrimaryCompletion() attached a whenComplete callback to the primary future, then ran on the inbound channel's event loop thread via ctx.channel().eventLoop().execute(...). Inside that callback, collectSingleResponse() called future.join() — a blocking call on the event loop thread.

The backend connection pool (FixedChannelPool) shares the same NioEventLoopGroup as the proxy's inbound channels. When the secondary target's pooled channel happened to be assigned to the same event loop thread as the inbound channel, its PooledTargetResponseHandler.channelRead0() could never fire because the thread was blocked in join() — a classic Netty deadlock that manifested as a TimeoutException.

Fix

Move the blocking collectResponses() call off the Netty event loop thread via CompletableFuture.supplyAsync(), then write the response back on the event loop via thenAcceptAsync(_, ctx.channel().eventLoop()).

This preserves the original behavior: response triggers as soon as the primary completes, secondaries are collected with their existing timeout on a ForkJoinPool worker thread, and the response is written on the event loop thread for thread safety.

Changes

  • MultiTargetRoutingHandler.java — 1 file changed, +15/-6 lines. No test changes, no new methods, no removed methods.

Signed-off-by: Andre Kurait [email protected]

@codecov

codecov Bot commented Apr 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.43%. Comparing base (cdacaad) to head (82d85de).
⚠️ Report is 26 commits behind head on main.

Files with missing lines Patch % Lines
...ransform/shim/netty/MultiTargetRoutingHandler.java 80.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2718      +/-   ##
============================================
- Coverage     73.43%   73.43%   -0.01%     
  Complexity      106      106              
============================================
  Files           721      721              
  Lines         33368    33372       +4     
  Branches       2910     2907       -3     
============================================
+ Hits          24504    24506       +2     
- Misses         7533     7535       +2     
  Partials       1331     1331              
Flag Coverage Δ
gradle 69.85% <80.00%> (+<0.01%) ⬆️
node 90.97% <ø> (ø)
python 77.75% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Move collectResponses() (which blocks on secondary future.join()) off the
Netty event loop thread via CompletableFuture.supplyAsync(). The secondary
FixedChannelPools share the same NioEventLoopGroup, so calling join() on
the event loop can deadlock when a pooled channel's response handler needs
that same thread to fire channelRead0.

The fix preserves the original behavior: response triggers as soon as the
primary completes, secondaries are collected with their existing timeout on
a ForkJoinPool worker thread, then the response is written back on the
event loop thread via thenAcceptAsync.

Signed-off-by: Andre Kurait <[email protected]>
@AndreKurait
AndreKurait force-pushed the fix-shim-proxy-test-flaky branch from a47e58e to 82d85de Compare April 14, 2026 19:48
@AndreKurait
AndreKurait enabled auto-merge April 14, 2026 19:49
@AndreKurait AndreKurait changed the title Fix flaky ShimProxyTest.dualTarget_validationPass() Fix event loop deadlock in MultiTargetRoutingHandler Apr 14, 2026
@jugal-chauhan jugal-chauhan self-assigned this Apr 15, 2026

@jugal-chauhan jugal-chauhan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for brining in these changes

@AndreKurait
AndreKurait merged commit abe6949 into opensearch-project:main Apr 15, 2026
71 checks passed
AndreKurait added a commit to AndreKurait/opensearch-migrations that referenced this pull request Apr 15, 2026
…proxy-test-flaky

Fix flaky ShimProxyTest.dualTarget_validationPass()

This branch was previously deployed

1 inactive deployment
migrations-cicd — 82d85de8 Deployed Apr 14, 2026 by AndreKurait via elasticsearch-5x-k8s-local-test #8110
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants