fix(cluster): use lazy connections to prevent unresponsive cluster on node power-off #1044
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… node power-off This fixes issue rustfs#1001 where the cluster becomes unresponsive when a node is abruptly powered off (vs. graceful kill which works correctly). Root cause: The gRPC client was using connect().await which blocks waiting for a connection to be established. When a node is abruptly powered off without sending 'goodbye' messages, TCP connections can hang for extended periods (2+ minutes) waiting for OS TCP timeouts. Changes: - Changed from connect().await to connect_lazy() in node_service_time_out_client - connect_lazy() returns immediately without establishing a connection - Connection is established lazily on first request - Tonic's lazy channel handles automatic reconnection when nodes come back - Requests to unreachable nodes fail quickly with timeout errors instead of blocking indefinitely - Reduced default request timeout from 60s to 30s for faster failure detection - Added clear_connection() and clear_all_connections() helper functions to allow manual clearing of potentially stale connections from the cache This ensures the cluster remains responsive even when nodes vanish abruptly, as requests will fail fast with proper timeout errors rather than blocking indefinitely. Fixes: rustfs#1001
15 tasks
Contributor
|
Hey @tennisleng , There may be a conflict between 1054 and 1044 regarding feature fixes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes issue #1001 where the cluster becomes unresponsive when a node is abruptly powered off (e.g., by cutting the power), while gracefully killing the process works correctly.
Root Cause Analysis
The gRPC client was using
connect().awaitwhich blocks waiting for a connection to be established. When a node is abruptly powered off:connect().awaitblocks waiting for OS TCP timeouts (can be 2+ minutes)In contrast, when a process is gracefully killed:
Changes
Core Fix
Changed from
connect().awaittoconnect_lazy()innode_service_time_out_client:connect_lazy()returns immediately without establishing a connectionAdditional Improvements
clear_connection()andclear_all_connections()helper functions to allow manual clearing of potentially stale connections from the cacheFiles Changed
crates/protos/src/lib.rs: Changed connection strategy from eager to lazycrates/common/src/globals.rs: Added helper functions for connection cache managementTesting
This fix should be tested by:
Fixes: #1001