You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A thrift_client_retry_interval_ms config should be added in be to avoid avalanche disaster in fe thrift server #4023
Although we have max_running_txn_num_per_db config to prevent excessive system load in fe, but frequent retry may also cause excessive system load in fe, in our production environment, we see many logs like retrying call frontend service, address=TNetworkAddress(hostname=10.118.32.48, port=9020), reason=THRIFT_EAGAIN (timed out) within a short time, and find high system load in fe, some times cause fe core, the main reason is that there are already many backlogs in fe, which cause be thrift client rpc timeout, but be thrift client retry immediately without any breaks, which cause things more worse, finally make avalanche disaster in fe.
changed the title [-]a thrift_client_retry_interval_ms config should be added in be to avoid avalanche disaster in fe thrift server[/-][+]A thrift_client_retry_interval_ms config should be added in be to avoid avalanche disaster in fe thrift server[/+]on Jul 5, 2020
Activity
[-]a thrift_client_retry_interval_ms config should be added in be to avoid avalanche disaster in fe thrift server[/-][+]A thrift_client_retry_interval_ms config should be added in be to avoid avalanche disaster in fe thrift server[/+]