fix(etcd): load full data from etcd while worker restart#12523
Conversation
Signed-off-by: Nic <[email protected]>
| -- if the startup time of a worker differs significantly from that of the master process, | ||
| -- we consider it to have restarted, and at this point, | ||
| -- it is necessary to reload the full configuration from etcd. | ||
| if configuration_loaded_time and ngx_time() - configuration_loaded_time > 60 then |
There was a problem hiding this comment.
why hardcode it as 60 seconds? could it not be made a configurable parameter?
There was a problem hiding this comment.
This is an empirical value. I think 60 seconds is a sufficient duration to distinguish between the worker's first-time and restart scenarios. It can be made into a configuration parameter, but I don't think users would know how to set this configuration parameter properly.
There was a problem hiding this comment.
but I don't think users would know how to set this configuration parameter properl
makes sense 👍🏼
| end | ||
| local created_obj = {} | ||
| local loaded_configuration = {} | ||
| local configuration_loaded_time |
There was a problem hiding this comment.
I can’t sure if we can use Lua global var to cache the value. If I do this, will store it in sharedict
Pls add test it to confirm
There was a problem hiding this comment.
Okay, I will add a test case, but this one is obviously working. You can see another variable above loaded_configuration, which uses the same mechanism to allow worker processes to reuse the etcd data loaded when the master process starts.
Signed-off-by: Nic <[email protected]>
Signed-off-by: Nic <[email protected]>
Description
When using etcd as the config server, the apisix master process performs a full data load during the init phase and saves the result in
loaded_configuration. After the master forks other worker processes, these worker processes will directly initialize configurations based onloaded_configuration:apisix/apisix/core/config_etcd.lua
Lines 1023 to 1031 in 1b0bb84
Since
loaded_configurationin the master process is not continuously updated, after running for some time,loaded_configurationmay become outdated. If a worker restarts at this point, it will use an outdated configuration for initialization. If any requests are sent to this worker at that time, these requests may be processed by incorrect confiurations.Benchmark
Because the issue addressed by this PR only becomes apparent when APISIX is continuously handling high QPS requests, I am unable to provide effective automated integration tests.
So I conducted a continuous one hour stability test, here are the testing steps and results.
Steps
Result
Which issue(s) this PR fixes:
Fixes #
Checklist