Skip to content

fix(etcd): load full data from etcd while worker restart#12523

Merged
nic-6443 merged 3 commits into
apache:masterfrom
nic-6443:nic/load-full-data-when-worker-restart
Aug 20, 2025
Merged

fix(etcd): load full data from etcd while worker restart#12523
nic-6443 merged 3 commits into
apache:masterfrom
nic-6443:nic/load-full-data-when-worker-restart

Conversation

@nic-6443

@nic-6443 nic-6443 commented Aug 14, 2025

Copy link
Copy Markdown
Member

Description

When using etcd as the config server, the apisix master process performs a full data load during the init phase and saves the result in loaded_configuration. After the master forks other worker processes, these worker processes will directly initialize configurations based on loaded_configuration:

if loaded_configuration[key] then
local res = loaded_configuration[key]
loaded_configuration[key] = nil -- tried to load
log.notice("use loaded configuration ", key)
local dir_res, headers = res.body, res.headers
load_full_data(obj, dir_res, headers)
end

Since loaded_configuration in the master process is not continuously updated, after running for some time, loaded_configuration may become outdated. If a worker restarts at this point, it will use an outdated configuration for initialization. If any requests are sent to this worker at that time, these requests may be processed by incorrect confiurations.

Benchmark

Because the issue addressed by this PR only becomes apparent when APISIX is continuously handling high QPS requests, I am unable to provide effective automated integration tests.
So I conducted a continuous one hour stability test, here are the testing steps and results.

Steps

  1. start APISIX with two workers
nginx_config:
  worker_processes: 2
  1. create 1000 routes after APISIX startup
#!/usr/bin/env bash
set -euo pipefail

APISIX_HOST="http://127.0.0.1:9180"
ADMIN_API_KEY="${ADMIN_API_KEY:-edd1c9f034335f136f87ad84b625c8f1}"

for i in $(seq 1 1000); do
  echo ">>> creating upstream $i"
  curl -s -o /dev/null -X PUT "$APISIX_HOST/apisix/admin/upstreams/$i" \
    -H "X-API-KEY: $ADMIN_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{
      \"id\": $i,
      \"nodes\": {\"127.0.0.1:1980\": 1},
      \"type\": \"roundrobin\"
    }"

  echo ">>> creating route $i"
  curl -s -o /dev/null -X PUT "$APISIX_HOST/apisix/admin/routes/$i" \
    -H "X-API-KEY: $ADMIN_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{
      \"id\": $i,
      \"uri\": \"/$i\",
      \"upstream_id\": $i
    }"
done

echo ">>> All routes and upstreams created"
  1. kill one worker process randomly every 60s
#!/usr/bin/env bash
set -euo pipefail

MASTER_PID="${NGINX_MASTER_PID:?must set NGINX_MASTER_PID}"

while true; do
  worker_pids=$(pgrep -P "$MASTER_PID" -f "nginx: worker process" || true)

  if [ -n "$worker_pids" ]; then
    pid=$(echo "$worker_pids" | shuf -n 1)
    echo ">>> killing worker $pid (master $MASTER_PID)"
    kill "$pid"
  else
    echo ">>> no worker process found for master $MASTER_PID"
  fi

  sleep 60
done
  1. use wrk to continuously request APISIX
wrk -t2 -c100 -R6000 -d1h -s request.lua http://127.0.0.1:9080

Result

❯ wrk -t2 -c100 -R6000 -d1h -s request.lua http://127.0.0.1:9080

Running 60m test @ http://127.0.0.1:9080
  2 threads and 100 connections
  Thread calibration: mean lat.: 1.313ms, rate sampling interval: 10ms
  Thread calibration: mean lat.: 1.331ms, rate sampling interval: 10ms
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.60ms    9.28ms 337.15ms   99.31%
    Req/Sec     2.92k   508.56    18.00k    74.18%
  19835460 requests in 60.00m, 3.31GB read
  Socket errors: connect 0, read 2623, write 0, timeout 14660
Requests/sec:   5509.85
Transfer/sec:      0.94MB

Which issue(s) this PR fixes:

Fixes #

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

Comment thread apisix/core/config_etcd.lua Outdated
-- if the startup time of a worker differs significantly from that of the master process,
-- we consider it to have restarted, and at this point,
-- it is necessary to reload the full configuration from etcd.
if configuration_loaded_time and ngx_time() - configuration_loaded_time > 60 then

@shreemaan-abhishek shreemaan-abhishek Aug 15, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why hardcode it as 60 seconds? could it not be made a configurable parameter?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an empirical value. I think 60 seconds is a sufficient duration to distinguish between the worker's first-time and restart scenarios. It can be made into a configuration parameter, but I don't think users would know how to set this configuration parameter properly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but I don't think users would know how to set this configuration parameter properl

makes sense 👍🏼

@nic-6443
nic-6443 marked this pull request as ready for review August 19, 2025 09:04
@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Aug 19, 2025
@dosubot dosubot Bot added the bug Something isn't working label Aug 19, 2025
end
local created_obj = {}
local loaded_configuration = {}
local configuration_loaded_time

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can’t sure if we can use Lua global var to cache the value. If I do this, will store it in sharedict
Pls add test it to confirm

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I will add a test case, but this one is obviously working. You can see another variable above loaded_configuration, which uses the same mechanism to allow worker processes to reuse the etcd data loaded when the master process starts.

Signed-off-by: Nic <[email protected]>
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Aug 19, 2025
Signed-off-by: Nic <[email protected]>
@nic-6443
nic-6443 merged commit 33b9632 into apache:master Aug 20, 2025
30 of 31 checks passed
@nic-6443
nic-6443 deleted the nic/load-full-data-when-worker-restart branch August 20, 2025 01:40
jizhuozhi pushed a commit to jizhuozhi/apisix that referenced this pull request Oct 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants