Skip to content

bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large #7042

Description

@redynasc

Current Behavior

Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large

Expected Behavior

smooth memory usage

Error Logs

Memory information of the production environment, captured using the openresty xray tool
job-4378273856

Steps to Reproduce

  1. change worker_processes to 1 for easy observation
  2. create upstream
    curl "127.0.0.1:9080/apisix/admin/upstreams" -XPOST -H"x_api_key:edd1c9f034335f136f87ad84b625c8f1" -d '
    {
    "type": "chash",
    "key": "chash-key",
    "retries": 0,
    "name": "stress_apisix-memory-test_28002",
    "desc": "stress_apisix-memory-test_28002",
    "hash_on": "header",
    "timeout": {
    "send": 0.1,
    "connect": 0.1,
    "read": 0.1
    },
    "nodes": [
    {
    "port": 28002,
    "weight": 5000,
    "host": "192.168.0.1"
    },
    {
    "port": 28002,
    "weight": 5000,
    "host": "192.168.0.2"
    }
    ]
    }
    '
  3. create route
    curl "127.0.0.1:9080/apisix/admin/routes" -XPOST -H"x_api_key:edd1c9f034335f136f87ad84b625c8f1" -d '
    {
    "upstream_id": "the upstream id created in step 2",
    "hosts": [
    "sc.test.com"
    ],
    "uris": [
    "/mem/test"
    ],
    "name": "duisys>apisix-memory-test>prod",
    "desc": "duisys>apisix-memory-test>prod"
    }
    '
  4. run below command
    for((i=1;i<100;i++)); do
    curl "127.0.0.1:9080/apisix/admin/upstreams/{the upstream id created in step 2}" -XPATCH -H"x_api_key:edd1c9f034335f136f87ad84b625c8f1" -d '
    {
    "desc": "stress_apisix-memory-test_28002"
    }
    ' > /dev/null 2>&1 ;
    curl "127.0.0.1:9080/mem/test" -H"host:sc.test.com" > /dev/null 2>&1;
    done
  5. the memory cost of worker is up to 1.2GB
    image

Environment

  • APISIX version :2.13.1
  • Operating system:Linux 3.10.0-693.el7.x86_64 change: added doc of how to load plugin. #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • OpenResty:
    nginx version: openresty/1.19.9.1
    built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
    built with OpenSSL 1.1.1n 15 Mar 2022
    TLS SNI support enabled
    configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.19.9.1.5 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.20 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.10 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../mod_dubbo --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../ngx_multi_upstream_module --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../apisix-nginx-module --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../apisix-nginx-module/src/stream --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../wasm-nginx-module --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../lua-var-nginx-module --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
  • etcd version: 3.5.0

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions