Skip to content

bug: apisix 内存管理-过期监控key不释放问题 #9627

Description

@wood-zhang

Current Behavior

如果打开了prometheus插件,并且upstream使用了k8s服务发现或者upstream ip随着发布而改变的话,在apisix中就会产生过多的监控key,从而导致内存不断增长,如果不重启apisix最终OOM
Uploading image.png…

Expected Behavior

我期望在upstream ip改变的时候有一个自动检测机制,把内存里的监控指标中不存在的node节点的key进行释放

Error Logs

No response

Steps to Reproduce

我通过将指标中的node纬度关闭从而规避了因发布而导致的upstream ip 改变产生过多key的问题
image

Environment

  • APISIX version (run apisix version):
  • Operating system (run uname -a):
  • OpenResty / Nginx version (run openresty -V or nginx -V):
  • etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
  • APISIX Dashboard version, if relevant:
  • Plugin runner version, for issues related to plugin runners:
  • LuaRocks version, for installation issues (run luarocks --version):

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status
    ✅ Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions