Skip to content

bug: always reports etcd /v3/watch connection timeout 900s after apisix startup #9578

Description

@DroidEye2ONGU

Current Behavior

There are two servers , one for single etcd and the other for apisix. When I started apisix, everything looks fine, but onec you get past 900s, the apisix log will keep saying this
[error] 41073#41073: *777193 upstream timed out (110: Connection timed out) while reading upstream, client: unix:, server: , request: "POST /v3/watch HTTP/1.1", upstream: "http://myetcdhost:2379/v3/watch", host: "127.0.0.1" server: , request: "POST /v3/watch HTTP/1.1", upstream: "http://myetcdhost:2379/v3/watch", host: "127.0.0.1"

When this error started, every apisix functions work fine, no matter proxy or update config through apisix-dashboard, and etcd can also receive requests. I can also send request from the apisix server to etcd server.

Expected Behavior

I hope this error won't appear

Error Logs

[error] 41073#41073: *777193 upstream timed out (110: Connection timed out) while reading upstream, client: unix:, server: , request: "POST /v3/watch HTTP/1.1", upstream: "http://myetcdhost:2379/v3/watch", host: "127.0.0.1" server: , request: "POST /v3/watch HTTP/1.1", upstream: "http://myetcdhost:2379/v3/watch", host: "127.0.0.1"

Steps to Reproduce

  1. deploy single etcd node on server.
  2. install apisix by apisix installation-guide using rpm. By the way, the server is online, it cannot touch internet.
  3. change some apisix config, mainly ext java-plugin, additional nginx config, and etcd address.
    etcd:
    host:
  4. start apisix using systemctl start apisix

Environment

  • APISIX version (run apisix version): 3.3.0
  • Operating system (run uname -a): Linux archive2 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
  • OpenResty / Nginx version (run openresty -V or nginx -V):
  • etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.0 (apisix said it's 3.5.0 but it's actually 3.5.4 )
  • APISIX Dashboard version, if relevant: 3.0.1
  • Plugin runner version, for issues related to plugin runners:
  • LuaRocks version, for installation issues (run luarocks --version):

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status
    ✅ Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions