Current Behavior

apisix 3.3 runs with etcd v3.5. It generates core dump at night. It seems connection to etcd has problem. Then grpc library generates error response structure. It uses the field which is nil in this structure at that time.

Expected Behavior

No response

Error Logs

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7f0bc8650e60]

goroutine 4038275 [running]:
google.golang.org/grpc/internal/status.(*Error).GRPCStatus(0x7f0bc86de826?)
/root/go/pkg/mod/google.golang.org/[email protected]/internal/status/status.go:155
google.golang.org/grpc/status.FromError({0x7f0bc88aa660, 0x0})
/root/go/pkg/mod/google.golang.org/[email protected]/status/status.go:94 +0xea
google.golang.org/grpc.toRPCErr({0x7f0bc88aa660, 0x0})
/root/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:848 +0x288
google.golang.org/grpc.(*csAttempt).recvMsg(0xc005bb4210, {0x7f0bc8895460?, 0xc0050ba810}, 0x80?)
/root/go/pkg/mod/google.golang.org/[email protected]/stream.go:1002 +0x3ba
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc005d19eb8?)
/root/go/pkg/mod/google.golang.org/[email protected]/stream.go:845 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc005b850e0, 0xc005d19f20, 0xc005d19ef0)
/root/go/pkg/mod/google.golang.org/[email protected]/stream.go:705 +0x2f1
google.golang.org/grpc.(*clientStream).RecvMsg(0xc005b850e0, {0x7f0bc8895460?, 0xc0050ba810?})
/root/go/pkg/mod/google.golang.org/[email protected]/stream.go:844 +0x11f
github.com/api7/grpc-client-nginx-module/conn.(*Stream).Recv(0xc0000989c0?)
/tmp/tmp.cLFc1OlJDR/grpc-client-nginx-module-v0.4.2/grpc-engine/conn/grpc.go:192 +0x7a
main.grpc_engine_stream_recv.func1()
/tmp/tmp.cLFc1OlJDR/grpc-client-nginx-module-v0.4.2/grpc-engine/main.go:233 +0x26
created by main.grpc_engine_stream_recv
/tmp/tmp.cLFc1OlJDR/grpc-client-nginx-module-v0.4.2/grpc-engine/main.go:232 +0x91
2023/05/13 21:16:50 [notice] 21446#21446: signal 17 (SIGCHLD) received from 21447
2023/05/13 21:16:50 [alert] 21446#21446: worker process 21447 exited on signal 6
2023/05/13 21:16:50 [notice] 21446#21446: start worker process 9113
2023/05/13 21:16:50 [notice] 21446#21446: signal 29 (SIGIO) received

Steps to Reproduce

run apisix in centos 7
config apisix:

etcd:
use_grpc: true

Environment

• APISIX version (run apisix version): apisix 3.3
• Operating system (run uname -a): centos 7
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):