Current Behavior

There're too many tcp connections when using http1.1 to communicate with etcd cluster before apisix 3.1.0. To reduce
connections, I upgrade apisix to 3.1.0 to use grpc connecting to etcd cluster. At beginning, it does reduce connections to etcd cluster. However, it generates core dump without any manually operation in the early morning.

Apisix's error log indicates that Apisix 3.1.0 generate core dump while receiving grpc client stream. Is the function that grpc connects to etcd cluster in apisix 3.1.0 still experimental? When could it be ready for productive environment, please?

use gdb to show backstack of core dump generated by openresty is as below:

(gdb) bt
warning: skipping .debug_frame info of /usr/local/openresty/libgrpc_engine.so: Found an FDE when not expecting it.
#0 0x00007fa821c631c1 in runtime.raise.abi0 () from /usr/local/openresty/libgrpc_engine.so
#1 0x00007fa821c47365 in runtime.dieFromSignal () from /usr/local/openresty/libgrpc_engine.so
#2 0x000000c000000006 in ?? ()
#3 0x00007fa821c3075a in runtime.preprintpanics () from /usr/local/openresty/libgrpc_engine.so
#4 0x000000c009d23ac8 in ?? ()
#5 0x00007fa821c31d2d in runtime.fatalpanic () from /usr/local/openresty/libgrpc_engine.so
#6 0x000000c000000006 in ?? ()
#7 0x01007fa821c30909 in ?? ()
#8 0x00007fa821c31d60 in ?? () from /usr/local/openresty/libgrpc_engine.so
#9 0x000000c009d23b50 in ?? ()
#10 0x000000c00abd36c0 in ?? ()
#11 0x00007fa821c313fa in runtime.gopanic () from /usr/local/openresty/libgrpc_engine.so
#12 0x000000c009d23ad8 in ?? ()
#13 0x000000c009d23a97 in ?? ()
#14 0x000000c009d23b88 in ?? ()
#15 0x00007fa821c313fa in runtime.gopanic () from /usr/local/openresty/libgrpc_engine.so
#16 0x0000000000000000 in ?? ()

Expected Behavior

No response

Error Logs

the panic backstack in apisix error log as below:

2023/03/06 05:57:08 [notice] 4157#4157: signal 17 (SIGCHLD) received from 4164
2023/03/06 05:57:08 [alert] 4157#4157: worker process 4164 exited on signal 6 (core dumped)

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7fa821f82220]

goroutine 5384582 [running]:
google.golang.org/grpc/internal/status.(*Error).GRPCStatus(0x7fa82200fbe6?)
/root/go/pkg/mod/google.golang.org/[email protected]/internal/status/status.go:155
google.golang.org/grpc/status.FromError({0x7fa8221db020, 0x0})
/root/go/pkg/mod/google.golang.org/[email protected]/status/status.go:94 +0xea
google.golang.org/grpc.toRPCErr({0x7fa8221db020, 0x0})
/root/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:848 +0x288
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00986adc0, {0x7fa8221c5e40?, 0xc0097c0090}, 0x1?)
/root/go/pkg/mod/google.golang.org/[email protected]/stream.go:1002 +0x3ba
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc009f9984c?)
/root/go/pkg/mod/google.golang.org/[email protected]/stream.go:845 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc009f997a0, 0xc00ad6cf20, 0xc00ad6cef0)
/root/go/pkg/mod/google.golang.org/[email protected]/stream.go:705 +0x2f1
google.golang.org/grpc.(*clientStream).RecvMsg(0xc009f997a0, {0x7fa8221c5e40?, 0xc0097c0090?})
/root/go/pkg/mod/google.golang.org/[email protected]/stream.go:844 +0x11f
github.com/api7/grpc-client-nginx-module/conn.(*Stream).Recv(0xc00ad6cfb0?)
/tmp/tmp.tYflEs0Gp1/grpc-client-nginx-module-v0.4.1/grpc-engine/conn/grpc.go:192 +0x7a
main.grpc_engine_stream_recv.func1()
/tmp/tmp.tYflEs0Gp1/grpc-client-nginx-module-v0.4.1/grpc-engine/main.go:233 +0x26
created by main.grpc_engine_stream_recv
/tmp/tmp.tYflEs0Gp1/grpc-client-nginx-module-v0.4.1/grpc-engine/main.go:232 +0x91

Steps to Reproduce

1. configure apisix3.1.0 as below:

deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
admin_key:
- name: admin
key: ilikecocacola # using fixed API token has security risk, please update it when you deploy to production environment
role: admin
etcd:
use_grpc: true
host:
- "http://10.26.21.245:12378"

nginx_config:
error_log: /data0/apisix_error.log
error_log_level: notice
worker_processes: auto

Environment

apisix 3.1.0
openresty/1.21.4.1
centos 7 linux kernel:3.10.0-862.14.4.el7.x86_64