Skip to content

bug: Use upstream keepalive, resulting in long connection reuse problem #7053

Description

@hahayyum

Current Behavior

example 1:
route:/aaa upstream: test.com, mtls: use client.key and client.cert
route:/bbb upstream; test.com does not use mtls
Result 1:
When accessing /bbb, sometimes the connection of /aaa is reused, causing the request to carry client.key and client.cert .
But sometimes client.key and client.cert are lost when accessing /aaa

or
example 2:
route: https://testA.com/aaa upstream: testA.test.com, mtls: use client.key and client.cert
route: https://testB.com/aaa upstream; testB.test.com does not use mtls
os: /etc/hosts
192.168.35.3 testA.test.com testB.test.com
Result 2:
When accessing https://testB.com/aaa, sometimes the connection of testB.com/aaa is reused, causing the request to carry client.key and client.cert .
But sometimes client.key and client.cert are lost when accessing https://testA.com/aaa

Expected Behavior

expect:
When keepalive multiplexes the connection, use the connection of client.cert and client.key to multiplex this type of connection, and the connection that does not use client authentication is reused for the same type of connection。

After investigation, it is the problem caused by upstream keepalive, which can be solved by using a short connection

What is the reuse of upstream.keepalive long connections in APISIX-OpenResty? only host IP?

Error Logs

No response

Steps to Reproduce

1、create Route /aaa
curl http://localhost:9080/apisix/admin/routes/395051487660606160?api_key=admin -X PUT -i -d '
{
"uri": "/aaa",
"name": "aaa",
"methods": [
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE"
],
"upstream": {
"nodes": [
{
"host": "test.com",
"port": 8090,
"weight": 100
}
],
"timeout": {
"connect": 6,
"send": 6,
"read": 6
},
"type": "roundrobin",
"scheme": "https",
"pass_host": "pass",
"tls": {
"client_cert": "clent_cert",
"client_key": "client_key"
},
"keepalive_pool": {
"idle_timeout": 60,
"requests": 1000,
"size": 320
}
},
"status": 1
}'
2、create Route /bbb
curl http://localhost:9080/apisix/admin/routes/39505148743344535345?api_key=admin -X PUT -i -d '
{
"uri": "/bbb",
"name": "bbb",
"methods": [
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE"
],
"upstream": {
"nodes": [
{
"host": "test.com",
"port": 8090,
"weight": 100
}
],
"timeout": {
"connect": 6,
"send": 6,
"read": 6
},
"type": "roundrobin",
"scheme": "https",
"pass_host": "pass",
"keepalive_pool": {
"idle_timeout": 60,
"requests": 1000,
"size": 320
}
},
"status": 1
}'
3、create upstream server
1)Supports both mtls and no mtls, such as k8s apiserver
2)Different hosts, corresponding to the same service

Environment

environment:
apisix 2.7
use APISIX-OpenResty

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions