Current Behavior
We are now using APISIX in a Kubernetes setup with Helm (https://github.com/apache/apisix-helm-chart).
APISIX is running as a service in the Istio Service Mesh with Envoy sidecar applied on it.
We faced an issue where after we enabled mTLS with Istio, requests targeted to APISIX failed.
During debugging we saw that the authority header for the outgoing requests from the APISIX was always set to apisix_backend. We think that his is confusing Istio during the mTLS and results in the request failure.
Expected Behavior
Requests should be successful both with Istio mTLS enabled and disabled.
Error Logs
Request Headers Info (from client grpcurl)
authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.eyJleHAiOjE2NTY2ODg4NzUsImlhdCI6MTY1NjYwMjQ3NSwiYXV0aF90aW1lIjoxNjU2NDIwNzAzLCJqdGkiOiI5ZTk1ZDdhZS0yNzlmLTRlNTktODE0Yi1mYzNkMzNmYmM4MDEiLCJpc3MiOiJodHRwczovL3RvZ2dpZC50b2dnLmNsb3VkL2F1dGgvcmVhbG1zL3RvZ2dpZCIsInN1YiI6IjgwZTkxMzYwLWVmOTAtNDI5OC05ODRkLTcxNDBiNDY5NTFlMyIsInR5cCI6IkJlYXJlciIsImF6cCI6InN1cGVyLWFwcCIsInNlc3Npb25fc3RhdGUiOiI5ZDUwNmFjMS1mNGZlLTRiOTUtYjJkNy1iYTdjOTg0ODc1MjUiLCJhY3IiOiIwIiwic2NvcGUiOiJvcGVuaWQgZW1haWwgcHJvZmlsZSIsInNpZCI6IjlkNTA2YWMxLWY0ZmUtNGI5NS1iMmQ3LWJhN2M5ODQ4NzUyNSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiVGVzdCBUZXN0IiwicHJlZmVycmVkX3VzZXJuYW1lIjoidGVzdEB0b2dnLmNsb3VkIiwiZ2l2ZW5fbmFtZSI6IlRlc3QiLCJmYW1pbHlfbmFtZSI6IlRlc3QiLCJlbWFpbCI6InRlc3RAdG9nZy5jbG91ZCJ9.BK-EGfdyi7DoIQTRYxUFBK54f4g2IyAK6DlQDinDldjf2OXFRyWIK9OwN7Q5-hW5BO0hn0huJ4aQZ59WGUdZ5RjZqVCV3-w2ybr7BXHkwKJYnjrB0lcFy4in1WB_eiD4TMBdqb7vG6dxC8bGdm8YmBfFvJ7Ufghle33pjj67k8SJj3zUFRBK-f4umKesakfTlhlMMdALbCTxV9jIoXPtDpvDEF6V89N7LKnnoV8Q3lPBF56PGeBokdqEJLfsb5ZQcaMeW8Fi38adqZTa8A4WefoRRsOrgEhXMYoU8DrY1EWvatgms4vJKag6bygkp_2nsNKT__hoYIDBvvJMke60VQ
Response headers (from client grpcurl)
content-length: 0
content-type: application/grpc
date: Thu, 30 Jun 2022 15:39:53 GMT
server: istio-envoy
x-envoy-upstream-service-time: 84
Logs from Envoy Proxy sidecar container for APISIX
{
"duration"1,
"downstream_remote_address":"172.20.50.105:0",
"upstream_service_time":null,
"upstream_local_address":"10.234.106.162:50972",
"response_code_details":"upstream_reset_before_response_started{connection_termination}",
"upstream_transport_failure_reason":null,
"route_name":"allow_any",
"response_code":200,
"upstream_host":"10.234.29.1:80","user_agent":"grpcurl/v1.8.1 grpc-go/1.37.0",
"downstream_local_address":"10.234.29.1:80",
"x_forwarded_for":"172.20.50.105",
"connection_termination_details":null,
"protocol":"HTTP/2",
"upstream_cluster":"PassthroughCluster",
"authority":"apisix_backend",
"method":"POST",
"start_time":"2022-06-30T15:39:53.730Z",
"path":"/xxxx.yyyy.ms.profile.ProfileService/GetUserProfile",
"bytes_received":51,
"response_flags":"UC",
"request_id":"7e4abc76-27f4-4f40-b663-056c648608b7",
"bytes_sent":0,
"requested_server_name":null
}
Logs from APISIX container
127.0.0.6 - - [30/Jun/2022:15:39:53 +0000] xxxxx-api-gateway.xxxx.cloud:9443 "POST /xxxx.yyyy.ms.profile.ProfileService/GetUserProfile HTTP/2.0" 200 0 0.005 "-" "grpcurl/v1.8.1 grpc-go/1.37.0" 10.234.29.1:80 200 0.004 "grpc://xxxxxx-api-gateway.xxxx.cloud:9443"
Steps to Reproduce
- Install Istio in k8s cluster
- Enable Istio Strict mTLS
$ kubectl apply -n istio-system -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: "default"
spec:
mtls:
mode: STRICT
EOF
- Create namespace for APISIX and enable auto injection for Istio
- Install APISIX with a Helm chart
- Install a gRPC Service in the same namespace
- Create Route and Upstream in APISIX
{
"uris": [
"/xxxx.yyyyy.ms.profile.ProfileService/*"
],
"name": "Profile_Service",
"methods": [
"POST",
"GET"
],
"upstream_id": "369132756648592144",
"status": 1
}
{
"nodes": [
{
"host": "grpc-service.namespace.svc.cluster.local",
"port": 80,
"weight": 1
}
],
"timeout": {
"connect": 6,
"read": 6,
"send": 6
},
"type": "roundrobin",
"scheme": "grpc",
"pass_host": "pass",
"name": "Profile Service"
}
Environment
- APISIX version: 2.12.1
- k8s version: 1.20.7
- Istio version: 1.10.3
Current Behavior
We are now using APISIX in a Kubernetes setup with Helm (https://github.com/apache/apisix-helm-chart).
APISIX is running as a service in the Istio Service Mesh with Envoy sidecar applied on it.
We faced an issue where after we enabled mTLS with Istio, requests targeted to APISIX failed.
During debugging we saw that the
authorityheader for the outgoing requests from the APISIX was always set toapisix_backend. We think that his is confusing Istio during the mTLS and results in the request failure.Expected Behavior
Requests should be successful both with Istio mTLS enabled and disabled.
Error Logs
Request Headers Info (from client grpcurl)
Response headers (from client grpcurl)
Logs from Envoy Proxy sidecar container for APISIX
Logs from APISIX container
Steps to Reproduce
Environment