mTLS for TCP not working as expected with AMQP traffic #4999
Description
Bug Report
What is the issue?
Investigating mTLS for TCP using the edge-20.9.3 release. Discussion on Slack suggests that AMQP traffic should be successfully meshed and encrypted with mTLS in this release, as AMQP is a client-speaks-first protocol. The proxies are injected fine, traffic is proxied through them and AMQP messages can be published and consumed as normal. But the output of linkerd edges pod does not show the AMQP client and server appearing as a pair, and the proxy logs appear to show that an mTLS connection is not being created for this traffic.
How can it be reproduced?
# Fire up rabbitmq server
cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: rabbitmq
name: rabbitmq-controller
spec:
replicas: 1
selector:
matchLabels:
component: rabbitmq
template:
metadata:
labels:
app: taskQueue
component: rabbitmq
spec:
containers:
- image: rabbitmq
name: rabbitmq
ports:
- containerPort: 5672
resources:
limits:
cpu: 100m
---
apiVersion: v1
kind: Service
metadata:
labels:
component: rabbitmq
name: rabbitmq-service
spec:
ports:
- port: 5672
selector:
app: taskQueue
component: rabbitmq
EOF
# set up an AMQP client
kubectl run -i --tty temp --image ubuntu:18.04
# run the rest of these commands in the client
apt-get update && apt-get install -y curl ca-certificates amqp-tools python dnsutils
export BROKER_URL=amqp://guest:guest@rabbitmq-service:5672
# Wait for the AMQP server to complete initialization before running the rest of these commands; it can take a few minutes.
# Create the queue
/usr/bin/amqp-declare-queue --url=$BROKER_URL -q foo -d
# Publish two messages to the queue
/usr/bin/amqp-publish --url=$BROKER_URL -r foo -p -b Hello
/usr/bin/amqp-publish --url=$BROKER_URL -r foo -p -b there
# Get them back
/usr/bin/amqp-consume --url=$BROKER_URL -q foo -c 1 cat && echo
/usr/bin/amqp-consume --url=$BROKER_URL -q foo -c 1 cat && echo
Logs, error output, etc
tcpdump output for an AMQP connection:
20:59:19.172106 IP 10.0.1.7.43496 > 172.20.160.93.5672: Flags [S], seq 2122865703, win 26883, options [mss 8961,sackOK,TS val 4159879790 ecr 0,nop,wscale 7], length 0
20:59:19.172168 IP 172.20.160.93.5672 > 10.0.1.7.43496: Flags [S.], seq 2179216082, ack 2122865704, win 26847, options [mss 8961,sackOK,TS val 586747547 ecr 4159879790,nop,wscale 7], length 0
20:59:19.172175 IP 10.0.1.7.43496 > 172.20.160.93.5672: Flags [.], ack 1, win 211, options [nop,nop,TS val 4159879791 ecr 586747547], length 0
20:59:19.172220 IP 10.0.1.7.43496 > 172.20.160.93.5672: Flags [P.], seq 1:9, ack 1, win 211, options [nop,nop,TS val 4159879791 ecr 586747547], length 8
20:59:19.172241 IP 172.20.160.93.5672 > 10.0.1.7.43496: Flags [.], ack 9, win 210, options [nop,nop,TS val 586747548 ecr 4159879791], length 0
20:59:19.173644 IP 172.20.160.93.5672 > 10.0.1.7.43496: Flags [P.], seq 1:536, ack 9, win 210, options [nop,nop,TS val 586747549 ecr 4159879791], length 535
Output of linkerd edges pod:
[adrianp@centos-01]$ linkerd edges pod
SRC DST SRC_NS DST_NS SECURED
linkerd-prometheus-65bc77bcd8-9t874 rabbitmq-controller-5886db8469-xq859 linkerd adrian √
linkerd-prometheus-65bc77bcd8-9t874 temp-66fc9bc84d-rnpnb linkerd adrian √
rabbitmq pod logs (rabbitmq and linkerd-proxy containers):
rabbitmq-controller-5886db8469-xq859 rabbitmq 2020-09-23 21:10:48.475 [info] <0.1084.0> accepting AMQP connection <0.1084.0> (127.0.0.1:56614 -> 127.0.0.1:5672)
rabbitmq-controller-5886db8469-xq859 linkerd-proxy [ 326.135520569s] DEBUG ThreadId(09) inbound:accept{peer.addr=10.0.1.134:49858}: linkerd2_app_inbound::require_identity_for_ports: port=5672 peer.id=Some("default.adrian.serviceaccount.identity.linkerd.cluster.local") id_required=false
rabbitmq-controller-5886db8469-xq859 linkerd-proxy [ 326.135562939s] DEBUG ThreadId(09) inbound:accept{peer.addr=10.0.1.134:49858}: linkerd2_app_inbound::prevent_loop: port=5672 self.port=4143
rabbitmq-controller-5886db8469-xq859 linkerd-proxy [ 326.135637091s] DEBUG ThreadId(09) inbound:accept{peer.addr=10.0.1.134:49858}: linkerd2_proxy_http::version: Not HTTP
rabbitmq-controller-5886db8469-xq859 linkerd-proxy [ 326.135684160s] DEBUG ThreadId(09) inbound:accept{peer.addr=10.0.1.134:49858}: linkerd2_proxy_transport::connect: Connecting peer.addr=127.0.0.1:5672
rabbitmq-controller-5886db8469-xq859 linkerd-proxy [ 326.135880283s] DEBUG ThreadId(07) inbound:accept{peer.addr=10.0.1.134:49858}: linkerd2_proxy_transport::connect: Connected local.addr=127.0.0.1:56614 keepalive=None
rabbitmq-controller-5886db8469-xq859 linkerd-proxy [ 326.135922634s] DEBUG ThreadId(07) inbound:accept{peer.addr=10.0.1.134:49858}: linkerd2_proxy_transport::metrics: client connection open
rabbitmq-controller-5886db8469-xq859 rabbitmq 2020-09-23 21:10:48.478 [info] <0.1084.0> connection <0.1084.0> (127.0.0.1:56614 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
rabbitmq-controller-5886db8469-xq859 rabbitmq 2020-09-23 21:10:48.483 [info] <0.1084.0> closing AMQP connection <0.1084.0> (127.0.0.1:56614 -> 127.0.0.1:5672, vhost: '/', user: 'guest')
rabbitmq-controller-5886db8469-xq859 linkerd-proxy [ 326.145104510s] INFO ThreadId(07) inbound:accept{peer.addr=10.0.1.134:49858}: linkerd2_app_core::serve: Connection closed error=Connection reset by peer (os error 104)
Environment
- Kubernetes Version: 1.15
- Cluster Environment: EKS
- Linkerd version: edge-20.9.3