Linkerd drops CONNECT requests with content-length header set #8539
Description
What is the issue?
Requests made through a proxy that responds with content-length header are being dropped. Tested against 2.11.1 and edge-22.5.2.
For unmeshed pods calls with the proxies go through, eg:
➭ curl -lvvs https://www.example.com -x "http://user:[email protected]:10000" > /dev/null
* Trying xxx.xxx.xxx.xxx:10000...
* Connected to xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) port 10000 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.example.com:443
* Proxy auth using Basic with user 'XXXXXXXXXXXXXXXXXX'
> CONNECT www.example.com:443 HTTP/1.1
> Host: www.example.com:443
> Proxy-Authorization: Basic XXXXXXXXXXXXXXXXXXX=
> User-Agent: curl/7.79.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< x-exit-node-ip: xxx.xxx.xxx.xxx
< content-length: 0
* Ignoring Content-Length in CONNECT 200 response
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
(...)
< x-cache: HIT
< content-length: 1256
<
{ [1256 bytes data]
* Connection #0 to host xxx.xxx.xxx.xxx left intact
Proxied calls get dropped though:
curl -lvv https://www.example.com -x "http://user:[email protected]:10000"
* Trying xxx.xxx.xxx.xxx:10000...
* Connected to xxx.xxx.xxx.xxx(xxx.xxx.xxx.xxx) port 10000 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.example.com:443
* Proxy auth using Basic with user 'XXXXXXXXXXXXXXXXXX'
> CONNECT www.example.com:443 HTTP/1.1
> Host: www.example.com:443
> Proxy-Authorization: Basic XXXXXXXXXXXXXXXXXX=
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
>
* Proxy CONNECT aborted
* CONNECT phase completed!
* Closing connection 0
curl: (56) Proxy CONNECT aborted
The proxy logs show:
932.146639s] DEBUG ThreadId(01) outbound:accept{client.addr=192.168.2.188:56340}:server{orig_dst=xxx.xxx.xxx.xxx:10000}:profile:http{v=1.x}:http1: linkerd_proxy_transport::connect: Connected local.addr=192.168.2.188:56342 keepalive=Some(10s)
[ 932.146713s] DEBUG ThreadId(01) outbound:accept{client.addr=192.168.2.188:56340}:server{orig_dst=xxx.xxx.xxx.xxx:10000}:profile:http{v=1.x}:http1: linkerd_transport_metrics::client: client connection open
[ 932.287280s] TRACE ThreadId(01) outbound:accept{client.addr=192.168.2.188:56340}:server{orig_dst=xxx.xxx.xxx.xxx:10000}:profile:http{v=1.x}:http1: linkerd_proxy_http::h1: Client response is HTTP/1.1 upgrade
[ 932.287311s] TRACE ThreadId(01) outbound:accept{client.addr=192.168.2.188:56340}:server{orig_dst=xxx.xxx.xxx.xxx:10000}:profile:http{v=1.x}:http1: linkerd_proxy_http::upgrade: HTTP/1.1 upgrade has both halves
[ 932.287332s] WARN ThreadId(01) outbound:accept{client.addr=192.168.2.188:56340}:server{orig_dst=xxx.xxx.xxx.xxx:10000}:profile:http{v=1.x}: hyper::proto::h1::role: unexpected content-length found, canceling
[ 932.287347s] DEBUG ThreadId(01) outbound:accept{client.addr=192.168.2.188:56340}:server{orig_dst=xxx.xxx.xxx.xxx:10000}:profile:http{v=1.x}: linkerd_proxy_http::server: The client is shutting down the connection res=Err(hyper::Error(User(UnexpectedHeader)))
[ 932.287413s] TRACE ThreadId(01) outbound:accept{client.addr=192.168.2.188:56340}:server{orig_dst=xxx.xxx.xxx.xxx:10000}:profile:http{v=1.x}: linkerd_stack_tracing: drop
[ 932.287444s] INFO ThreadId(01) outbound:accept{client.addr=192.168.2.188:56340}: linkerd_app_core::serve: Connection closed error=user sent unexpected header
[ 932.287453s] TRACE ThreadId(01) outbound:accept{client.addr=192.168.2.188:56340}:server{orig_dst=xxx.xxx.xxx.xxx:10000}: linkerd_stack_tracing: drop
My suspicion is hyper is dropping these in https://github.com/hyperium/hyper/blob/67b73138f110979f3c77ef7b56588f018837e592/src/proto/h1/role.rs#L656-L662 .
How can it be reproduced?
- Make a proxied call to a proxy server that responds with header
content-length. - See errors.
Logs, error output, etc
See above
output of linkerd check -o short
Linkerd core checks
===================
linkerd-identity
----------------
‼ trust anchors are valid for at least 60 days
Anchors expiring soon:
* 1 identity.linkerd. will expire on 2022-07-19T13:44:14Z
see https://linkerd.io/2.11/checks/#l5d-identity-trustAnchors-not-expiring-soon for hints
‼ issuer cert is valid for at least 60 days
issuer certificate will expire on 2022-07-19T13:44:14Z
see https://linkerd.io/2.11/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
linkerd-version
---------------
‼ cli is up-to-date
is running version 2.11.1 but the latest stable version is 2.11.2
see https://linkerd.io/2.11/checks/#l5d-version-cli for hints
control-plane-version
---------------------
‼ control plane is up-to-date
is running version 2.11.1 but the latest stable version is 2.11.2
see https://linkerd.io/2.11/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-5bc8dfd8cb-27crj (stable-2.11.1)
* linkerd-destination-5bc8dfd8cb-4l6zr (stable-2.11.1)
* linkerd-destination-5bc8dfd8cb-5b5t9 (stable-2.11.1)
* linkerd-identity-84df86c455-dfrwc (stable-2.11.1)
* linkerd-identity-84df86c455-hhmmd (stable-2.11.1)
* linkerd-identity-84df86c455-l85c4 (stable-2.11.1)
* linkerd-proxy-injector-767b77445d-dsghx (stable-2.11.1)
* linkerd-proxy-injector-767b77445d-fzgwq (stable-2.11.1)
* linkerd-proxy-injector-767b77445d-h7767 (stable-2.11.1)
see https://linkerd.io/2.11/checks/#l5d-cp-proxy-version for hints
Status check results are √
Linkerd extensions checks
=========================
linkerd-jaeger
--------------
‼ jaeger extension proxies are up-to-date
some proxies are not running the current version:
* collector-569c5599cf-2l7rw (stable-2.11.1)
* jaeger-7bcb4b4cc-dd594 (stable-2.11.1)
* jaeger-injector-6864c98f86-fhj4s (stable-2.11.1)
see https://linkerd.io/2.11/checks/#l5d-jaeger-proxy-cp-version for hints
Status check results are √
linkerd-viz
-----------
‼ viz extension proxies are up-to-date
some proxies are not running the current version:
* grafana-bb7457748-gf4xd (stable-2.11.1)
* metrics-api-c5f6454b6-d77ds (stable-2.11.1)
* prometheus-9749754c7-mjhz8 (stable-2.11.1)
* tap-69c5c56f98-4mn6j (stable-2.11.1)
* tap-69c5c56f98-crxrf (stable-2.11.1)
* tap-69c5c56f98-mmnlb (stable-2.11.1)
* tap-injector-6978c6f49f-tjhrj (stable-2.11.1)
* web-6974d77657-4nh8v (stable-2.11.1)
see https://linkerd.io/2.11/checks/#l5d-viz-proxy-cp-version for hints
Status check results are √
Environment
- Kubernetes version: v1.21.11-gke.900
- linkerd version: stable-2.11.1 (also tested against edge-22.5.2)
Possible solution
- Drop the content-length header before it gets to hyper?
- Fix hyper to ignore content-length head in CONNECT requests, as protocol requires (the server shouldn't send it either).
Additional context
No response
Would you like to work on fixing this bug?
maybe