since controller 1.10.0 (chart 4.10.0): ingress rejects duplicate "Transfer-Encoding: chunked" and returns 502 #11162
Description
TLDR: controller 1.10.0 updated to a more recent nginx that started to reject certain behaviour from downstream systems, e.g. transfering 2x Transfere-Encoding: Chunked headers.
as per this:
... as the upstream response in question is obviously invalid and should never be accepted in the first place, and the change is more or less a minor cleanup work.
There is no setting in ingress-nginx that suppresses this behaviour
possible next steps:
-
fix the offending downstream systems so that they do not lead to nginx rejecting thje response
-
raise an issue with the nginx project
-
possibly in conjunction with 2), add an option that may be able to suppress the behaviour when there is feedback from the nginx project
The issue has been closed as I, the submitter, went for option 1), as the stance from the nginx project has been rather clear.
What happened: when controller from 1.9.6 (chart 4.9.1) to 1.10.0 (chart 4.10.0), we observe ingress answering 502 on behalve of the service. When rolling back to 1.9.6, restored to healthty behaviour
What you expected to happen: ingress behaving similar to 1.9.6 and not answering 502 where it did not before
there appears to be a regression of some sort. We have updated from 1.9.6 to 1.10.0 and observed that some requests return 502 right when we upgraded. We then downgraded and saw the 502s drop to previous numbers.
One instance where we observe it is when setting Transfer-Encoding: chunked twice, once in code and once via Spring Boot.
We also observe following error message in the logs
ingress-nginx-controller-8bf5b5f98-w8gbz 2024/03/22 11:01:54 [error] 575#575: *28680 upstream sent duplicate header line: "Transfer-Encoding: chunked", previous value: "Transfer-Encoding: chunked" while reading response header from upstream, client: 127.0.0.1, server: localhost, request: "POST /retrieve HTTP/1.1", upstream: "http://10.26.195.10:8080/retrieve", host: "localhost:8076"
ingress-nginx-controller-8bf5b5f98-w8gbz 127.0.0.1 - - [22/Mar/2024:11:01:54 +0000] "POST /retrieve HTTP/1.1" 502 150 "-" "PostmanRuntime/7.36.3" 3208 2.317 [default-ebilling-retrieval-http] ] 10.26.195.10:8080 0 2.318 502 94bc05d81342c91791fac0f02cb64434
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): 1.25.3
/etc/nginx $ /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.10.0
Build: 71f78d49f0a496c31d4c19f095469f3f23900f8a
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.3
-------------------------------------------------------------------------------
/etc/nginx $
Kubernetes version (use kubectl version): v1.28.5
Environment:
- Cloud provider or hardware configuration: Azure AKS
- OS (e.g. from /etc/os-release): Ubuntu on nodes
/etc/nginx $ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.19.1
PRETTY_NAME="Alpine Linux v3.19"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
/etc/nginx $
- Kernel (e.g.
uname -a):
/etc/nginx $ uname -a
Linux ingress-nginx-controller-8bf5b5f98-w8gbz 5.15.0-1057-azure #65-Ubuntu SMP Fri Feb 9 18:39:24 UTC 2024 x86_64 Linux
/etc/nginx $
- Install tools: AKS via terraform, ingress via helm (but also via terraform)
Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
- Basic cluster related info:
kubectl version
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVers
ion:"go1.20.5", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.5", GitCommit:"506050d61cf291218dfbd41ac93913945c9aa0da", GitTreeState:"clean", BuildDate:"2023-12-23T00:10:25Z", GoVers
ion:"go1.20.12", Compiler:"gc", Platform:"linux/amd64"}
kubectl get nodes -o wide
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
aks-default-15207550-vmss000000 Ready agent 7d2h v1.28.5 10.26.194.4 <none> Ubuntu 22.04.4 LTS 5.15.0-1057-azure containerd://1.7.7-1
aks-pool001-49772321-vmss000000 Ready agent 7d1h v1.28.5 10.26.195.9 <none> Ubuntu 22.04.4 LTS 5.15.0-1057-azure containerd://1.7.7-1
aks-pool001-49772321-vmss000001 Ready agent 7d1h v1.28.5 10.26.194.207 <none> Ubuntu 22.04.4 LTS 5.15.0-1057-azure containerd://1.7.7-1
aks-pool001-49772321-vmss00000b Ready agent 7d1h v1.28.5 10.26.194.33 <none> Ubuntu 22.04.4 LTS 5.15.0-1057-azure containerd://1.7.7-1
aks-pool002-37360131-vmss00000h Ready agent 7d v1.28.5 10.26.194.91 <none> Ubuntu 22.04.4 LTS 5.15.0-1057-azure containerd://1.7.7-1
aks-pool002-37360131-vmss00000q Ready agent 7d v1.28.5 10.26.194.120 <none> Ubuntu 22.04.4 LTS 5.15.0-1057-azure containerd://1.7.7-1
aks-pool002-37360131-vmss00000v Ready agent 7d v1.28.5 10.26.194.149 <none> Ubuntu 22.04.4 LTS 5.15.0-1057-azure containerd://1.7.7-1
- How was the ingress-nginx-controller installed:
- If helm was used then please show output of
helm ls -A | grep -i ingress
- If helm was used then please show output of
$ helm ls -A | grep -i ingress
ingress-nginx ap-system 1 2024-03-21 08:44:29.913568481 +0000 UTC deployed ingress-nginx-4.10.0
- If helm was used then please show output of
helm -n <ingresscontrollernamespace> get values <helmreleasename>
$ helm -n ap-system get values ingress-nginx
USER-SUPPLIED VALUES:
controller:
ingressClassResource:
name: nginx
service:
type: ClusterIP
-
If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
-
if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances
-
Current State of the controller:
kubectl describe ingressclasses
$ kubectl describe ingressclasses
Name: nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.10.0
helm.sh/chart=ingress-nginx-4.10.0
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ap-system
Controller: k8s.io/ingress-nginx
Events: <none>
-
kubectl -n <ingresscontrollernamespace> get all -A -o wide -
kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname> -
kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename> -
Current state of ingress object, if applicable:
kubectl -n <appnamespace> get all,ing -o widekubectl -n <appnamespace> describe ing <ingressname>- If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag
-
Others:
- Any other related information like ;
- copy/paste of the snippet (if applicable)
kubectl describe ...of any custom configmap(s) created and in use- Any other related information that may help
- Any other related information like ;
How to reproduce this issue:
we have a service that somestimes anwers with 2x Transfer-Encoding: chunked , sometimes with 1x Transfer-Encoding: chunked, when it answers with 2x the header, the request does not reach the client and ingress answers with 502
we hook into the ingress via port-forward to reproduce it, but have no better setup for local reproducing
kubectl port-forward -n ap-system ingress-nginx-controller-8bf5b5f98-w8gbz 8076 8080
-
<send requests via postman> -
observe 502 & log when pod answers with duplicate header
Transfer-Encoding: chunked, regular 200 when answering with one header
We have also observed this for other clients with other backends, but so far have only particular endpoint reproduced with "always errors when X and always ok when Y"
Anything else we need to know:
we now have deployed a seperate ingress @ 1.10.0 (with seperate ingress class) and can observe this behaviour while our "hot" ingress that gets traffik is back to 1.9.6, the 1.10.0 still breaks while we have an operational ingress with 1.9.6. It does sound somewhat similar to spring-projects/spring-boot#37646