Bug description
A program that makes occasional requests to services outside of the mesh combined with the cluster having outboundTrafficPolicy:REGISTRY_ONLY set will result in the Envoy process accumulating an unbounded number of stats entries (? stuff available on /stats) eventually causing Envoy to consume lots of ram and in some cases stops functioning all together.
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[X] Networking
[X] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Ram/CPU stays bounded and Envoy remains functional
Steps to reproduce the bug
Important context: Istio has outboundTrafficPolicy:REGISTRY_ONLY set.
The Knative autoscaler acquired a feature which attempts to scrape statistics directly from individual pods. This direct scraping happens on autoscaler startup, and on occasion when some internal state gets reset. This results in periodic bursts of 1-10ish requests being attempted to individual pods to grab stats. After the autoscaler attempts to make these fetches, and finds out it isn't possible, it proceeds to scrape via the service (good!). As pods move around, and the autoscaler's internal state gets reset, it goes back to attempting to talk to the pods directly. This causes the number of /stats entries in Envoy to continuously grow because each attempt to scrape a pod results in an attempt to contact a unique IP. Eventually Envoy struggles to operate correctly, and seems to either lock up or becomes so slow as to be non-functional. With the Knative autoscaler it seems to take roughly a week to cause full breakdown of Envoy. The direct pod probes happen at a rate of about 10ish every 15 minutes (really slow burn). Lots of details in knative/serving#8761 for the curious.
https://github.com/nairb774/experimental/blob/28f76fcc2db73ed30c46aff7ce4b25a47515d25c/http-prober/main.go is a simplified and accelerated reproduction of the behavior of the Knative autoscaler component. This can be deployed/run with ko to simulate a simplified Knative autoscaler behavior. Within a minute or two of running the http-prober, the /stats/prometheus page, on the Envoy dashboard (istioctl dashboard envoy), takes forever to return, and the response size is well over 100MiB (in the one test I ran) with about 140k rows. The memory usage of Envoy also ballooned to about 500MiB in that minute time.
Ideally the slow burn version of the Knative autoscaler wouldn't cause Envoy to topple over (though it is just as easy to say the autoscaler is as much at fault). It doesn't look like the blackhole metrics get garbage collected if they sit idle for a long time.
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
$ istioctl version --remote
client version: 1.6.5
control plane version: 1.6.4
data plane version: 1.6.4 (57 proxies)
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.8-eks-e16311", GitCommit:"e163110a04dcb2f39c3325af96d019b4925419eb", GitTreeState:"clean", BuildDate:"2020-03-27T22:37:12Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Knative v0.16.0 (for what it is worth)
How was Istio installed?
Istio Operator (version 1.6.4)
Environment where bug was observed (cloud vendor, OS, etc)
AWS EKS 1.16
AWS Managed Node Groups (1.16.8-20200618 / ami-0d960646974cf9e5b)
Here is an operator config for Istio: istio-system.yaml.gz which is from the cluster exhibiting the issue both via the Knative autoscaler as well as the minimal program above. I'm a little apprehensive dumping so much information from the cluster and making it publically accessible. I'm happy to pull specific logs/info that might be useful. I'm even happy to hop on a VC and do some live debugging/reproductions if that would help out.
Bug description
A program that makes occasional requests to services outside of the mesh combined with the cluster having
outboundTrafficPolicy:REGISTRY_ONLYset will result in the Envoy process accumulating an unbounded number of stats entries (? stuff available on /stats) eventually causing Envoy to consume lots of ram and in some cases stops functioning all together.[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[X] Networking
[X] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Ram/CPU stays bounded and Envoy remains functional
Steps to reproduce the bug
Important context: Istio has
outboundTrafficPolicy:REGISTRY_ONLYset.The Knative autoscaler acquired a feature which attempts to scrape statistics directly from individual pods. This direct scraping happens on autoscaler startup, and on occasion when some internal state gets reset. This results in periodic bursts of 1-10ish requests being attempted to individual pods to grab stats. After the autoscaler attempts to make these fetches, and finds out it isn't possible, it proceeds to scrape via the service (good!). As pods move around, and the autoscaler's internal state gets reset, it goes back to attempting to talk to the pods directly. This causes the number of /stats entries in Envoy to continuously grow because each attempt to scrape a pod results in an attempt to contact a unique IP. Eventually Envoy struggles to operate correctly, and seems to either lock up or becomes so slow as to be non-functional. With the Knative autoscaler it seems to take roughly a week to cause full breakdown of Envoy. The direct pod probes happen at a rate of about 10ish every 15 minutes (really slow burn). Lots of details in knative/serving#8761 for the curious.
https://github.com/nairb774/experimental/blob/28f76fcc2db73ed30c46aff7ce4b25a47515d25c/http-prober/main.go is a simplified and accelerated reproduction of the behavior of the Knative autoscaler component. This can be deployed/run with
koto simulate a simplified Knative autoscaler behavior. Within a minute or two of running the http-prober, the/stats/prometheuspage, on the Envoy dashboard (istioctl dashboard envoy), takes forever to return, and the response size is well over 100MiB (in the one test I ran) with about 140k rows. The memory usage of Envoy also ballooned to about 500MiB in that minute time.Ideally the slow burn version of the Knative autoscaler wouldn't cause Envoy to topple over (though it is just as easy to say the autoscaler is as much at fault). It doesn't look like the blackhole metrics get garbage collected if they sit idle for a long time.
Version (include the output of
istioctl version --remoteandkubectl versionandhelm versionif you used Helm)Knative v0.16.0 (for what it is worth)
How was Istio installed?
Istio Operator (version 1.6.4)
Environment where bug was observed (cloud vendor, OS, etc)
AWS EKS 1.16
AWS Managed Node Groups (1.16.8-20200618 / ami-0d960646974cf9e5b)
Here is an operator config for Istio: istio-system.yaml.gz which is from the cluster exhibiting the issue both via the Knative autoscaler as well as the minimal program above. I'm a little apprehensive dumping so much information from the cluster and making it publically accessible. I'm happy to pull specific logs/info that might be useful. I'm even happy to hop on a VC and do some live debugging/reproductions if that would help out.