I'm just starting to dig into this, but I'm filing the bug to raise visibility and hopefully be able to get some help debugging.

What version of Knative?

Both:

0.15.2
0.16.0

but not 0.14.0

Expected Behavior

The memory usage of the autoscaler's Istio proxy stays relatively flat, or at least proportional to some reasonable metric (number of pods, number of ksvcs, ...).

Actual Behavior

Over time the memory usage of the istio proxy container seems to grow over time.

Steps to Reproduce the Problem

On July 8th, I updated our cluster from 0.14.0 to 0.15.2. A about a week later on July 14th, traffic going to Knative services ended up grinding to a halt. In a quick effort to get things running again, I restarted a number of pods, which quickly got things back working. I didn't spend much time looking through logs and metrics as I knew at the time there was a 0.16.0 release available. On July 15th, I updated the cluster to 0.16.0. Just a few hours ago (July 22nd), traffic seized up again.

Looking at the Knative pods I saw:

NAMESPACE                 NAME                                                             READY   STATUS             RESTARTS   AGE
knative-serving           activator-6768988647-npxzd                                       1/2     CrashLoopBackOff   59         4h24m
knative-serving           autoscaler-57cb4c8475-vvtn7                                      1/2     Running            0          5d23h
knative-serving           controller-5448b975d-k7z7q                                       2/2     Running            0          5d12h
knative-serving           istio-webhook-594cc55456-25sfj                                   2/2     Running            0          7d2h
knative-serving           networking-istio-7b9854496d-h6r56                                1/1     Running            0          7d2h
knative-serving           webhook-756b79fb8d-b2q2j                                         2/2     Running            0          7d2h

I noticed that the autoscaler only had one of the two containers running. To quickly fix the problem, the autoscaler pod was deleted, and once the new autoscaler pod was started, traffic began to flow again. I only noticed this now, but the age of the activator-6768988647-npxzd roughly aligns with the start of the outage. This cluster is serving low priority traffic (batch pipelines and whatnot), so the top line monitoring is quite delayed.

The autoscaler-57cb4c8475-vvtn7 pod was suffering the following events:

LAST SEEN   TYPE      REASON      OBJECT                            MESSAGE
20m         Warning   Unhealthy   pod/autoscaler-57cb4c8475-vvtn7   Readiness probe failed: Get http://10.11.9.139:15021/healthz/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Graphs of memory and CPU usage over the last month which show the change in behavior and ramping memory usage in the Istio proxy container:

A few notes about this cluster which might be contributing:

• Traffic coming into the cluster is unreasonably spiky, which results in lots of scale up/scale down behaviors. 1->100->1 in small numbers of minutes, over and over.
• Overall traffic isn't really high. On the order of 1kqps, but the concurrency of several ksvcs is <10 which combined with traffic changes results in lots of pod churn.
• The cluster is running on EKS 1.16, Istio 1.6.4 for most of this.
• Early logs of the activator-6768988647-npxzd has a number of messages like Websocket connection could not be established as well as a bunch of Envoy request logs having the response_code: UH, which means No healthy upstream hosts in upstream cluster in addition to 503 response code I believe.

I'm going to let the current autoscaler run a little and then try to see what metrics/stats/(core?) I can grab off it and try to figure out what might be contributing to the problem.

I am fully aware that the problem is manifesting in the Istio container, and there is a high likelihood that the bug is over there. Any help I can get collecting application specific data, or diagnostics to pin down which system has the problem would be greatly appreciated. I started on this side as the problem didn't manifest with 0.14.0+Istio 1.6.4, and only started once upgrading Knative to 0.15.2 (and later 0.16.0). My guess is that there is something that changed on the Knative side which is having a bad interaction with Istio - just need help finding it.

Feel free to ping me on the Knative slack (nairb774) if some live debugging would help.