What happened:

We have a pod running in our k8s cluster that I connect to via kubectl port-forward. I am able to connect to this pod (using the following command) but then start getting broken pipe error messages after maintaining a connection for what is typically 30-60s.

kubectl port-forward --namespace monitoring deployment/cost-monitor 9090

There seems to be a correlation between error rate and the amount of data being transferred. I see the following error message initially:

E0225 15:20:06.212139 26392 portforward.go:363] error copying from remote stream to local connection: readfrom tcp6 [::1]:9090->[::1]:57794: write tcp6 [::1]:9090->[::1]:57794: write: broken pipe

These errors are oftentimes followed by timeout messages but necessarily immediately:

E0225 15:22:30.454203 26392 portforward.go:353] error creating forwarding stream for port 9090 -> 9090: Timeout occured

What you expected to happen:
No error messages or major degradation in transfer rate.

How to reproduce it (as minimally and precisely as possible):
Connect via port-forward and transfer ~5 mb over several minutes.

What else?:
We have multiple HTTP requests being made at any given time on this port-forward connection.

Environment:
Experiencing on both AWS kops and GKE

// kops
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:25:46Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

// GKE
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-04T04:48:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.11-gke.1", GitCommit:"5c4fddf874319c9825581cc9ab1d0f0cf51e1dc9", GitTreeState:"clean", BuildDate:"2018-11-30T16:18:58Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}