Is there an existing issue for this?
What happened?
We run Cilium on AWS in ipam=eni mode and we isolate pod interfaces from the main node one (using "first-interface-index": 1). At high throughput, we noticed a significant amount of TCP retransmits that we tracked back to qdic drops (by looking at the output of tc -s qdisc show dev ens6). When we looked more closely we noticed that:
- we have one qdisc per tx queue on the ENI (expected)
- all traffic from pods use the first qdisc and the first tx queue of the ENI (unexpected and this limits the maximum throughput we can achieve)
We performed a few tests to reproduce and isolate the problem and here is what we found:
- host network traffic using node interface (ens5): all qdisc / tx queues are used
- host network traffic using pod interface (ens6): all qdisc / tx queues are used (to achieve this we added an available pod IP to ens6 directly and forced outgoing traffic to use it for a test destination using
ip rule to X lookup 11
- pod traffic, routed through ens6: only first qdisc / first queue used
- manually created netns, traffic routed through ens6: only first qdisc / first queue used (expected of course as we used the same routing strategy as cilium)
- manually created netns, traffic routed through ens5: only first qdisc / first queue used (we allocated an additional IP to the ens5 ENI and added the necessary routing)
We performed the last 2 tests again but this time we created the veth with numtxqueues=4 on a node with 2 tx queues and this solved the issue (all qdisc / tx queues were used)
It looks like increasing rx/tx queues on veth devices will solve the issue and this is a pretty simple fix. However, I'm really not certain this is right answer though because we also run on GCP (with ipam=kubernetes and secondary cidrs for pods) and pod traffic is using all qdisc and tx queues (I checked on a node with 16 tx queues / qdisc and traffic is balanced across all of them).
On both AWS and GCP, root qdisc is mq and we have has many child ones as tx queues (as expected) using fq_codel (the Ubuntu default)
Cilium Version
1.8.7
Kernel Version
5.11.0-1022-aws and Ubuntu 20.04.1 on AWS
5.11.0-1020-gcp and Ubuntu 20.04.1 on GCP
Kubernetes Version
1.19.1
Sysdump
No response
Relevant log output
No response
Anything else?
No response
Code of Conduct
Is there an existing issue for this?
What happened?
We run Cilium on AWS in
ipam=enimode and we isolate pod interfaces from the main node one (using"first-interface-index": 1). At high throughput, we noticed a significant amount of TCP retransmits that we tracked back to qdic drops (by looking at the output oftc -s qdisc show dev ens6). When we looked more closely we noticed that:We performed a few tests to reproduce and isolate the problem and here is what we found:
ip rule to X lookup 11We performed the last 2 tests again but this time we created the veth with numtxqueues=4 on a node with 2 tx queues and this solved the issue (all qdisc / tx queues were used)
It looks like increasing rx/tx queues on veth devices will solve the issue and this is a pretty simple fix. However, I'm really not certain this is right answer though because we also run on GCP (with ipam=kubernetes and secondary cidrs for pods) and pod traffic is using all qdisc and tx queues (I checked on a node with 16 tx queues / qdisc and traffic is balanced across all of them).
On both AWS and GCP, root qdisc is mq and we have has many child ones as tx queues (as expected) using fq_codel (the Ubuntu default)
Cilium Version
1.8.7
Kernel Version
5.11.0-1022-aws and Ubuntu 20.04.1 on AWS
5.11.0-1020-gcp and Ubuntu 20.04.1 on GCP
Kubernetes Version
1.19.1
Sysdump
No response
Relevant log output
No response
Anything else?
No response
Code of Conduct