Skip to content

Fix caching in ingress mode#965

Merged
olix0r merged 3 commits intomainfrom
kleimkuhler/logical-caching-fix
Apr 6, 2021
Merged

Fix caching in ingress mode#965
olix0r merged 3 commits intomainfrom
kleimkuhler/logical-caching-fix

Conversation

@kleimkuhler
Copy link
Contributor

When the proxy is in ingress mode, requests with different l5d-dst-override headers but the same address share the same key in the cache. This causes consecutive requests to the same address to all go to the value of the first l5d-dst-override header, ignoring the header values of later requests—unless the services are evicted from the cache.

This occurs because each key into the cache is calculated based off LogicalAddr's orig_dst and protocol fields (source). When the proxy is in ingress mode, the value of orig_dst is overridden to 0.0.0.0:port (source). This results in an easy key conflict: two different requests with the same port and protocol with result in the same key.

To fix this, the logical_addr field is added to LogicalAddr and is used as a third field in calculating the cache key. logical_addr is the value of the l5d-dst-override header which means requests to the same address but with different header values will result in different keys.

Fixes linkerd/linkerd2#5975

Testing

For manual testing, Linkerd can be installed with this version of the proxy:

linkerd install --set proxy.image.name='ghcr.io/kleimkuhler/proxy',proxy.image.version='5381fd45' |kubectl apply -f -

Then, apply this manifest which creates two 3 deployments and 2 services:

  1. injected nginx server and service

  2. injected nginx server and service

  3. injected ingress pod with curl used to send requests to 1 and 2

From the curl pod, send requests with the l5d-dst-override header set and observer with linkerd viz tap that the requests are received by the correct pods:

$ curl-5ffd49f57-fwk2d:/$ curl -H "l5d-dst-override: svc-1.default.svc.cluster.local:8000" 10.0.0.1
$ curl-5ffd469f57-fwk2d:/$ curl -H "l5d-dst-override: svc-2.default.svc.cluster.local:8000" 10.0.0.1

Proxy tests

There are currently no tests for the l5d-dst-override header. I'll keep working on adding some additional tests for this, but those may come as a follow-up

Signed-off-by: Kevin Leimkuhler <[email protected]>
@kleimkuhler kleimkuhler requested a review from a team April 6, 2021 01:13
Copy link
Contributor

@hawkw hawkw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me!

for the tests, it might be nice to have a function for constructing Logicals to make that code a little more concise, but that's not a hard blocker.

Signed-off-by: Kevin Leimkuhler <[email protected]>
@kleimkuhler
Copy link
Contributor Author

I may rename logical_addr field to addr. It feels a little redundant to to have a LogicalAddr struct with a logical_addr field. Any strong preference about keeping or changing it?

Signed-off-by: Kevin Leimkuhler <[email protected]>
Copy link
Member

@olix0r olix0r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @kleimkuhler!

@olix0r olix0r merged commit 74a1632 into main Apr 6, 2021
@olix0r olix0r deleted the kleimkuhler/logical-caching-fix branch April 6, 2021 15:02
olix0r added a commit to linkerd/linkerd2 that referenced this pull request Apr 7, 2021
This release fixes a caching issue in the outbound proxy's "ingress
mode" that could cause the incorrect client to be used for requests.
This caching has been fixed so that clients cannot be incorrectly reused
across logical destinations.

---

* transport: introduce Bind trait and move orig dst to the type level (linkerd/linkerd2-proxy#957)
* Initial fuzzer integration (linkerd/linkerd2-proxy#961)
* Fix caching in ingress mode (linkerd/linkerd2-proxy#965)
olix0r added a commit to linkerd/linkerd2 that referenced this pull request Apr 7, 2021
This release fixes a caching issue in the outbound proxy's "ingress
mode" that could cause the incorrect client to be used for requests.
This caching has been fixed so that clients cannot be incorrectly reused
across logical destinations.

---

* transport: introduce Bind trait and move orig dst to the type level (linkerd/linkerd2-proxy#957)
* Initial fuzzer integration (linkerd/linkerd2-proxy#961)
* Fix caching in ingress mode (linkerd/linkerd2-proxy#965)
Pothulapati pushed a commit to linkerd/linkerd2 that referenced this pull request Apr 14, 2021
This release fixes a caching issue in the outbound proxy's "ingress
mode" that could cause the incorrect client to be used for requests.
This caching has been fixed so that clients cannot be incorrectly reused
across logical destinations.

---

* transport: introduce Bind trait and move orig dst to the type level (linkerd/linkerd2-proxy#957)
* Initial fuzzer integration (linkerd/linkerd2-proxy#961)
* Fix caching in ingress mode (linkerd/linkerd2-proxy#965)
jijeesh pushed a commit to jijeesh/linkerd2 that referenced this pull request Apr 21, 2021
This release fixes a caching issue in the outbound proxy's "ingress
mode" that could cause the incorrect client to be used for requests.
This caching has been fixed so that clients cannot be incorrectly reused
across logical destinations.

---

* transport: introduce Bind trait and move orig dst to the type level (linkerd/linkerd2-proxy#957)
* Initial fuzzer integration (linkerd/linkerd2-proxy#961)
* Fix caching in ingress mode (linkerd/linkerd2-proxy#965)

Signed-off-by: Jijeesh <[email protected]>
kleimkuhler pushed a commit to linkerd/linkerd2 that referenced this pull request May 12, 2021
This release fixes a caching issue in the outbound proxy's "ingress
mode" that could cause the incorrect client to be used for requests.
This caching has been fixed so that clients cannot be incorrectly reused
across logical destinations.

---

* transport: introduce Bind trait and move orig dst to the type level (linkerd/linkerd2-proxy#957)
* Initial fuzzer integration (linkerd/linkerd2-proxy#961)
* Fix caching in ingress mode (linkerd/linkerd2-proxy#965)
kleimkuhler pushed a commit to linkerd/linkerd2 that referenced this pull request May 12, 2021
This release fixes a caching issue in the outbound proxy's "ingress
mode" that could cause the incorrect client to be used for requests.
This caching has been fixed so that clients cannot be incorrectly reused
across logical destinations.

---

* transport: introduce Bind trait and move orig dst to the type level (linkerd/linkerd2-proxy#957)
* Initial fuzzer integration (linkerd/linkerd2-proxy#961)
* Fix caching in ingress mode (linkerd/linkerd2-proxy#965)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Linkerd-proxy forwards traffic to the wrong Pod

3 participants