Skip to content

bpf: Introduce wildcard service entries to drop traffic towards unknown protocol/port combinations - #40684

Merged
borkmann merged 8 commits into
cilium:mainfrom
ajmmm:pr/slb-drop-wildcard-vip
Aug 27, 2025
Merged

borkmann merged 8 commits into
cilium:mainfrom
ajmmm:pr/slb-drop-wildcard-vip

Conversation

@ajmmm

@ajmmm ajmmm commented Jul 23, 2025 •

Copy link
Copy Markdown
Member

PR to resolve a forwarding bug in data path.

This work is derived from work undertaken by Mikael Knutsson [email protected] so has been credited in the relevant commit logs.

Summary of issue

Given a configured LoadBalancer or Cluster IP on a specific L4 destination port and protocol, ingress traffic towards the IP on an alternate destination port or protocol will not match eBPF service map lookup, so traffic be punted back to the network. Where the upstream network re-delivers the traffic back to Cilium, a packet loop is formed.

Summary of Changes

  • Introduces the concept of "wildcard" service entries in eBPF Service Maps, being a service entry with a zero destination and protocol field.
  • LoadBalancer and ClusterIP frontend services with external scope will be cross-referenced to a wildcard entry:
    • The first reference will trigger addition of wildcard entry in eBPF
    • The last reference deleted will trigger removal of wildcard entry from eBPF
  • Data path behaviour modified on traffic ingress. Where service lookup fails, an additional lookup is undertaken with zero destination and protocol. On successful match to such a wildcard entry, traffic is dropped.
  • Removal of data path logic that maintained compatibility with eBPF maps programmed by older versions of Cilium. Protocol differentiation was deprecated in 1.17, and automatic conversion of undifferentiated protocol was added in 1.18.
  • Appropriate unit test updates for the new logic
  • Minor typo fixes in unit test logic

Release Notes:

Introduce wildcard service entries to ensure traffic towards a LoadBalancer and ClusterIPs with an unknown protocol/port combination is dropped by the data path, rather than being forwarded back to the network.

@maintainer-s-little-helper maintainer-s-little-helper Bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Jul 23, 2025
@github-actions github-actions Bot added the kind/community-contribution This was a contribution made by a community member. label Jul 23, 2025
@ajmmm
ajmmm force-pushed the pr/slb-drop-wildcard-vip branch 12 times, most recently from bd5a7e7 to f5a6f60 Compare July 29, 2025 14:01
@ajmmm
ajmmm marked this pull request as ready for review July 29, 2025 14:31
@ajmmm
ajmmm requested review from a team as code owners July 29, 2025 14:31
@ajmmm
ajmmm requested a review from dylandreimerink July 29, 2025 14:31
@ajmmm
ajmmm force-pushed the pr/slb-drop-wildcard-vip branch from fb75646 to adf633a Compare July 29, 2025 16:02
@ajmmm
ajmmm requested a review from joamaki July 29, 2025 16:10
@ajmmm
ajmmm force-pushed the pr/slb-drop-wildcard-vip branch from adf633a to c391c35 Compare July 31, 2025 08:50
@ajmmm
ajmmm force-pushed the pr/slb-drop-wildcard-vip branch 4 times, most recently from 8b5e388 to f408d79 Compare August 1, 2025 08:42

@dylandreimerink dylandreimerink left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great overall. One point of feedback though.

Comment thread bpf/lib/nodeport.h Resolved
Comment thread bpf/lib/nodeport.h Resolved
@ajmmm
ajmmm force-pushed the pr/slb-drop-wildcard-vip branch from f408d79 to 6b2266c Compare August 1, 2025 11:27
ajmmm added 6 commits August 27, 2025 10:28
…paths.

Previously, lbX_lookup_service() would zeroise the key protocol field,
such that callers would	need to	reinstate the key protocol field if their
logic re-used the key.

Following the previous commit, the lbX_lookup_service()	routines will now
only zeroise the key protocol (and dport) fields if an external	wild-card
entry is returned. Thus, the restoration of key values can be removed.

This commit also removes lbX_key_set_protocol() helpers as they became
broadly unnecessary.

Signed-off-by: Alasdair McWilliam <[email protected]>
This set of tests verifies that if we receive a packet towards a VIP on either
an unknown destination port, or unknown protocol, the data plane will correctly
drop if a wildcard entry is present in the service map.

Signed-off-by: Alasdair McWilliam <[email protected]>
This change extends the nodePortAddrKey structure to include the L4 Protocol
number of an address, such that addresses with differing addresses will be
better distributed in the nodePortAddrByPort map.

This required the introduction of a helper routine to translate the L4Addr
protocol field into the real IANA number.

Signed-off-by: Alasdair McWilliam <[email protected]>
Introduce logic in bpf_reconciler to map frontend service entries to wildcard
service entries, and program them in the data path, for LoadBalancer and ClusterIP
frontends of external scope. This will ensure the data path drops traffic towards
unknown destination ports and/or protocols.

Wildcard state is tracked as a map (indexed by IP Address) that contains a slice
of Frontend ServiceIDs. The first entry that requires a wildcard will trigger
programming of the data path entry. The last entry that depends on a wildcard
entry to be removed will trigger removal of the wildcard entry.

Signed-off-by: Alasdair McWilliam <[email protected]>
Signed-off-by: Mikael Knutsson <[email protected]>
Update the unit tests for loadbalancer and ciliumenvoyconfig to include
wildcard entries.

Signed-off-by: Alasdair McWilliam <[email protected]>
Remove lb_v4_add_mixed_proto_service_with_flags() as it no longer makes sense.

Signed-off-by: Alasdair McWilliam <[email protected]>
@ajmmm
ajmmm force-pushed the pr/slb-drop-wildcard-vip branch from fe966e5 to abf0fec Compare August 27, 2025 09:28
@ajmmm

ajmmm commented Aug 27, 2025

Copy link
Copy Markdown
Member Author

/test

@ajmmm

ajmmm commented Aug 27, 2025

Copy link
Copy Markdown
Member Author

ci-ginkgo failures is likely a flake - happens across different tests on main, eg:

https://github.com/cilium/cilium/actions/workflows/conformance-ginkgo.yaml?query=is%3Afailure+branch%3Amain

@borkmann
borkmann merged commit 3091b2e into cilium:main Aug 27, 2025
@pippolo84 pippolo84 mentioned this pull request Sep 24, 2025
9 of 17 tasks
@ajmmm
ajmmm deleted the pr/slb-drop-wildcard-vip branch September 29, 2025 10:26
Comment thread bpf/lib/nodeport.h Resolved
@rastislavs rastislavs mentioned this pull request Nov 3, 2025
9 of 12 tasks
@cilium-release-bot cilium-release-bot Bot moved this to Released in cilium v1.19.0 Feb 3, 2026
lllamnyp added a commit to cozystack/cozystack that referenced this pull request Apr 8, 2026
)

## What this PR does

Adds the `service.kubernetes.io/service-proxy-name: "cozy-proxy"` label
to VM LoadBalancer services when `external: true`. This tells Cilium to
completely ignore the service — no BPF entries, no DNAT, no wildcard
drops.

Routing continues to work through kube-ovn
(`networking.cozystack.io/wholeIP` annotation) and MetalLB L2
advertisement is unaffected.

**This fixes two issues:**

1. **Inter-tenant connectivity via public LB IPs**: With
`kube-proxy-replacement: true`, Cilium DNATs traffic destined to LB IPs
(on ports declared in the Service) to the backend pod IP *before*
evaluating network policies. The `CiliumClusterwideNetworkPolicy` then
sees a cross-tenant pod-to-pod flow and blocks it. Ports not in the
Service pass as "world" traffic and work fine — so does ICMP. With this
label, Cilium doesn't DNAT anything, all traffic is classified as
"world", and the existing `allow-external-communication` policy allows
it. Pod IP isolation between tenants is preserved.

2. **WholeIP broken on Cilium 1.19+** (#2327): Cilium 1.19 introduced
wildcard service drop entries (`cilium/cilium#40684`) that drop all
traffic to a LB IP on ports not declared in the Service, before it
reaches netfilter/cozy-proxy. With this label, no BPF entries are
created at all, so there's nothing to drop.

**Tested on our lab cluster:**
- Cilium 1.18.6: inter-tenant via LB IP ✅, pod IP blocked ✅, external
SSH ✅, WholeIP all ports ✅
- Cilium 1.19.1: same results ✅, no wildcard drops ✅
- MetalLB L2 advertisement: unaffected ✅
- Other services without the label: unaffected ✅

**Note:** The `externalPorts` / PortList filtering is currently not
enforced at the network level (the `wholeIP` annotation causes kube-ovn
to route all ports regardless). This was the case before this change and
is a separate concern — per-VM CiliumNetworkPolicies could be
re-introduced for that (ref: `4cbf562a`, #1601).

```release-note
[virtual-machine] Fix inter-tenant connectivity via public LB IPs and WholeIP on Cilium 1.19+
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/community-contribution This was a contribution made by a community member. release-note/minor This PR changes functionality that users may find relevant to operating Cilium.

Projects

No open projects
Status: Released

Development

Successfully merging this pull request may close these issues.

9 participants