Skip to content

L2Announcement with externalTrafficPolicy: Local #27800

New issue
New issue
Closed
Bug
#46399
Closed
L2Announcement with externalTrafficPolicy: Local#27800
Bug
#46399
Labels
area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.feature/l2-announcementkind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.pinnedThese issues are not marked stale by our issue bot.

Description

@ansarhun
ansarhun
opened on Aug 29, 2023

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

I have

  • a CiliumL2AnnouncementPolicy configured to expose 2 LoadBalancer type services
  • 2 Service one with externalTrafficPolicy: Local the other externalTrafficPolicy: Cluster

The service with the Cluster type works as expected.
The service with the Local type only works if I force the pod to the same node which have the lease for the given service.

I could see that the announcement worked and the node with the lease was exposed but if the pod is not running on that node with the lease it will not handle the traffic.

Cilium Version

cilium-cli: v0.15.6 compiled with go1.21.0 on darwin/arm64
cilium image (default): v1.14.0
cilium image (stable): v1.14.1
cilium image (running): 1.14.1

Kernel Version

Linux k3s-worker-4 6.1.0-11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4 (2023-08-08) x86_64 GNU/Linux

Kubernetes Version

Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.5+k0s

Sysdump

No response

Relevant log output

No response

Anything else?

When pod is on a different node:

❯ curl -vvvvvvv -k http://192.168.2.110:8443
*   Trying 192.168.2.110:8443...
* connect to 192.168.2.110 port 8443 failed: Operation timed out
* Failed to connect to 192.168.2.110 port 8443 after 75006 ms: Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to 192.168.2.110 port 8443 after 75006 ms: Couldn't connect to server

When pod is running on the same node:

❯ curl -vvvvvvv -k http://192.168.2.110:8443
*   Trying 192.168.2.110:8443...
* Connected to 192.168.2.110 (192.168.2.110) port 8443 (#0)
> GET / HTTP/1.1
> Host: 192.168.2.110:8443
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 400
< Content-Type: text/plain;charset=UTF-8
< Connection: close
<

Code of Conduct

  • I agree to follow this project's Code of Conduct

Activity

  1. ansarhun
    added
    kind/bugThis is a bug in the Cilium logic.
    kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.
    needs/triageThis issue requires triaging to establish severity and affected area.
    on Aug 29, 2023

  2. aditighag commented on Aug 29, 2023

    @aditighag
    aditighag
    on Aug 29, 2023
    Member

    I could see that the announcement worked and the node with the lease was exposed but if the pod is not running on that node with the lease it will not handle the traffic.

    I wonder if that's expected with externalTrafficPolicy: Local. /cc @dylandreimerink

  3. aditighag
    added
    feature/l2-announcement
    on Aug 29, 2023

  4. dylandreimerink commented on Aug 30, 2023

    @dylandreimerink
    dylandreimerink
    on Aug 30, 2023
    Member

    Hmm. So the L2Announcement feature doesn't "implement" externalTrafficPolicy as in, we ignore the setting at the present. That means that we might announce the IP from a node that doesn't have a pod for the service, thus violating the wish expressed with externalTrafficPolicy.

    The feature relies on the generic service LB functionality to forward traffic received for a given IP. It makes sense to me that when this field is set, any node with a local endpoint will prefer to forward to the local endpoint, but I do not know what is supposed to happen when traffic for a service reaches a node without local endpoint. This seems to indicate we drop the traffic in that case.

    Most of the groundwork for implementing this feature has already been laid by #25477 so it should not be difficult to also implement this for L2 Announcements.

  5. dylandreimerink
    removed
    needs/triageThis issue requires triaging to establish severity and affected area.
    on Aug 30, 2023

  6. mccasian commented on Sep 3, 2023

    @mccasian
    mccasian
    on Sep 3, 2023

    I also want this, since I am struggling with the same issue.

  7. youngnick
    added
    area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.
    on Sep 4, 2023

  8. github-actions commented on Nov 4, 2023

    @github-actions
    github-actionsBot
    on Nov 4, 2023 – with GitHub Actions

    This issue has been automatically marked as stale because it has not
    had recent activity. It will be closed if no further activity occurs.

  9. github-actions
    added
    staleThe stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
    on Nov 4, 2023

  10. ansarhun commented on Nov 6, 2023

    @ansarhun
    ansarhun
    on Nov 6, 2023
    Author

    As this issue is still present should I update something so it's not closed by stale bot?

  11. dylandreimerink
    added
    pinnedThese issues are not marked stale by our issue bot.
    and removed
    staleThe stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
    on Nov 6, 2023

  12. dylandreimerink commented on Nov 6, 2023

    @dylandreimerink
    dylandreimerink
    on Nov 6, 2023
    Member

    Marked it as pinned so we don't have to keep it manually alive

  13. 29 remaining items

  14. mhkarimi1383 commented on Jun 17, 2026

    @mhkarimi1383
    mhkarimi1383
    on Jun 17, 2026

    Okay, I will try to reproduce and raise issue if needed

  15. yuito-it commented on Jun 17, 2026

    @yuito-it
    yuito-it
    on Jun 17, 2026
    Contributor

    @mhkarimi1383

    I think we have the same problem with BGP, @yuito-it will your Fix solve it?

    The answer is NO.
    I can't test BGP because my environment can only use L2, so I fixed the issue regarding L2Announcement.

  16. yuito-it
    added a commit that references this issue on Aug 4, 2026

    loadbalancer: Add support for external traffic policy to l2 announcement

    4f4bd6f
  17. IonBlaster77
    added a commit that references this issue on Aug 4, 2026

    Work around cilium/cilium#31689 L2 announcement bug with node split

    9636850
  18. yuito-it
    added 2 commits that reference this issue on Aug 18, 2026

    loadbalancer: Add support for external traffic policy to l2 announcement

    88383e3

    loadbalancer: Add support for external traffic policy to l2 announcement

    af05d8c
  19. qnimbus
    added a commit that references this issue on Aug 24, 2026

    fix(media): drop externalTrafficPolicy: Local, Cilium L2 incompatible

    09a84bc
  20. dylandreimerink
    closed this as completedin #46399on Sep 4, 2026
  21. pull
    added a commit that references this issue on Sep 4, 2026

    loadbalancer: Add support for external traffic policy to l2 announcement

    1ba263d

  22. yuito-it commented on Sep 4, 2026

    @yuito-it
    yuito-it
    on Sep 4, 2026
    Contributor

    Hi everyone, I finished fix it!!!!!!
    I'm so proud! I wish to increase the number of L2 announcements users.

  23. mccasian commented on Sep 4, 2026

    @mccasian
    mccasian
    on Sep 4, 2026

    fantastic work, thank you. Trust me, many many maaaaaaany users that will not write anything here will be thankful that this is now working

  24. Syntax3rror404 commented on Sep 6, 2026

    @Syntax3rror404
    Syntax3rror404
    on Sep 6, 2026

    Cilium L2 is now finally usable @yuito-it thank you :)

  25. gandro
    added a commit that references this issue on Sep 8, 2026

    loadbalancer: Add support for external traffic policy to l2 announcement

    b5157d5
  26. nobiit
    added a commit that references this issue on Sep 10, 2026

    loadbalancer: Add support for external traffic policy to l2 announcement

    84f51ba
  27. sypakine
    added a commit that references this issue on Sep 25, 2026

    loadbalancer: Add support for external traffic policy to l2 announcement

    edaabaa
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.feature/l2-announcementkind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.pinnedThese issues are not marked stale by our issue bot.

    Type

    Bug

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions