Skip to content

[BUG] layer2_forward have no effect #5061

Closed

Description

@kvaps

Kube-OVN Version

v1.13.2

Kubernetes Version

v1.30.3

Operation-system/Kernel Version

6.12.9-talos

Description

ovn.kubernetes.io/layer2_forward=false is not blocking egress traffic from pod with wrong source-ip.
I guess the default behavior should block it?

Steps To Reproduce

  1. Install Kube-OVN and Cilium in chaining mode
  2. Disable enable-source-ip-verification option in cilium (details Option to disable SourceIPVerification while chaining CNI cilium/cilium#33889)
  3. Create VM:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: virtual-machine-vpn-test
  namespace: tenant-kvaps
spec:
  dataVolumeTemplates:
  - metadata:
      name: virtual-machine-vpn-test
    spec:
      pvc:
        accessModes:
        - ReadWriteMany
        resources:
          requests:
            storage: 30Gi
        storageClassName: replicated
        volumeMode: Block
      source:
        http:
          url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
  running: true
  template:
    metadata:
      annotations:
        kubevirt.io/allow-pod-bridge-network-live-migration: "true"
        ovn.kubernetes.io/layer2_forward: "true"
    spec:
      accessCredentials:
      - sshPublicKey:
          source:
            secret:
              secretName: virtual-machine-vpn-test-ssh-keys
      architecture: amd64
      domain:
        devices:
          disks:
          - disk:
              bus: scsi
            name: systemdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
          interfaces:
          - name: default
        machine:
          type: q35
      networks:
      - name: default
      terminationGracePeriodSeconds: 30
      volumes:
      - dataVolume:
          name: virtual-machine-vpn-test
        name: systemdisk
      - cloudInitNoCloud:
          secretRef:
            name: virtual-machine-vpn-test-cloud-init
        name: cloudinitdis
  1. Exec into this VM and enable forwarding:
sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
  1. Get IP for this virtual machine

Create pod with the custom route:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-routes
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        ovn.kubernetes.io/routes: |
          [{
            "gw": "10.244.6.136"
          }]
    spec:
      containers:
      - name: nginx
        image: docker.io/library/nginx:alpine
  1. Exec into this pod
kubectl exec -ti custom-routes-6ffbd9b6fd-cpmnd -- sh
  1. Try accessing external network
ping 8.8.8.8
curl icanhazip.com

It will be working.

  1. Now edit VM and remove this annotation:
ovn.kubernetes.io/layer2_forward: "true"
  1. Restart the VM
virtctl stop virtual-machine-vpn-test
virtctl start virtual-machine-vpn-test
  1. Try accessing external network from the pod again
ping 8.8.8.8
curl icanhazip.com

It is still working

Current Behavior

You can access the external network from pod via VM in both ways:

  • ovn.kubernetes.io/layer2_forward set to true
  • ovn.kubernetes.io/layer2_forward set to false or unspecified

Expected Behavior

You should not be able to do this, until ovn.kubernetes.io/layer2_forward set to true.

As virtual machine forward packets with external source-ip.

# pod starts pinging 8.8.8.8
15:54:21.739219 fa13882615d1_h P   IP 10.244.6.163 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
# packet comming to the router vm
15:54:21.742666 a7a63286962b_h Out IP 10.244.6.163 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
# router vm masquerades it
15:54:21.743652 a7a63286962b_h P   IP 10.244.6.136 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
# then packet coming outside
15:54:21.744846 ovn0  In  IP 10.244.6.136 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
15:54:21.744912 bond0.200 Out IP 91.223.132.53 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
15:54:21.744915 bond0 Out IP 91.223.132.53 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
15:54:21.744921 eno12399np0 Out IP 91.223.132.53 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
# response recieved by the node
15:54:21.757117 enp139s0f1np1 In  IP 8.8.8.8 > 91.223.132.53: ICMP echo reply, id 146, seq 0, length 64
15:54:21.757121 bond0 In  IP 8.8.8.8 > 91.223.132.53: ICMP echo reply, id 146, seq 0, length 64
15:54:21.757124 bond0.220 In  IP 8.8.8.8 > 91.223.132.53: ICMP echo reply, id 146, seq 0, length 64
15:54:21.757157 ovn0  Out IP 8.8.8.8 > 10.244.6.136: ICMP echo reply, id 146, seq 0, length 64
# node sends it back to router vm
15:54:21.758012 a7a63286962b_h Out IP 8.8.8.8 > 10.244.6.136: ICMP echo reply, id 146, seq 0, length 64
# router vm sends it back to the pod
15:54:21.758514 a7a63286962b_h P   IP 8.8.8.8 > 10.244.6.163: ICMP echo reply, id 146, seq 0, length 64

<it should drop it right here>

Activity

  1. added
    bugSomething isn't working
    on Mar 7, 2025
  2. kvaps commented on Mar 7, 2025

    @kvaps
    ContributorAuthor
    kubectl -n cozy-kubeovn exec svc/ovn-nb -- ovn-nbctl list logical_switch_port virtual-machine-vpn-test.tenant-kvaps
    

    When option enabled:

    _uuid               : e23d02c8-af3d-4aea-8009-d93a82559f93
    addresses           : ["c2:98:0b:74:43:0a 10.244.6.136", unknown]
    dhcpv4_options      : []
    dhcpv6_options      : []
    dynamic_addresses   : []
    enabled             : []
    external_ids        : {ls=ovn-default, pod="tenant-kvaps/virtual-machine-vpn-test", security_groups="", vendor=kube-ovn}
    ha_chassis_group    : []
    mirror_rules        : []
    name                : virtual-machine-vpn-test.tenant-kvaps
    options             : {}
    parent_name         : []
    port_security       : []
    tag                 : []
    tag_request         : []
    type                : ""
    up                  : true

    When option disabled:

    _uuid               : e23d02c8-af3d-4aea-8009-d93a82559f93
    addresses           : ["c2:98:0b:74:43:0a 10.244.6.136"]
    dhcpv4_options      : []
    dhcpv6_options      : []
    dynamic_addresses   : []
    enabled             : []
    external_ids        : {ls=ovn-default, pod="tenant-kvaps/virtual-machine-vpn-test", vendor=kube-ovn}
    ha_chassis_group    : []
    mirror_rules        : []
    name                : virtual-machine-vpn-test.tenant-kvaps
    options             : {}
    parent_name         : []
    port_security       : []
    tag                 : []
    tag_request         : []
    type                : ""
    up                  : true

    Unfortunately absence of unknown in addresses makes no effect for blocking IPs with wrong src-address

  3. dosubot commented on Mar 7, 2025

    @dosubot

    The issue you're experiencing seems to be related to the ovn.kubernetes.io/layer2_forward annotation not functioning as expected in your Kube-OVN setup. The expected behavior is that setting layer2_forward to false should block egress traffic from a pod with an incorrect source IP, but this is not happening.

    From the information provided, it appears that the logical switch port configuration does not change significantly when the layer2_forward annotation is toggled. The presence or absence of unknown in the addresses field does not seem to affect the blocking of IPs with incorrect source addresses.

    This could be due to a few reasons:

    1. Annotation Handling: The layer2_forward annotation might not be correctly implemented or recognized in the current version of Kube-OVN you are using (v1.13.2). It's possible that the annotation does not trigger the expected changes in the logical switch port configuration.

    2. Integration with Cilium: Since you are using Cilium in chaining mode, there might be an interaction between Kube-OVN and Cilium that affects how source IP verification is handled. Disabling the enable-source-ip-verification option in Cilium might be contributing to this behavior.

    3. Version or Configuration Issue: There might be a bug or a configuration issue in the specific versions of Kube-OVN or Kubernetes you are using. Checking for any updates or patches that address this issue could be beneficial.

    To further investigate, you might want to:

    • Review the Kube-OVN documentation or release notes for any mentions of changes or known issues related to the layer2_forward annotation.
    • Test with different versions of Kube-OVN or Kubernetes to see if the behavior changes.
    • Examine the interaction between Kube-OVN and Cilium more closely, especially regarding how network policies and annotations are processed in chaining mode.

    Unfortunately, without specific documentation or code references from the Kube-OVN repository regarding the layer2_forward annotation, it's challenging to provide a definitive solution.

    To continue talking to Dosu, mention @dosu.


    Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

  4. zhangzujian commented on Mar 7, 2025

    @zhangzujian
    Member

    This option works as expected because it does nothing but adding an unknown address to the logical switch port. It does not affect layer 3 and above.

    I think what you need is a security function such as network policy.

  5. kvaps commented on Mar 7, 2025

    @kvaps
    ContributorAuthor

    I just tried to enable ovn.kubernetes.io/port_security=true, and my config turned to:

    _uuid               : e23d02c8-af3d-4aea-8009-d93a82559f93
    addresses           : ["c2:98:0b:74:43:0a 10.244.6.136", unknown]
    dhcpv4_options      : []
    dhcpv6_options      : []
    dynamic_addresses   : []
    enabled             : []
    external_ids        : {ls=ovn-default, pod="tenant-kvaps/virtual-machine-vpn-test", vendor=kube-ovn}
    ha_chassis_group    : []
    mirror_rules        : []
    name                : virtual-machine-vpn-test.tenant-kvaps
    options             : {}
    parent_name         : []
    port_security       : ["c2:98:0b:74:43:0a 10.244.6.136"]
    tag                 : []
    tag_request         : []
    type                : ""
    up                  : true

    This way it does not allow vm to send packets not related to 10.244.6.136.

    1. Souldn't
    ovn.kubernetes.io/layer2_forward: "true"
    ovn.kubernetes.io/port_security: "true"
    

    also update port_security to this:

    port_security       : ["c2:98:0b:74:43:0a 10.244.6.136", unknown]
    1. Is there any way to enable pod_security for all, except specific pods?

    UPD, I just tried:

    kubectl -n cozy-kubeovn exec svc/ovn-nb -- ovn-nbctl set logical_switch_port virtual-machine-vpn-test.tenant-kvaps 'port_security=["c2:98:0b:74:43:0a 10.244.6.136", unknown]'
    

    but it still does not allow to send my pod packets withdifferent than this source-ip 10.244.6.136

  6. oilbeater commented on Mar 10, 2025

    @oilbeater
    Member

    According to ovn-nb Each element in the set must begin with one Ethernet address. So I think the unknown is an Illegal element and is skipped.

    To enable port security for all, I believe this feature is similar to the issue discussed in #5062. The default settings are designed for common use cases, and Kube-OVN also provide options to customize the behavior at the pod level. For specific use cases, such as enabling port security for all pods, a namespace, or a group of label selected pods, it would be better to use a separate controller or webhook to handle this. This approach would allow the functionality to be isolated from the Kube-OVN in-tree code, making it easier to adapt as requirements evolve.

  7. kvaps commented on Mar 10, 2025

    @kvaps
    ContributorAuthor

    Got it, I'm going to prepare webhook, and I will let you know.
    Maybe you'll be interested to get it as optional component to upstream.

  8. kvaps commented on Mar 13, 2025

    @kvaps
    ContributorAuthor

    So we have implemented a webhook for kube-ovn, which just copies annotations port_security and routes annotations from the namespace to all pods created in it.

    Source code is available here:
    https://github.com/cozystack/cozystack/tree/main/packages/system/kubeovn-webhook

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions