Repository navigation
Closed
Description
Kube-OVN Version
v1.13.2
Kubernetes Version
v1.30.3
Operation-system/Kernel Version
6.12.9-talos
Description
ovn.kubernetes.io/layer2_forward=false is not blocking egress traffic from pod with wrong source-ip.
I guess the default behavior should block it?
Steps To Reproduce
- Install Kube-OVN and Cilium in chaining mode
- Disable
enable-source-ip-verificationoption in cilium (details Option to disable SourceIPVerification while chaining CNI cilium/cilium#33889) - Create VM:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: virtual-machine-vpn-test
namespace: tenant-kvaps
spec:
dataVolumeTemplates:
- metadata:
name: virtual-machine-vpn-test
spec:
pvc:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 30Gi
storageClassName: replicated
volumeMode: Block
source:
http:
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
running: true
template:
metadata:
annotations:
kubevirt.io/allow-pod-bridge-network-live-migration: "true"
ovn.kubernetes.io/layer2_forward: "true"
spec:
accessCredentials:
- sshPublicKey:
source:
secret:
secretName: virtual-machine-vpn-test-ssh-keys
architecture: amd64
domain:
devices:
disks:
- disk:
bus: scsi
name: systemdisk
- disk:
bus: virtio
name: cloudinitdisk
interfaces:
- name: default
machine:
type: q35
networks:
- name: default
terminationGracePeriodSeconds: 30
volumes:
- dataVolume:
name: virtual-machine-vpn-test
name: systemdisk
- cloudInitNoCloud:
secretRef:
name: virtual-machine-vpn-test-cloud-init
name: cloudinitdis- Exec into this VM and enable forwarding:
sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
- Get IP for this virtual machine
Create pod with the custom route:
apiVersion: apps/v1
kind: Deployment
metadata:
name: custom-routes
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
annotations:
ovn.kubernetes.io/routes: |
[{
"gw": "10.244.6.136"
}]
spec:
containers:
- name: nginx
image: docker.io/library/nginx:alpine- Exec into this pod
kubectl exec -ti custom-routes-6ffbd9b6fd-cpmnd -- sh
- Try accessing external network
ping 8.8.8.8
curl icanhazip.com
It will be working.
- Now edit VM and remove this annotation:
ovn.kubernetes.io/layer2_forward: "true"- Restart the VM
virtctl stop virtual-machine-vpn-test
virtctl start virtual-machine-vpn-test
- Try accessing external network from the pod again
ping 8.8.8.8
curl icanhazip.com
It is still working
Current Behavior
You can access the external network from pod via VM in both ways:
ovn.kubernetes.io/layer2_forwardset totrueovn.kubernetes.io/layer2_forwardset tofalseor unspecified
Expected Behavior
You should not be able to do this, until ovn.kubernetes.io/layer2_forward set to true.
As virtual machine forward packets with external source-ip.
# pod starts pinging 8.8.8.8
15:54:21.739219 fa13882615d1_h P IP 10.244.6.163 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
# packet comming to the router vm
15:54:21.742666 a7a63286962b_h Out IP 10.244.6.163 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
# router vm masquerades it
15:54:21.743652 a7a63286962b_h P IP 10.244.6.136 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
# then packet coming outside
15:54:21.744846 ovn0 In IP 10.244.6.136 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
15:54:21.744912 bond0.200 Out IP 91.223.132.53 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
15:54:21.744915 bond0 Out IP 91.223.132.53 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
15:54:21.744921 eno12399np0 Out IP 91.223.132.53 > 8.8.8.8: ICMP echo request, id 146, seq 0, length 64
# response recieved by the node
15:54:21.757117 enp139s0f1np1 In IP 8.8.8.8 > 91.223.132.53: ICMP echo reply, id 146, seq 0, length 64
15:54:21.757121 bond0 In IP 8.8.8.8 > 91.223.132.53: ICMP echo reply, id 146, seq 0, length 64
15:54:21.757124 bond0.220 In IP 8.8.8.8 > 91.223.132.53: ICMP echo reply, id 146, seq 0, length 64
15:54:21.757157 ovn0 Out IP 8.8.8.8 > 10.244.6.136: ICMP echo reply, id 146, seq 0, length 64
# node sends it back to router vm
15:54:21.758012 a7a63286962b_h Out IP 8.8.8.8 > 10.244.6.136: ICMP echo reply, id 146, seq 0, length 64
# router vm sends it back to the pod
15:54:21.758514 a7a63286962b_h P IP 8.8.8.8 > 10.244.6.163: ICMP echo reply, id 146, seq 0, length 64
<it should drop it right here>Metadata
Metadata
Assignees
Labels
No labels
Activity
kvaps commentedon Mar 7, 2025
When option enabled:
When option disabled:
Unfortunately absence of
unknownin addresses makes no effect for blocking IPs with wrong src-addressdosubot commentedon Mar 7, 2025
The issue you're experiencing seems to be related to the
ovn.kubernetes.io/layer2_forwardannotation not functioning as expected in your Kube-OVN setup. The expected behavior is that settinglayer2_forwardtofalseshould block egress traffic from a pod with an incorrect source IP, but this is not happening.From the information provided, it appears that the logical switch port configuration does not change significantly when the
layer2_forwardannotation is toggled. The presence or absence ofunknownin theaddressesfield does not seem to affect the blocking of IPs with incorrect source addresses.This could be due to a few reasons:
Annotation Handling: The
layer2_forwardannotation might not be correctly implemented or recognized in the current version of Kube-OVN you are using (v1.13.2). It's possible that the annotation does not trigger the expected changes in the logical switch port configuration.Integration with Cilium: Since you are using Cilium in chaining mode, there might be an interaction between Kube-OVN and Cilium that affects how source IP verification is handled. Disabling the
enable-source-ip-verificationoption in Cilium might be contributing to this behavior.Version or Configuration Issue: There might be a bug or a configuration issue in the specific versions of Kube-OVN or Kubernetes you are using. Checking for any updates or patches that address this issue could be beneficial.
To further investigate, you might want to:
layer2_forwardannotation.Unfortunately, without specific documentation or code references from the Kube-OVN repository regarding the
layer2_forwardannotation, it's challenging to provide a definitive solution.To continue talking to Dosu, mention @dosu.
Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other
zhangzujian commentedon Mar 7, 2025
This option works as expected because it does nothing but adding an
unknownaddress to the logical switch port. It does not affect layer 3 and above.I think what you need is a security function such as network policy.
kvaps commentedon Mar 7, 2025
I just tried to enable
ovn.kubernetes.io/port_security=true, and my config turned to:This way it does not allow vm to send packets not related to
10.244.6.136.also update port_security to this:
UPD, I just tried:
but it still does not allow to send my pod packets withdifferent than this source-ip
10.244.6.136oilbeater commentedon Mar 10, 2025
According to ovn-nb
Each element in the set must begin with one Ethernet address.So I think theunknownis an Illegal element and is skipped.To enable port security for all, I believe this feature is similar to the issue discussed in #5062. The default settings are designed for common use cases, and Kube-OVN also provide options to customize the behavior at the pod level. For specific use cases, such as enabling port security for all pods, a namespace, or a group of label selected pods, it would be better to use a separate controller or webhook to handle this. This approach would allow the functionality to be isolated from the Kube-OVN in-tree code, making it easier to adapt as requirements evolve.
kvaps commentedon Mar 10, 2025
Got it, I'm going to prepare webhook, and I will let you know.
Maybe you'll be interested to get it as optional component to upstream.
kvaps commentedon Mar 13, 2025
So we have implemented a webhook for kube-ovn, which just copies annotations
port_securityandroutesannotations from the namespace to all pods created in it.Source code is available here:
https://github.com/cozystack/cozystack/tree/main/packages/system/kubeovn-webhook