Skip to content

portmap nftables backend invalid rules (dnat from input hook + invalid ipv6 rules) #1115

@champtar

Description

@champtar

Trying the new nftables backend, it fails on EL 9.4

# nft --version
nftables v1.0.9 (Old Doc Yak #3)
# uname -r
5.14.0-427.40.1.el9_4.x86_64

conf

{
  "type": "portmap",
  "capabilities": {"portMappings": true},
  "backend": "nftables",
  "conditionsV4": ["ip", "daddr", "!=", "{ 127.0.0.0/8, 198.19.254.254 }"]
},

Error

Nov 05 17:15:34 atsc2 kubelet[4258]: E1105 17:15:34.352838    4258 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err=<
Nov 05 17:15:34 atsc2 kubelet[4258]:         rpc error: code = Unknown desc = failed to setup network for sandbox "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621": plugin type="multus-cni" name="multus-cni-network" failed (add): [traefik/appliance-ingress-traefik-698cd97568-5xg88/3908060c-0419-481c-865d-3230b95c3a84:mgmt]: error adding container to network "mgmt": plugin type="portmap" failed (add): unable to set up nftables rules for port mappings: /dev/stdin:13:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 81 dnat ip addr . port to 198.18.1.194 . 10081 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:14:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8444 dnat ip addr . port to 198.18.1.194 . 18444 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:15:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 82 dnat ip addr . port to 198.18.1.194 . 10082 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:16:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8445 dnat ip addr . port to 198.18.1.194 . 18445 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:17:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 83 dnat ip addr . port to 198.18.1.194 . 10083 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:18:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8446 dnat ip addr . port to 198.18.1.194 . 18446 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:19:1-168: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport masquerading ip saddr 198.18.1.194 ip daddr 198.18.1.194 masquerade comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:20:1-165: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport masquerading ip saddr 127.0.0.1 ip daddr 198.18.1.194 masquerade comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:  >

Using strace, the rules are:

add table ip cni_hostport { comment "CNI portmap plugin" ; }
add chain ip cni_hostport hostports
add chain ip cni_hostport hostip_hostports
add chain ip cni_hostport input { type nat hook input priority -100 ; }
flush chain ip cni_hostport input
add rule ip cni_hostport input ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostip_hostports
add rule ip cni_hostport input ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostports
add chain ip cni_hostport output { type nat hook output priority -100 ; }
flush chain ip cni_hostport output
add rule ip cni_hostport output ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostip_hostports
add rule ip cni_hostport output ip daddr != { 127.0.0.0/8, 198.19.254.254 } fib daddr type local jump hostports
add chain ip cni_hostport masquerading { type nat hook postrouting priority 100 ; }
add rule ip cni_hostport hostports ip protocol tcp th dport 81 dnat ip addr . port to 198.18.0.56 . 10081 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8444 dnat ip addr . port to 198.18.0.56 . 18444 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 82 dnat ip addr . port to 198.18.0.56 . 10082 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8445 dnat ip addr . port to 198.18.0.56 . 18445 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 83 dnat ip addr . port to 198.18.0.56 . 10083 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8446 dnat ip addr . port to 198.18.0.56 . 18446 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport masquerading ip saddr 198.18.0.56 ip daddr 198.18.0.56 masquerade comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport masquerading ip saddr 127.0.0.1 ip daddr 198.18.0.56 masquerade comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"

I think instead of

add chain ip cni_hostport input { type nat hook input priority -100 ; }

we should use

add chain ip cni_hostport input { type nat hook prerouting priority -100 ; }

(and rename input to prerouting)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions