Kernel panic on flatcar 3139.2.0 with loadBalancer.acceleration native

### Is there an existing issue for this?

- [X] I have searched the existing issues

### What happened?

All worker nodes got stuck in rebooting due to kernel panics after we upgrade those OS from flatcar 3033.2.4, kernel 5.10.107 to flatcar 3139.2.0, kernel 5.15.32. It seems that something is wrong with `loadBalancer.acceleration: native` on flatcar 3139.2.0.

## Test Result

- flatcar 3139.2.0, kernel 5.15.32
  - cilium 1.11.3
    - loadBalancer.acceleration: native -> Panic
    - loadBalancer.acceleration: disabled -> OK
  - cilium 1.11.2
    - loadBalancer.acceleration: native -> Panic
    - loadBalancer.acceleration: disabled -> OK
- flatcar 3033.2.4, kernel 5.10.107
  - cilium 1.11.3
    - loadBalancer.acceleration: native -> OK
    - loadBalancer.acceleration: disabled -> OK
  - cilium 1.11.2
    - loadBalancer.acceleration: native -> OK
    - loadBalancer.acceleration: disabled -> OK



### Cilium Version

v1.11.3

### Kernel Version

$ uname -a
Linux rack1-cs5 5.15.32-flatcar #1 SMP Tue Apr 5 17:17:31 -00 2022 x86_64 AMD EPYC 7413 24-Core Processor AuthenticAMD GNU/Linux

### Kubernetes Version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"archive", BuildDate:"2021-12-20T06:57:14Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

### Sysdump

I couldn't run `cilium sysdump` because all workers got stuck.

### Relevant log output

```shell
[  419.309274] SELinux:  Context system_u:object_r:container_file_t:s0 is not valid (left unmapped).
[  419.309274] SELinux:  Context system_u:object_r:container_file_t:s0 is not valid (left unmapped).
[  516.835776] IPv6: ADDRCONF(NETDEV_CHANGE): cilium_net: link becomes ready
[  516.846413] IPv6: ADDRCONF(NETDEV_CHANGE): cilium_host: link becomes ready
[  516.835776] IPv6: ADDRCONF(NETDEV_CHANGE): cilium_net: link becomes ready
[  516.846413] IPv6: ADDRCONF(NETDEV_CHANGE): cilium_host: link becomes ready
[  520.006295] NET: Registered PF_ALG protocol family
[  520.006295] NET: Registered PF_ALG protocol family
[  520.293618] general protection fault, probably for non-canonical address 0xffff12784e218580: 0000 [#1] SMP PTI
[  520.307566] CPU: 10 PID: 0 Comm: swapper/10 Not tainted 5.15.32-flatcar #1
[  520.317475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[  520.332753] RIP: 0010:uncharge_page+0x11a/0x170
[  520.338345] Code: 8d e5 ff f6 45 54 01 75 bc 48 8b 45 10 a8 03 75 27 65 48 ff 08 5b 5d 41 5c 41 5d 41 5e e9 7e 8d e5 ff 48 8b 45 10 a8 03 75 49 <65> 48 ff 00 e8 6d 8d e5 ff e9 51 ff ff ff 48 8b 45 18 f0 488
[  520.363046] RSP: 0018:ffff9fcc00294c98 EFLAGS: 00010246
[  520.369842] RAX: ffff893a22598580 RBX: ffff9fcc00294cc8 RCX: 0000000000000000
[  520.378313] RDX: 0000000000000000 RSI: ffff9fcc00294cc8 RDI: ffffcd5b04db68c0
[  520.386854] RBP: ffff893a3505e000 R08: 000000000000000c R09: 0000000000001000
[  520.394920] R10: ffff9fcc00294df0 R11: ffff893ac85fc940 R12: ffffcd5b04db68c0
[  520.403387] R13: ffff893a36da3108 R14: 0000000000000000 R15: 0000000000001000
[  520.412159] FS:  0000000000000000(0000) GS:ffff893e2bc80000(0000) knlGS:0000000000000000
[  520.423741] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  520.431023] CR2: 0000561b82ea2f00 CR3: 0000000136dea003 CR4: 0000000000370ee0
[  520.439861] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  520.448639] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  520.457505] Call Trace:
[  520.460841]  <IRQ>
[  520.463425]  __mem_cgroup_uncharge+0x66/0x90
[  520.468909]  __put_page+0x2d/0x40
[  520.473681]  0xffffffffc05b1807
[  520.478029]  0xffffffffc05b40a3
[  520.481641]  ? detach_buf_split+0x6c/0x130
[  520.486641]  0xffffffffc05b47a4
[  520.490499]  __napi_poll+0x2a/0x130
[  520.494837]  net_rx_action+0x24b/0x2a0
[  520.499612]  __do_softirq+0xe1/0x296
[  520.504069]  irq_exit_rcu+0x84/0xa0
[  520.508218]  common_interrupt+0x80/0xa0
[  520.513053]  </IRQ>
[  520.515977]  <TASK>
[  520.518777]  asm_common_interrupt+0x1e/0x40
[  520.524051] RIP: 0010:native_safe_halt+0xb/0x10
[  520.529823] Code: 80 48 02 20 48 8b 00 a8 08 74 83 eb c2 cc cc eb 07 0f 00 2d d9 6e 5d 00 f4 c3 0f 1f 44 00 00 eb 07 0f 00 2d c9 6e 5d 00 fb f4 <c3> cc cc cc cc 0f 1f 44 00 00 53 65 8b 15 2b 06 7e 78 66 90c
[  520.552596] RSP: 0018:ffff9fcc000cbec8 EFLAGS: 00000202
[  520.559248] RAX: ffffffff87834eb0 RBX: 000000000000000a RCX: ffff893e2bc9be40
[  520.567815] RDX: ffff893e2bcacfe0 RSI: 7fffff8708ca87a1 RDI: 00000000000c38a8
[  520.576742] RBP: ffff893a003dbf00 R08: 00000000000c38a7 R09: 00000078f7633f1e
[  520.585745] R10: 0000000000000001 R11: 0000000000000001 R12: ffff893a003dbf00
[  520.593716] R13: ffff893a003dbf00 R14: 0000000000000000 R15: 0000000000000000
[  520.601770]  ? __cpuidle_text_start+0x8/0x8
[  520.606708]  default_idle+0xa/0x10
[  520.611437]  default_idle_call+0x31/0xc0
[  520.616562]  do_idle+0x1ef/0x240
[  520.620859]  cpu_startup_entry+0x19/0x20
[  520.626026]  start_secondary+0x119/0x150
[  520.630714]  secondary_startup_64_no_verify+0xc2/0xcb
[  520.636711]  </TASK>
[  520.639427] Modules linked in: algif_hash af_alg veth xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6table_filter ip6table_raw ip6table_mangle ip6_tables iptable_raw iptable_mangle xt_MASQUERADE xt_conntrack xc
[  520.744070] BUG: unable to handle page fault for address: ffff893a1c054000
:     52502.029.367184] 4g3e7ne7ra]l  pr-ot-ec-ti[o en nfadul t,t prroabacble 6dc7y for non-canonical add78rdesbsb 05xfcf3fdf1276814ee 2]1-85-8-0
  00[0 0  [5#210] SM.P7 5PT3093] #PF: supervisor write access in kernel mode
[  520.761981] RIP: 0010:uncharge_page+0x11a/0x170
[  520.768107] #PF: error_code(0x0003) - permissions violation
[  520.773417] Code: 8d e5 ff f6 45 54 01 75 bc 48 8b 45 10 a8 03 75 27 65 48 ff 08 5b 5d 41 5c 41 5d 41 5e e9 7e 8d e5 ff 48 8b 45 10 a8 03 75 49 <65> 48 ff 00 e8 6d 8d e5 ff e9 51 ff ff ff 48 8b 45 18 f0 488
[  520.779352] PGD 1c8e01067 P4D 1c8e01067 PUD 100b40063 PMD 1397c3063 PTE 800000011c054161
[  520.799732] RSP: 0018:ffff9fcc00294c98 EFLAGS: 00010246
[  520.808476] Oops: 0003 [#2] SMP PTI
[  520.808627]
[  520.814189] CPU: 0 PID: 7141 Comm: dockerd Tainted: G      D           5.15.32-flatcar #1
[  520.818272] RAX: ffff893a22598580 RBX: ffff9fcc00294cc8 RCX: 0000000000000000
[  520.819827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[  520.829165] RDX: 0000000000000000 RSI: ffff9fcc00294cc8 RDI: ffffcd5b04db68c0
[  520.836706] RIP: 0010:clear_page_erms+0x7/0x10
[  520.850425] RBP: ffff893a3505e000 R08: 000000000000000c R09: 0000000000001000
[  520.859183] Code: 48 89 47 18 48 89 47 20 48 89 47 28 48 89 47 30 48 89 47 38 48 8d 7f 40 75 d9 90 c3 0f 1f 80 00 00 00 00 b9 00 10 00 00 31 c0 <f3> aa c3 cc cc cc cc cc cc 0f 1f 44 00 00 48 85 ff 0f 84 f20
[  520.865186] R10: ffff9fcc00294df0 R11: ffff893ac85fc940 R12: ffffcd5b04db68c0
[  520.874086] RSP: 0018:ffff9fcc0175faa8 EFLAGS: 00010246
[  520.874093] RAX: 0000000000000000 RBX: ffffcd5b04701500 RCX: 0000000000001000
[  520.874095] RDX: ffffcd5b04701500 RSI: ffff893a3cee9f80 RDI: ffff893a1c054000
[  520.874097] RBP: 0000000000500dc0 R08: ffffcd5b04701540 R09: 0000000000000000
[  520.896457] R13: ffff893a36da3108 R14: 0000000000000000 R15: 0000000000001000
[  520.903855] R10: 0000000000000293 R11: ffff893e2ba30c70 R12: 0000000000000000
[  520.910209] FS:  0000000000000000(0000) GS:ffff893e2bc80000(0000) knlGS:0000000000000000
[  520.918749] R13: 0000000000000901 R14: ffff893e2ba29688 R15: 0000000000000000
[  520.927353] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  520.935856] FS:  00007fda9affd640(0000) GS:ffff893e2ba00000(0000) knlGS:0000000000000000
[  520.944061] CR2: 0000561b82ea2f00 CR3: 0000000136dea003 CR4: 0000000000370ee0
[  520.952508] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  520.962629] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  520.971774] CR2: ffff893a1c054000 CR3: 0000000101032004 CR4: 0000000000370ef0
[  520.978576] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  520.989157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  520.996976] Kernel panic - not syncing: Fatal exception in interrupt
[  521.004219] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  521.054549] Call Trace:
[  521.057530]  <TASK>
[  521.060298]  kernel_init_free_pages.part.0+0x46/0x60
[  521.066664]  prep_new_page+0x6c/0x80
[  521.070909]  get_page_from_freelist+0xa6f/0xc70
[  521.076228]  __alloc_pages+0x181/0x310
[  521.081133]  __get_free_pages+0xd/0x30
[  521.085438]  __pud_alloc+0x2c/0x110
[  521.089573]  __handle_mm_fault+0x7de/0x1370
[  521.094367]  handle_mm_fault+0xcf/0x2a0
[  521.099230]  __get_user_pages+0x210/0x680
[  521.104299]  __get_user_pages_remote+0xda/0x330
[  521.109385]  get_arg_page+0x5f/0x100
[  521.113456]  copy_string_kernel+0x100/0x1e0
[  521.118206]  do_execveat_common.isra.0+0x109/0x1d0
[  521.123657]  __x64_sys_execve+0x33/0x40
[  521.127999]  do_syscall_64+0x3b/0x90
[  521.132146]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  521.137640] RIP: 0033:0x55ea4a3e74d6
[  521.141598] Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 1b 48 c7 44 24 28 ff ff ff ff 48 c7 440
[  521.163987] RSP: 002b:000000c0017e6868 EFLAGS: 00000206 ORIG_RAX: 000000000000003b
[  521.173370] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 000055ea4a3e74d6
[  521.181649] RDX: 000000c0018cc000 RSI: 000000c000952228 RDI: 000000c000160200
[  521.189774] RBP: 000000c0017e6a10 R08: 0000000000000052 R09: 0000000000000000
[  521.198755] R10: 0000000000000008 R11: 0000000000000206 R12: 000055ea4a3db931
[  521.207146] R13: 0000000000000001 R14: 000000c0012704e0 R15: ffffffffffffffff
[  521.214837]  </TASK>
[  521.217448] Modules linked in: algif_hash af_alg veth xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6table_filter ip6table_raw ip6table_mangle ip6_tables iptable_raw iptable_mangle xt_MASQUERADE xt_conntrack xc
[  521.315858] CR2: ffff893a1c054000
[  521.320959] ---[ end trace 6dc778dbb5c3d61f ]---
[  521.327623] RIP: 0010:uncharge_page+0x11a/0x170
[  521.333146] Code: 8d e5 ff f6 45 54 01 75 bc 48 8b 45 10 a8 03 75 27 65 48 ff 08 5b 5d 41 5c 41 5d 41 5e e9 7e 8d e5 ff 48 8b 45 10 a8 03 75 49 <65> 48 ff 00 e8 6d 8d e5 ff e9 51 ff ff ff 48 8b 45 18 f0 488
[  521.356648] RSP: 0018:ffff9fcc00294c98 EFLAGS: 00010246
[  521.362798] RAX: ffff893a22598580 RBX: ffff9fcc00294cc8 RCX: 0000000000000000
[  521.371761] RDX: 0000000000000000 RSI: ffff9fcc00294cc8 RDI: ffffcd5b04db68c0
[  521.380494] RBP: ffff893a3505e000 R08: 000000000000000c R09: 0000000000001000
[  521.389813] R10: ffff9fcc00294df0 R11: ffff893ac85fc940 R12: ffffcd5b04db68c0
[  521.398655] R13: ffff893a36da3108 R14: 0000000000000000 R15: 0000000000001000
[  521.407056] FS:  00007fda9affd640(0000) GS:ffff893e2ba00000(0000) knlGS:0000000000000000
[  521.417495] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  521.425198] CR2: ffff893a1c054000 CR3: 0000000101032004 CR4: 0000000000370ef0
[  521.434272] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  521.444319] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  522.132705] Shutting down cpus with NMI
[  522.137871] Kernel Offset: 0x6000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  522.149691] Rebooting in 10 seconds..
```


### Anything else?

helm values.yaml

```
bgp:
  announce:
    loadbalancerIP: true
  enabled: true
bpf:
  hostRouting: true
cni:
  chainingMode: "generic-veth"
  customConf: true
datapathMode: "veth"
devices: "eth+ eno1+ eno2+"
enableIdentityMark: false
enableIPv4Masquerade: false
externalIPs:
  enabled: true
hostPort:
  enabled: false
hostServices:
  enabled: true
  hostNamespaceOnly: true
hubble:
  relay:
    enabled: true
    tls:
      server:
        enabled: true
    rollOutPods: true
    resources:
      requests:
        cpu: 100m
        memory: 200Mi
  tls:
    auto:
      method: "cronJob"
  metrics:
    enabled:
      - "dns"
      - "drop:destinationContext=pod|dns|ip;sourceContext=pod|dns|ip"
      - "tcp"
      - "flow:destinationContext=pod|dns|ip;sourceContext=pod|dns|ip"
      - "icmp"
      - "http"
k8sServiceHost: 127.0.0.1
k8sServicePort: 16443
kubeProxyReplacement: "partial"
loadBalancer:
  acceleration: native
  algorithm: maglev
  mode: dsr
maglev:
  hashSeed: 3HCx6JennjWtot2U
nodePort:
  directRoutingDevice: "e+"
  enabled: true
operator:
  rollOutPods: true
  prometheus:
    enabled: true
  resources:
    requests:
      cpu: 100m
      memory: 200Mi
policyAuditMode: false
policyEnforcementMode: "default"
prometheus:
  enabled: true
rollOutCiliumPods: true
resources:
  requests:
    cpu: 100m
    memory: 400Mi
tunnel: "disabled"
```

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kernel panic on flatcar 3139.2.0 with loadBalancer.acceleration native #19453

Is there an existing issue for this?

What happened?

Test Result

Cilium Version

Kernel Version

Kubernetes Version

Sysdump

Relevant log output

Anything else?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Kernel panic on flatcar 3139.2.0 with loadBalancer.acceleration native #19453

Description

Is there an existing issue for this?

What happened?

Test Result

Cilium Version

Kernel Version

Kubernetes Version

Sysdump

Relevant log output

Anything else?

Code of Conduct

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions