Repository navigation
Cilium breaks the cluster on 6.18.5-talos because of regressin in kernel/bpf/verifier.c #44216
Description
Is there an existing issue for this?
- I have searched the existing issues
Version
equal or higher than v1.19.0 and lower than v1.20.0
What happened?
Comes from this Talos issue.
Cluster network completely goes down after an upgrade to kernel 6.18.5.
How can we reproduce the issue?
Talos: v1.12.2
Cilium: v1.19.0 or v1.18.6
Kernel: 6.18.5-talos
And set any of these to true
kubeProxyReplacement: false
bpf:
masquerade: false
hostFirewall:
enabled: falseCilium Version
Reproduced on both v1.19.0 or v1.18.6
Kernel Version
6.18.5-talos
Kubernetes Version
v1.34.3
Regression
Kernel bug, I think.
Sysdump
I could not execute it, since everything was down.
Relevant log output
kern: warning: [2026-02-06T01:31:33.774368243Z]: ------------[ cut here ]------------
kern: warning: [2026-02-06T01:31:33.774378243Z]: verifier bug: REG INVARIANTS VIOLATION (false_reg1): range bounds violation u64=[0xe01, 0xe00] s64=[0xe01,
0xe00] u32=[0xe01, 0xe00] s32=[0xe01, 0xe00] var_off=(0xe00, 0x0)
kern: warning: [2026-02-06T01:31:33.774397243Z]: WARNING: CPU: 3 PID: 7322 at kernel/bpf/verifier.c:2731 reg_bounds_sanity_check+0x19d/0x210
kern: warning: [2026-02-06T01:31:33.774406243Z]: Modules linked in: virtio_rng virtio_balloon iTCO_wdt iTCO_vendor_support intel_rapl_msr watchdog intel_rapl_common
i2c_i801 ahci i2c_smbus libahci lpc_ich mfd_core virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev
kern: warning: [2026-02-06T01:31:33.774436243Z]: CPU: 3 UID: 0 PID: 7322 Comm: cilium-agent Not tainted 6.18.5-talos #1 NONE
kern: warning: [2026-02-06T01:31:33.774440243Z]: Hardware name: Hetzner vServer/Standard PC (Q35 + ICH9, 2009), BIOS 20171111 11/11/2017
kern: warning: [2026-02-06T01:31:33.774443243Z]: RIP: 0010:reg_bounds_sanity_check+0x19d/0x210
kern: warning: [2026-02-06T01:31:33.774447243Z]: Code: ff 73 18 41 50 57 48 c7 c7 50 34 82 ae 56 4c 89 d6 50 ff 73 30 4c 8b 4b 28 4c 8b 43 40 48 89 55 d8 4c 89 55 e0 e8 b3
7e e3 ff <0f> 0b 48 8b 55 d8 4c 8b 55 e0 48 83 c4 38 e9 ec fe ff ff 48 8b 7b
kern: warning: [2026-02-06T01:31:33.774451243Z]: RSP: 0018:ffffccc64bff3680 EFLAGS: 00010246
kern: warning: [2026-02-06T01:31:33.774454243Z]: RAX: 0000000000000000 RBX: ffff8afe2e429bf0 RCX: 0000000000000000
kern: warning: [2026-02-06T01:31:33.774456243Z]: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
kern: warning: [2026-02-06T01:31:33.774458243Z]: RBP: ffffccc64bff36e8 R08: 0000000000000000 R09: 0000000000000000
kern: warning: [2026-02-06T01:31:33.774460243Z]: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8afe60460000
kern: warning: [2026-02-06T01:31:33.774462243Z]: R13: 0000000000000010 R14: ffff8afe60466300 R15: ffff8afe2e429bf0
kern: warning: [2026-02-06T01:31:33.774467243Z]: FS: 000000c002bbc090(0000) GS:ffff8aff3b7aa000(0000) knlGS:0000000000000000
kern: warning: [2026-02-06T01:31:33.774470243Z]: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kern: warning: [2026-02-06T01:31:33.774472243Z]: CR2: 000000c00486c000 CR3: 00000001a35bc000 CR4: 0000000000350ef0
kern: warning: [2026-02-06T01:31:33.774476243Z]: Call Trace:
kern: warning: [2026-02-06T01:31:33.774479243Z]: <TASK>
kern: warning: [2026-02-06T01:31:33.774484243Z]: ? reg_bounds_sync+0x123/0x1b0
kern: warning: [2026-02-06T01:31:33.774488243Z]: reg_set_min_max+0x1b9/0x200
kern: warning: [2026-02-06T01:31:33.774492243Z]: check_cond_jmp_op+0x408/0x1b70
kern: warning: [2026-02-06T01:31:33.774497243Z]: ? do_check_common+0x29f5/0x35f0
kern: warning: [2026-02-06T01:31:33.774499243Z]: do_check_common+0x29f5/0x35f0
kern: warning: [2026-02-06T01:31:33.774502243Z]: ? mark_fastcall_pattern_for_call+0x4c/0x230
kern: warning: [2026-02-06T01:31:33.774507243Z]: bpf_check+0x276d/0x33b0
kern: warning: [2026-02-06T01:31:33.774510243Z]: bpf_prog_load+0x6fa/0xef0
kern: warning: [2026-02-06T01:31:33.774517243Z]: __sys_bpf+0xbc6/0x2930
kern: warning: [2026-02-06T01:31:33.774522243Z]: __x64_sys_bpf+0x21/0x40
kern: warning: [2026-02-06T01:31:33.774525243Z]: do_syscall_64+0x78/0x940
kern: warning: [2026-02-06T01:31:33.774532243Z]: entry_SYSCALL_64_after_hwframe+0x76/0x7e
kern: warning: [2026-02-06T01:31:33.774536243Z]: RIP: 0033:0x40758e
kern: warning: [2026-02-06T01:31:33.774539243Z]: Code: ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89
df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
kern: warning: [2026-02-06T01:31:33.774543243Z]: RSP: 002b:000000c003e52660 EFLAGS: 00000212 ORIG_RAX: 0000000000000141
kern: warning: [2026-02-06T01:31:33.774546243Z]: RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 000000000040758e
kern: warning: [2026-02-06T01:31:33.774548243Z]: RDX: 0000000000000098 RSI: 000000c003e52b88 RDI: 0000000000000005
kern: warning: [2026-02-06T01:31:33.774550243Z]: RBP: 000000c003e526a0 R08: 0000000000000000 R09: 0000000000000000
kern: warning: [2026-02-06T01:31:33.774552243Z]: R10: 0000000000000000 R11: 0000000000000212 R12: 0000000000000000
kern: warning: [2026-02-06T01:31:33.774554243Z]: R13: 000000c003fcc9c0 R14: 000000c001d568c0 R15: ffffffffffffffff
kern: warning: [2026-02-06T01:31:33.774557243Z]: </TASK>
kern: warning: [2026-02-06T01:31:33.774559243Z]: ---[ end trace 0000000000000000 ]---Anything else?
No response
Cilium Users Document
- Are you a user of Cilium? Please add yourself to the Users doc
Code of Conduct
- I agree to follow this project's Code of Conduct
Reacted by Devin Buhl, Domi, Harald Koch, Matt M., Skye Soss, Andrei Nistor, Albin Gustavsson, Nico Berlee, Artem, Bruno Bigras and 12 more
Pinned by joestringer
joestringerThanks for the reports. From the breadcrumbs in this issue seems like one potential mitigation is to roll back a Talos version update. If you do so, please also report if that helps and which versions you are using. The proper fix for this will need to be developed in the upstream Linux community and the fix backported to all relevant kernels (including Talos).
Activity
fix(talos): downgrade to v1.12.1 for kernel BPF bug
fix(cilium): roll back to 1.18.5 for kernel 6.18.5 compat
onedr0p commentedon Feb 6, 2026
Jumping in to comment, I am seeing this as well on all three of my controller nodes (no worker nodes in my cluster).
However my nodes seem "stable", and I don't see cluster network completely going down like @vlasov-y reported.
My configuration:
haraldkoch commentedon Feb 6, 2026
I see this error on every node in three clusters, but when it occurred on my single-node cluster, the node locked up and never recovered; I had to rebuild from scratch (with cilium 1.18.6).
Same versions as onedr0p; my talos and cilium configs are copied from there.
joestringer commentedon Feb 6, 2026
Thanks for the reports. From the breadcrumbs in this issue seems like one potential mitigation is to roll back a Talos version update. If you do so, please also report if that helps and which versions you are using. The proper fix for this will need to be developed in the upstream Linux community and the fix backported to all relevant kernels (including Talos).
pchaigno commentedon Feb 6, 2026
Thanks for the reports!
These warnings are a known issue upstream. They were reported by syzkaller several times (cf. BPF mailing list), but were thought to be hard to hit in real-world programs. We're working on a long-term solution but since this is hitting Cilium now, we may also need a shorter-term solution.
These warnings should however be guarded by
CONFIG_DEBUG_KERNEL. Does that mean debugging mode is enabled in your kernel? (Even without the warning, the program rejection would still happen, so we'll need a fix either way.)yv-miggo commentedon Feb 6, 2026
Talos documentation says that they use KSSP.
While in KSSP I can see
CONFIG_DEBUG_KERNEL=y, so yes.nberlee commentedon Feb 8, 2026
The kernel config is here:
albgus commentedon Feb 9, 2026
For what it's worth I'm seeing this with several Talos versions / kernel matches
Talos 1.12.3 / kernel 6.18.8-talosTalos 1.12.3 / kernel 6.18.5-talosTalos 1.12.0 / kernel 6.18.1-talosUnlike the first reporter cluster networking is not entirely down, but I am seeing the
verifier bug: REG INVARIANTS VIOLATIONin dmesg on all nodes. I am also seeing wierd intermittent connectivity errors reportingconnect: Operation not permitted, which I believe may be caused by eBPF issues(?)edit: It's possible that the issues I'm seeing is something else as I discovered the root for a lot of it seems to be that CoreDNS has issues resolving external domains, ex:
[ERROR] plugin/errors: 2 hel1.your-objectstorage.com. A: read udp 100.99.5.206:34084->169.254.116.108:53: i/o timeoutSyntax3rror404 commentedon Feb 10, 2026
I also have this issue, if you reboot all the nodes everything works, this issue happens if you delete a pod with a l2 service attached. While the new pod comes up, this kernel warning shows up in the dmesg log and you cannot reach the external ip from the service anymore.
If you reboot the nodes again you can reproduce this.
If you downgrade to cilium 1.18.6 this issue does not happen.
91 remaining items
pchaigno commentedon Apr 1, 2026
Yes, it was: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=efc11a667878. It should be the same as the v2 though; there were no functional changes to the fix itself.
We've seen the
const tnum out of sync with range boundstype of error in the past, but it always came with arange bounds violationerror. Only one would be reported as a kernel warning due to the use ofWARN_ONCE.I'd be very interested to understand if we also have a
range bounds violationhere or if we're looking at something new. @jackmaninov Could you share a sysdump or even just a bugtool report? I'll need to extract the BPF programs to reproduce.It's hard to say for sure without seeing the bytecode, but it's most likely unrelated to these changes. These fix a very specific case of invariant violation.
pchaigno commentedon Apr 1, 2026
We're also working on a patchset to fix the root cause for all these issues: https://lore.kernel.org/bpf/[email protected]/. If you're able to confirm this fixes your issues when applied to your kernel, that would be most helpful!
jackmaninov commentedon Apr 4, 2026
Here's a somewhat stripped bugtool output from an edge node shortly after a reboot (it still has the bpf .o files, don't worry).
I'll take a look at building a kernel with the additional patchset you linked.
jackmaninov commentedon Apr 5, 2026
I can confirm that I still get the
const tnum out of sync with range boundswarning after applying v3 of your patchset (minus the hunk in part 2/6 that removes the check for "false_reg1 == false_reg2" in reg_set_min_max, as I'm merging on top of kernel 6.18.18).I can also confirm that the "Skip bounds adjustment for conditional jumps on same scalar register" patchset does not remove the warning.
mschirrmeister commentedon Apr 6, 2026
@jackmaninov, do you have something with what this can be reliably reproduced? If yes, I would like to try to reproduce on my K3s/Cilium cluster.
jackmaninov commentedon Apr 6, 2026
This reproduces reliably for me; every boot I get one
REG INVARIANTS VIOLATION (alu)warning, and I can get another by setting /sys/kernel/debug/clear_warn_once and restarting the cilium agent).This is running cilium agent v1.17.7 or v1.17.8 on talos 1.12.6 amd64. Machine type doesn't seem to matter, I'm seeing it on bare metal as well as QEMU under Proxmox.
As far as I understand Talos' kernel build system it's just building stock kernel-6.18.18 with the config at https://github.com/siderolabs/pkgs/blob/release-1.12/kernel/build/config-amd64. They distribute their kernel build artifacts at:
https://github.com/siderolabs/talos/releases/download/v1.12.6/vmlinuz-amd64
https://github.com/siderolabs/talos/releases/download/v1.12.6/initramfs-amd64.xz
My cilium configmap is as follows:
pchaigno commentedon Apr 9, 2026
@jackmaninov I'm not able to reproduce. I was able to extract the BPF programs from the bugtool report, but they don't cause any invariant violation warning when running on v6.18.18 or v6.18.7. I've also setup a cluster with the same config you have but it works fine and doesn't cause any kernel warning either. Maybe @mschirrmeister has had more luck?
bpf: Improve bounds when tnum has a single possible value
pchaigno commentedon Apr 16, 2026
FYI, we're expecting this Linux patchset to fix any remaining invariant violations in the kernel. Note this version won't be merged; it requires more iterations.
mschirrmeister commentedon Apr 17, 2026
I was so far not able to reproduce the issue either. I have no Talos though. I am also on arm64.
I was testing on a 3 node cluster (K3s) with Cilium
1.19.2were each node has a different kernel. My initial tests were on6.18.11,6.9.21and7.0-rc7. Have also tried to latest stable release of all 3 versions.My own config was different compared to @jackmaninov posted config. I changed almost everything step by step and it never crashed when cilium agents were restarted. Also no issues on node reboots.
Below is the current diff of my config vs the config that was posted earlier.
I am out of ideas what else I can test at this point. Not sure if a specific workload can influence it?
jackmaninov commentedon Apr 26, 2026
Ok, I can report that upgrading to talos 1.12.7 has somehow cleared this warning for me. They bumped to kernel 6.18.24, but I'm afraid why this would resolve this warning is above my pay grade. I suppose it was a talos-specific bug rather than cilium itself.
pchaigno commentedon May 26, 2026
Closing this as we don't seem to have anyone able to reproduce anymore. There are still improvements planned on the kernel side related to this, but I don't think it needs to be tracked here. Thanks everyone for reporting and testing fixes!
bpf: Improve bounds when tnum has a single possible value