Repository navigation
Replies: 4 comments 16 replies
|
Thank you for the detailed report. Everything seems in place, I see no obvious mistakes or errors. I suggest checking with https://github.com/cilium/pwru using the |
|
I figured out --filter-trace-tc; I think the most confusing part here is the output hasn't appreciably changed: That seems like the packet hasn't touched the TC/BPF programs at all? |
|
Changing the command slightly: we do see it at least hitting the program we expect, cil_from_netdev, but not going any further. (I ran that without the specific |
|
That is helpful. The L2 responder code in cil_from_netdev should normally turn the ARP request into an ARP response and send it back out. But instead we seem to be passing the ARP response to the stack. That narrows it down. That would happen if Assuming the second, that would happen if the From everything I can see in the sysdump, the agent liveness value seems to get updated. So that would leave the entries in the ARP responder map not matching the ARP request. Looking at your original message you are trying to arp |
|
Apologies! The IP addressing is all over the place here. As part of my debugging (before that sysdump) I had been switching back and forth between 5.x and 6.x to see if there was a problem with arp caching. Generally, The L2 network is 192.168.4.0/22 Using a slightly modified announcement policy from before, I've limited everything to one host for debugging purposes: apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
spec:
loadBalancerIPs: true
nodeSelector:
matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- rk101
With the pool: And produced this new dump: same results with pwru. That's great RE: the agent check. That got me thinking, this looks pretty suspicious: Should those roughly be the same? Or same order of magnitude? I do see the agent liveness climbing, so it does seem like the liveness-updater is working -- Another thought, while we're talking about clock synchronization. My OS (definitely not a problem with cilium!) seems to have trouble with the RTC clock, and I will (not 100% of the time, but sometimes) see odd things like: (NOTE: not the same node). It looks like the first time cilium starts, the clock might be start at Jan 1 1970. Things like Looking at that agent... assuming nanoseconds, I get that it's I'm confused though, because as you mentioned/linked to, the agent liveness checker is updated constantly (by what should be the equivalent of ktime_get_ns() Lines 154 to 159 in 1cf856b |
|
I deployed this to see what the timers are: package main
import (
"bufio"
"fmt"
"log"
"os"
"runtime"
"time"
"github.com/cilium/cilium/pkg/bpf"
"golang.org/x/sys/unix"
)
func main() {
t, err := getBoottime()
if err != nil {
log.Fatal(err)
}
log.Printf("getBoottime: %+v", t)
mtime, err := bpf.GetMtime()
if err != nil {
log.Fatal(err)
}
log.Printf("GetMtime: %+v", mtime)
}
func getBoottime() (t time.Time, err error) {
// copy and pasted from source since it's an unexported fucntion
}(using v1.16.5 source) the boot time is likely accurate, Mtime I'm not sure how to judge. |
|
Also adding in: var ts unix.Timespec
if err := unix.ClockGettime(unix.CLOCK_MONOTONIC, &ts); err != nil {
log.Fatal(err)
}
log.Printf("CLOCK_MONOTONIC: %+v", ts)for good measure, and printing that out with the configmap list: Assuming [edit] it sounds like cgroups could change this value, but: (from the cilium pod) |
|
Curios-er and curios-er: arp responses are now working 😮 I'm guessing this means there's nothing wrong with the timestamps above, if native routing fixed things 🤔 I'm even more confused on what possible OS setting could have changed where routingMode: tunnel would work vs not work now (all of this started with an upgrade from talos 1.8.x to talos 1.9.x). [edit2] it certainly felt like it was working, and no longer is. I might have misinterpreted the gARP for the arp response |
|
Okay, now I really do have it down to one option that takes us from working to not working: I can flip that between best-effort and disabled all day long, and arping will work w/ it set to best-effort and will not when it's disabled. I'll read a bit more about xdp drivers in a bit, but this feels like a real good find Incase there's something else I'm doing that's causing the xdp driver to help my situation, this is what my helm values looks like (using chart version 1.16.5): The non-defaults here are / for the reasons:
|
|
I doubt it's all that helpful now, but just for comparison: this is pwru when it works w/ the load balancer mode best-effort! |
That is a really good find. So... with XDP we answer ARP in the XDP program, without it in TC which is later in the network stack. So that sort of narrows it down to the ARP being processed and dropped by the kernel between XDP and TC. Which is odd, since as far as I know ARP should be processed after TC (hence why this feature works most of the time). So that sort of narrows it down to the kernel. Not sure if its to do with kernel version, kernel config or something Talos did to their kernel. In your very first message you said |
|
I realized the first dump didn't have one of the hosts running the older version, the way I intended :'( We went from talos 1.8.2 to 1.9.x, the kernel on 1.8.2 was Talos 1.9.x is running: And yeah, it works on the 6.6 kernel with xdp acceleration disabled! (I just reverted one of my 4 nodes, so nodes 1-3 are running 1.9.x with 6.12, and node 4 is running 1.8.x with 6.6 so I can reproduce the "working" tc mode) |
|
Looking at that custom kernel, that was in this repo: I just found this highly suspicious commit nberlee/kernel@d3b8208
(I can't tell if the linked commit is in my particular version or not, but the linked commit is reverting that patch. The wording is very suspicious) I'll upload another sysdump shortly so we can see the full diff between the two OS versions; last I looked at the diff between kernel config there wasn't much interesting, except netfilter (which should be disabled, and was introduced between these two kernel versions I think?) [edit] 6.12 looks like it has the final version of that commit, the way I think it's ultimately meant to end up? |
|
In this dump, we have two hosts:
|
|
Rough estimate on interesting things we'd want to look at in config diff: (left hand side is working talos 1.8.x rk104, right hand is broken rk103 on 1.9.x) |
|
Thanks a ton for going to the lengths of debugging this. I've been revisiting this exact issue in my homelab on and off for months, and never found a solution (until now). |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hello!
I'm having trouble with L2 Announcements after having upgraded my cluster OS to talos-1.9.0. I have a sneaking suspicion there's a kernel misconfig (although everything in https://docs.cilium.io/en/stable/operations/system_requirements/ checks out.
CONFIG_IP_NF_IPTABLES_LEGACY=yis the oddest change between the two kernels), but am attempting to further debug.I've followed the troubleshooting steps in https://docs.cilium.io/en/stable/network/l2-announcements/#troubleshooting and have gotten to the point where the map
cilium_l2_responder_v4shows no responses.(But.. curiously, the working node shows the same responses_sent: 0, and there's still frequent ARP responses)
With a bit of tcpdumping, I can see that the gratuitous ARP happens when grabbing a lease, but not any replies to ARP requests (the ARP requests make it to the physical device).
(that's even from the same host namespace!)
I've read through some of the code
cilium/bpf/bpf_host.c
Line 1054 in 0aa1204
How can I debug further? I think I need a better understanding of how the IP addressing works in Cilium. Is there something else besides the bpf program that tells the network subsystem to not drop packets destined for other IP addresses? I know that's a pretty common feature, I'm wondering if the packet is getting dropped before BPF (and how I would debug something like that)
Is there something besides bpf_host.c that would be responding to ARP requests for these IPs? One other odd thing I've noted: on my working host (out of 4, I have 1 running Talos 1.8.3 with a custom kernel for my hardware; the reason I want to upgrade to 1.9.0 is because of upstream kernel support) I still see "0" responses in the bpf map:
I hope the output of this discussion turns into:
--
I have a dump, but would appreciate debugging advice! cilium-sysdump-20241226-201519.zip
All reactions