Seems like vmlinux spec keeps being a problem! Just checking: when you say memory usage you mean heap at idle? Calling https://pkg.go.dev/github.com/cilium/ebpf/btf#FlushKernelSpec does not help?

How would you determine which types to parse from vmlinux?

Seems like vmlinux spec keeps being a problem! Just checking: when you say memory usage you mean heap at idle? Calling https://pkg.go.dev/github.com/cilium/ebpf/btf#FlushKernelSpec does not help?

Yes, that would help for having a lower heap usage after starting the program. But if one sets a low memory limit in a pod spec, one also needs to avoid high memory while initializing -> while loading every program.

Furthermore with FlushKernelSpec we need to find and identify every possible call into the ebpf library which might parse and save the while KernelSpec. With a flag we can set the flag and forget about flushing the kernel spec.

How would you determine which types to parse from vmlinux?

For that we are reading the all relo.TypeNames out of the program where the relocations are getting applied: https://github.com/cilium/ebpf/pull/1755/files#diff-981ef293a9c93614e843135eb5b207f951babe5604aee33c72f66567ccaa01de

@lmb To me it sounds like lazy-decoding could've been a better avenue to explore after all?

Not sure how (in)feasible it is today, but iirc we had btf.Spec.Add() back in the day which was a blocker. Now we have btf.Builder, we could technically, hypothetically, make btf.Spec a querying layer over an encoded btf blob and only inflate what's queried, and cache the results to enable type comparisons. Or, implement comparers on all types if we don't want to cache anything. Seems like some (most?) users care more about keeping both resident and peak memory usage low rather than speed.

My main concern is / was complexity of a lazy decoder. The whole "fixups" concept needs to be redone... I think you are right that peak usage seems more important.

I see two avenues: there is some perf to be gained by not unmarshaling into an interface for rawType, I think. Two is the lazy decode you mentioned. I still have some old proof of concepts lying around, I'll push those somewhere.

@burak-ok could you come up with a list of types which you most frequently need from vmlinux? That way we can add a benchmark we can start optimising against. Right now that benchmark is decoding all of vmlinux which isn't useful.

I hope the following helps:

Here is a list from a single program which gets loaded every time for Inspektor Gadget:

syscall_trace_enter
task_struct
nsproxy
mnt_namespace

A list of from multiple programs combined which gets loaded every time for Inspektor Gadget:

pt_regs
file
inode
super_block
socket
syscall_trace_enter
task_struct
nsproxy
mnt_namespace
fanotify_event
pid
trace_event_raw_sched_process_exec

And another list from multiple programs, which get loaded every time and 4 gadgets(trace_tcp, trace_dns, trace_exec, top_file):

pt_regs
file
inode
super_block
socket
syscall_trace_enter
task_struct
nsproxy
mnt_namespace
fanotify_event
pid
trace_event_raw_sched_process_exec
fs_struct
path
mount
qstr
vfsmount
dentry
bpf_func_id
mm_struct
syscall_trace_exit
linux_binprm
sock
net
inet_sock

@burak-ok can you take a look at #1763? Doesn't address FlushKernelSpec, but I think that might be better done by relying on weak in Go 1.24.

@burak-ok can you take a look at #1763?

I added roughly the same benchmarks that you posted for this PR in the top post.
Comparing these results your PR saves roughly the same amount of memory and allocations for the InspektorGadget benchmark.
But the ParseVmlinux Benchmark shows that my Draft PR has more disadvantages for the general usecase of reading and parsing every type.

I'll take a deeper look into your PR some time later, thanks for opening it.

Doesn't address FlushKernelSpec, but I think that might be better done by relying on weak in Go 1.24.

With having memory limits I think this would be the best case scenario for us.

The new lazy BTF code is in. @burak-ok could you try the code and report back how much of a difference it makes?