Skip to content

fix(libcap): bump libcap to v1.2.75 & libpsx to v1.2.76-rc1#4688

Merged
github-actions[bot] merged 3 commits into
aquasecurity:mainfrom
geyslan:4678-fix-libcap
Mar 31, 2025
Merged

fix(libcap): bump libcap to v1.2.75 & libpsx to v1.2.76-rc1#4688
github-actions[bot] merged 3 commits into
aquasecurity:mainfrom
geyslan:4678-fix-libcap

Conversation

@geyslan
Copy link
Copy Markdown
Member

@geyslan geyslan commented Mar 28, 2025

Close: #4678

1. Explain what the PR does

fd21867 fix(deps): bump psx to v1.2.76-rc1
fa56872 fix(deps): bump to libbpfgo using libcap v1.2.75
7fd4519 fix(deps): bump libcap to v1.2.75

fd21867 fix(deps): bump psx to v1.2.76-rc1

- https://github.com/aquasecurity/tracee/pull/4688#issuecomment-2764255447
- https://web.git.kernel.org/pub/scm/libs/libcap/libcap.git/commit/?h=psx/v1.2.76-rc1&id=07d8ce731d5fe9063abfef4a77306e273b18b5f3

7fd4519 fix(deps): bump libcap to v1.2.75

The regression introduced in v1.2.72
https://bugzilla.kernel.org/show_bug.cgi?id=219687

was fixed since 1.2.74
https://web.git.kernel.org/pub/scm/libs/libcap/libcap.git/commit/?id=025f28ca4fe085fbcbf7933d53a42d335744e553

Nevertheless, this bumps libcap to v1.2.75 to match with the upstream
libbpfgo version.

2. Explain how to test it

3. Other comments

Copy link
Copy Markdown
Contributor

@rscampos rscampos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good finding @geyslan ... after applying the fix in libcap and conducting 61 runs, none of them exhibited the issue.
LGTM

@geyslan
Copy link
Copy Markdown
Member Author

geyslan commented Mar 28, 2025

x86_64 kernels tests passed with glory but libcap v1.2.75 seems to be failing on arm64:

some failures
munmap_chunk(): invalid pointer
SIGABRT: abort
PC=0xffff9118f200 m=7 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 1 gp=0x4000002380 m=7 mp=0x4001b7a808 [syscall]:
runtime.cgocall(0x1bde0c0, 0x4000054968)
	/tmp/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:167 +0x44 fp=0x4000054930 sp=0x40000548f0 pc=0xd00074
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0xffff386f3010, 0xedc9d8, 0xffff28000bb0)
	_cgo_gotypes.go:519 +0x38 fp=0x4000054960 sp=0x4000054930 pc=0x113de98
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func7(...)
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x40018d67b0, 0x2b}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x4001e80000, 0xedc9d8, ...}, ...})
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154 +0x364 fp=0x4000054bd0 sp=0x4000054960 pc=0x114ae64
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).initBPFProbes(0x40004de608)
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:1256 +0x9c fp=0x4000054ca0 sp=0x4000054bd0 pc=0x1aa3c4c
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init.func2()
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:357 +0x20 fp=0x4000054cc0 sp=0x4000054ca0 pc=0x1a9d5f0
github.com/aquasecurity/tracee/pkg/capabilities.(*Capabilities).EBPF(0x4001dfcce0, 0x4000055050)
	/actions-runner/_work/tracee/tracee/pkg/capabilities/capabilities.go:235 +0xe0 fp=0x4000054d20 sp=0x4000054cc0 pc=0x1186710
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init(0x40004de608, {0x31e0700, 0x400031e900})
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:355 +0x490 fp=0x4000055120 sp=0x4000054d20 pc=0x1a9caf0
SIGILL: illegal instruction
PC=0xffffa353cc90 m=4 sigcode=1
signal arrived during cgo execution
instruction bytes: 0x10 0x0 0x0 0x0 0xf4 0x2f 0x0 0x0 0x58 0xe0 0xef 0xff 0xc 0x0 0x0 0x0

goroutine 1 gp=0x4000002380 m=4 mp=0x4000075808 [syscall]:
runtime.cgocall(0x1bde0c0, 0x4000054968)
	/tmp/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:167 +0x44 fp=0x4000054930 sp=0x40000548f0 pc=0xd00074
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0xffff3f123010, 0xedc9d8, 0xffff48002db0)
	_cgo_gotypes.go:519 +0x38 fp=0x4000054960 sp=0x4000054930 pc=0x113de98
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func7(...)
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x4001d4e460, 0x1e}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x4001f00000, 0xedc9d8, ...}, ...})
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154 +0x364 fp=0x4000054bd0 sp=0x4000054960 pc=0x114ae64
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).initBPFProbes(0x40004d0608)
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:1256 +0x9c fp=0x4000054ca0 sp=0x4000054bd0 pc=0x1aa3c4c
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init.func2()
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:357 +0x20 fp=0x4000054cc0 sp=0x4000054ca0 pc=0x1a9d5f0
github.com/aquasecurity/tracee/pkg/capabilities.(*Capabilities).EBPF(0x40018a6000, 0x4000055050)
	/actions-runner/_work/tracee/tracee/pkg/capabilities/capabilities.go:235 +0xe0 fp=0x4000054d20 sp=0x4000054cc0 pc=0x1186710
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init(0x40004d0608, {0x31e0860, 0x40000c3240})
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:355 +0x490 fp=0x4000055120 sp=0x4000054d20 pc=0x1a9caf0
SIGABRT: abort
PC=0xffffaa89f200 m=7 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 1 gp=0x4000002380 m=7 mp=0x4001cfc008 [syscall]:
runtime.cgocall(0x1bde0c0, 0x4000054968)
	/tmp/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:167 +0x44 fp=0x4000054930 sp=0x40000548f0 pc=0xd00074
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0xffff4a913010, 0xedc9d8, 0xffff44000b70)
	_cgo_gotypes.go:519 +0x38 fp=0x4000054960 sp=0x4000054930 pc=0x113de98
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func7(...)
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x20ad6a8, 0xf}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x4001e00000, 0xedc9d8, ...}, ...})
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154 +0x364 fp=0x4000054bd0 sp=0x4000054960 pc=0x114ae64
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).initBPFProbes(0x40004dc608)
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:1256 +0x9c fp=0x4000054ca0 sp=0x4000054bd0 pc=0x1aa3c4c
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init.func2()
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:357 +0x20 fp=0x4000054cc0 sp=0x4000054ca0 pc=0x1a9d5f0
github.com/aquasecurity/tracee/pkg/capabilities.(*Capabilities).EBPF(0x4001de0a60, 0x4000055050)
	/actions-runner/_work/tracee/tracee/pkg/capabilities/capabilities.go:235 +0xe0 fp=0x4000054d20 sp=0x4000054cc0 pc=0x1186710
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init(0x40004dc608, {0x31e0700, 0x400031e8c0})
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:355 +0x490 fp=0x4000055120 sp=0x4000054d20 pc=0x1a9caf0
SIGABRT: abort
PC=0xffffadc8f200 m=6 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 1 gp=0x4000002380 m=6 mp=0x40004bf008 [syscall]:
runtime.cgocall(0x1bde0c0, 0x4000054968)
	/tmp/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:167 +0x44 fp=0x4000054930 sp=0x40000548f0 pc=0xd00074
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0xffff56103010, 0xedc9d8, 0xffff4c000d10)
	_cgo_gotypes.go:519 +0x38 fp=0x4000054960 sp=0x4000054930 pc=0x113de98
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func7(...)
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x4001af1e40, 0x1e}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x4001f00000, 0xedc9d8, ...}, ...})
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154 +0x364 fp=0x4000054bd0 sp=0x4000054960 pc=0x114ae64
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).initBPFProbes(0x40000fcf08)
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:1256 +0x9c fp=0x4000054ca0 sp=0x4000054bd0 pc=0x1aa3c4c
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init.func2()
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:357 +0x20 fp=0x4000054cc0 sp=0x4000054ca0 pc=0x1a9d5f0
github.com/aquasecurity/tracee/pkg/capabilities.(*Capabilities).EBPF(0x4001ebad40, 0x4000055050)
	/actions-runner/_work/tracee/tracee/pkg/capabilities/capabilities.go:235 +0xe0 fp=0x4000054d20 sp=0x4000054cc0 pc=0x1186710
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init(0x40000fcf08, {0x31e0700, 0x400031ed00})
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:355 +0x490 fp=0x4000055120 sp=0x4000054d20 pc=0x1a9caf0
*** stack smashing detected ***: terminated
SIGABRT: abort
PC=0xffffb827f200 m=0 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 1 gp=0x4000002380 m=0 mp=0x3f52380 [syscall]:
runtime.cgocall(0x1bde100, 0x4000054968)
	/tmp/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:167 +0x44 fp=0x4000054930 sp=0x40000548f0 pc=0xd000b4
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0xffff686f3010, 0xedc9d8, 0x28dc2160)
	_cgo_gotypes.go:519 +0x38 fp=0x4000054960 sp=0x4000054930 pc=0x113ded8
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func7(...)
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x4001b30e40, 0x1e}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x4001f00000, 0xedc9d8, ...}, ...})
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154 +0x364 fp=0x4000054bd0 sp=0x4000054960 pc=0x114aea4
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).initBPFProbes(0x40004de608)
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:1256 +0x9c fp=0x4000054ca0 sp=0x4000054bd0 pc=0x1aa3c8c
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init.func2()
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:357 +0x20 fp=0x4000054cc0 sp=0x4000054ca0 pc=0x1a9d630
github.com/aquasecurity/tracee/pkg/capabilities.(*Capabilities).EBPF(0x40000ba000, 0x4000055050)
	/actions-runner/_work/tracee/tracee/pkg/capabilities/capabilities.go:235 +0xe0 fp=0x4000054d20 sp=0x4000054cc0 pc=0x1186750
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init(0x40004de608, {0x31e0740, 0x4000105240})
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:355 +0x490 fp=0x4000055120 sp=0x4000054d20 pc=0x1a9cb30
SIGSEGV: segmentation violation
PC=0xffffbad8bac4 m=7 sigcode=1 addr=0xffffffffffffffff
signal arrived during cgo execution

goroutine 1 gp=0x4000002380 m=7 mp=0x4001eaa808 [syscall]:
runtime.cgocall(0x1c9cf80, 0x4000054968)
	/tmp/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:167 +0x44 fp=0x4000054930 sp=0x40000548f0 pc=0xdbef34
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0xffff5a913010, 0xedc9d8, 0xffff54000d10)
	_cgo_gotypes.go:519 +0x38 fp=0x4000054960 sp=0x4000054930 pc=0x11fcd58
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func7(...)
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x4001b48e60, 0x1d}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x4001f00000, 0xedc9d8, ...}, ...})
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154 +0x364 fp=0x4000054bd0 sp=0x4000054960 pc=0x1209d24
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).initBPFProbes(0x40004de608)
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:1256 +0x9c fp=0x4000054ca0 sp=0x4000054bd0 pc=0x1b62b0c
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init.func2()
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:357 +0x20 fp=0x4000054cc0 sp=0x4000054ca0 pc=0x1b5c4b0
github.com/aquasecurity/tracee/pkg/capabilities.(*Capabilities).EBPF(0x40000aa000, 0x4000055050)
	/actions-runner/_work/tracee/tracee/pkg/capabilities/capabilities.go:235 +0xe0 fp=0x4000054d20 sp=0x4000054cc0 pc=0x12455d0
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init(0x40004de608, {0x329f5c0, 0x400013d240})
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:355 +0x490 fp=0x4000055120 sp=0x4000054d20 pc=0x1b5b9b0
SIGSEGV: segmentation violation
PC=0xffffa4bbf1d8 m=0 sigcode=1 addr=0x54918
signal arrived during cgo execution

goroutine 1 gp=0x4000002380 m=0 mp=0x4012380 [syscall]:
runtime.cgocall(0x1c9d080, 0x4000054968)
	/tmp/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:167 +0x44 fp=0x4000054930 sp=0x40000548f0 pc=0xdbf034
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0xffff4cee3010, 0xedc9d8, 0x64f2160)
	_cgo_gotypes.go:519 +0x38 fp=0x4000054960 sp=0x4000054930 pc=0x11fce58
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func7(...)
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x4001b4b220, 0x1c}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x4001f00000, 0xedc9d8, ...}, ...})
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154 +0x364 fp=0x4000054bd0 sp=0x4000054960 pc=0x1209e24
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).initBPFProbes(0x40004c0608)
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:1256 +0x9c fp=0x4000054ca0 sp=0x4000054bd0 pc=0x1b62c0c
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init.func2()
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:357 +0x20 fp=0x4000054cc0 sp=0x4000054ca0 pc=0x1b5c5b0
github.com/aquasecurity/tracee/pkg/capabilities.(*Capabilities).EBPF(0x4001896000, 0x4000055050)
	/actions-runner/_work/tracee/tracee/pkg/capabilities/capabilities.go:235 +0xe0 fp=0x4000054d20 sp=0x4000054cc0 pc=0x12456d0
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init(0x40004c0608, {0x329f6c0, 0x4000132780})
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:355 +0x490 fp=0x4000055120 sp=0x4000054d20 pc=0x1b5bab0
SIGSEGV: segmentation violation
PC=0xf233ff70f1d8 m=6 sigcode=1 addr=0x54918
signal arrived during cgo execution

goroutine 1 gp=0x4000002380 m=6 mp=0x40002c5008 [syscall]:
runtime.cgocall(0x1c9d080, 0x4000054968)
	/tmp/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:167 +0x44 fp=0x4000054930 sp=0x40000548f0 pc=0xdbf034
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0xf233b05d3010, 0xedc9d8, 0xf2339c000ba0)
	_cgo_gotypes.go:519 +0x38 fp=0x4000054960 sp=0x4000054930 pc=0x11fce58
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func7(...)
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x4001b03480, 0x1d}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x4001f00000, 0xedc9d8, ...}, ...})
	/tmp/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154 +0x364 fp=0x4000054bd0 sp=0x4000054960 pc=0x1209e24
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).initBPFProbes(0x4000494f08)
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:1256 +0x9c fp=0x4000054ca0 sp=0x4000054bd0 pc=0x1b62c0c
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init.func2()
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:357 +0x20 fp=0x4000054cc0 sp=0x4000054ca0 pc=0x1b5c5b0
github.com/aquasecurity/tracee/pkg/capabilities.(*Capabilities).EBPF(0x4001ea2e60, 0x4000055050)
	/actions-runner/_work/tracee/tracee/pkg/capabilities/capabilities.go:235 +0xe0 fp=0x4000054d20 sp=0x4000054cc0 pc=0x12456d0
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init(0x4000494f08, {0x329f6c0, 0x400027c140})
	/actions-runner/_work/tracee/tracee/pkg/ebpf/tracee.go:355 +0x490 fp=0x4000055120 sp=0x4000054d20 pc=0x1b5bab0

@rscampos, if there are no RC fixes addressing this architecture, I think we should revert libbpfgo and Tracee to versions prior to libcap v1.2.72.

Hi there @AndrewGMorgan, have you encountered a similar issue on arm64 following the refactoring mentioned in https://bugzilla.kernel.org/show_bug.cgi?id=219687? Cheers.

@AndrewGMorgan
Copy link
Copy Markdown

Definitely revert to 1.2.71 for stability. 1.2.75 should be fixed for aarch64, so I am concerned this is a new issue.

@AndrewGMorgan AndrewGMorgan mentioned this pull request Mar 29, 2025
@AndrewGMorgan
Copy link
Copy Markdown

Are we at a stage where we understand this to be an issue with cap/v1.2.75 that is different from the one described in #4678 ? I am confident that that bug should have been fixed with 1.2.75.

Based on the crash dumps visible in this bug from the above test runs ("> some failures"), I don't see any evidence that the psx or cap Go packages are in any of the stack traces. Perhaps someone can help me understand what points the finger at these libcap packages?

@AndrewGMorgan
Copy link
Copy Markdown

I've just run my libcap regression tests on an rpi5 with a aarch64/arm64 Raspbian. All the packaging tests pass as expected with libcap-2.75 aka cap/v1.2.75.

The stack traces of the failures in this present bug all seem to implicate something related to mmap/munmap. I don't feel confident absolving libcap and friends without really understanding tracee, but could this issue be something elsewhere?

@geyslan
Copy link
Copy Markdown
Member Author

geyslan commented Mar 29, 2025

I don't feel confident absolving libcap and friends without really understanding tracee, but could this issue be something elsewhere?

Firstly I thought that could be a mismatch of the libbpfgo (using an older libcap) but, as you can see, this PR bumped libbpfgo to its latest commit aquasecurity/libbpfgo@c412f16 which is a bump per si.

@geyslan
Copy link
Copy Markdown
Member Author

geyslan commented Mar 29, 2025

Definitely revert to 1.2.71 for stability. 1.2.75 should be fixed for aarch64, so I am concerned this is a new issue.

As you suggested, @AndrewGMorgan, we're going that way through other PR. But we're going to let this open for discussion and debugging. BTW, thanks for the amazing libcap and family.

@geyslan
Copy link
Copy Markdown
Member Author

geyslan commented Mar 29, 2025

I don't feel confident absolving libcap and friends without really understanding tracee, but could this issue be something elsewhere?

We're using go toolchain 1.24.1

toolchain go1.24.1

libcap regression tests are based on which toolchain?

@AndrewGMorgan
Copy link
Copy Markdown

I regression test against all go tool chains (top of each release from go1.11.13 forward. The current set is:

go1.11.13  go1.12.17  go1.13.15  go1.14.15  go1.15.15  go1.16.15  go1.17.13  go1.18.10  go1.19.12  go1.20.14  go1.21.13  go1.22.12  go1.23.7  go1.24.1

@geyslan
Copy link
Copy Markdown
Member Author

geyslan commented Mar 29, 2025

The dumps come from Tracee cmd.

Build flags: https://github.com/aquasecurity/tracee/actions/runs/14138332508/job/39615080974#step:4:1139

GOOS=linux CC=clang GOARCH=arm64 CGO_CFLAGS="-I/actions-runner/_work/tracee/tracee/dist/libbpf/include" CGO_LDFLAGS="-L/actions-runner/_work/tracee/tracee/dist/libbpf/obj -lbpf" go build \
	-tags core,ebpf \
	-ldflags="-w \
		-extldflags \"\" \
		-X github.com/aquasecurity/tracee/pkg/version.version=-9696a4e \
		-X github.com/aquasecurity/tracee/pkg/version.metrics= \
		" \
	-v -o dist/tracee \
	./cmd/tracee

--- EDIT

$ clang --version
Ubuntu clang version 14.0.6
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@geyslan
Copy link
Copy Markdown
Member Author

geyslan commented Mar 29, 2025

As listed above the crashes occur exclusively on the following aarch64 kernel versions supported by Tracee: AMZN2 5.10 aarch64, Focal 5.13 aarch64, GKE 5.15 aarch64, Jammy 5.15 aarch64, Lunar 6.2 aarch64, Mantic 6.5 aarch64, and Mantic 6.6 aarch64.

However, it appears to work correctly on these versions: Noble 6.8 aarch64, Noble 6.10 aarch64, Noble 6.11 aarch64, and Noble 6.12 aarch64.

Could this be due to a (possible) change in the aarch64 signaling mechanism introduced between versions 6.6 and 6.8?


Follow an entire crash dump.

$ uname -a
Linux ip-172-31-28-167 6.6.0-14-generic #14-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 30 10:53:08 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
$ sudo ./dist/tracee
The futex facility returned an unexpected error code.
SIGABRT: abort
PC=0xefd6af3353f8 m=6 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 1 gp=0x40000021c0 m=6 mp=0x4001ee4008 [syscall]:
runtime.cgocall(0x1c9d080, 0x400004c968)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:167 +0x44 fp=0x400004c930 sp=0x400004c8f0 pc=0xdbf034
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0xefd664283010, 0xedc838, 0xefd648000ba0)
        _cgo_gotypes.go:519 +0x38 fp=0x400004c960 sp=0x400004c930 pc=0x11fce58
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func7(...)
        /home/ubuntu/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x4001afac00, 0x1d}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x4001f00000, 0xedc838, ...}, ...})
        /home/ubuntu/go/pkg/mod/github.com/aquasecurity/[email protected]/module.go:154 +0x364 fp=0x400004cbd0 sp=0x400004c960 pc=0x1209e24
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).initBPFProbes(0x40004da608)
        /home/ubuntu/tracee/pkg/ebpf/tracee.go:1256 +0x9c fp=0x400004cca0 sp=0x400004cbd0 pc=0x1b62c0c
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init.func2()
        /home/ubuntu/tracee/pkg/ebpf/tracee.go:357 +0x20 fp=0x400004ccc0 sp=0x400004cca0 pc=0x1b5c5b0
github.com/aquasecurity/tracee/pkg/capabilities.(*Capabilities).EBPF(0x4001e79b20, 0x400004d050)
        /home/ubuntu/tracee/pkg/capabilities/capabilities.go:235 +0xe0 fp=0x400004cd20 sp=0x400004ccc0 pc=0x12456d0
github.com/aquasecurity/tracee/pkg/ebpf.(*Tracee).Init(0x40004da608, {0x329f520, 0x400013ac00})
        /home/ubuntu/tracee/pkg/ebpf/tracee.go:355 +0x490 fp=0x400004d120 sp=0x400004cd20 pc=0x1b5bab0
github.com/aquasecurity/tracee/pkg/cmd.Runner.Run({{{0x40005610b0, 0x1, 0x1}, 0x40005aa230, 0x4000449300, 0x4001ae11b8, {0x0, 0x0}, {0x0, 0x2ab0, ...}, ...}, ...}, ...)
        /home/ubuntu/tracee/pkg/cmd/tracee.go:62 +0x190 fp=0x400004d7d0 sp=0x400004d120 pc=0x1c310e0
github.com/aquasecurity/tracee/cmd/tracee/cmd.init.func14(0x3f21b60, {0x400012e060?, 0x4?, 0x214825b?})
        /home/ubuntu/tracee/cmd/tracee/cmd/root.go:64 +0x288 fp=0x400004dc50 sp=0x400004d7d0 pc=0x1c45c98
github.com/spf13/cobra.(*Command).execute(0x3f21b60, {0x400012e060, 0x0, 0x0})
        /home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:1019 +0x810 fp=0x400004dde0 sp=0x400004dc50 pc=0xf6e230
github.com/spf13/cobra.(*Command).ExecuteC(0x3f21b60)
        /home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:1148 +0x350 fp=0x400004dec0 sp=0x400004dde0 pc=0xf6ea70
github.com/spf13/cobra.(*Command).Execute(...)
        /home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:1071
github.com/aquasecurity/tracee/cmd/tracee/cmd.Execute()
        /home/ubuntu/tracee/cmd/tracee/cmd/root.go:381 +0x38 fp=0x400004dee0 sp=0x400004dec0 pc=0x1c47e58
main.main()
        /home/ubuntu/tracee/cmd/tracee/main.go:11 +0x1c fp=0x400004df40 sp=0x400004dee0 pc=0x1c4814c
runtime.main()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:283 +0x284 fp=0x400004dfd0 sp=0x400004df40 pc=0xd8b624
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x400004dfd0 sp=0x400004dfd0 pc=0xdcb2b4

goroutine 2 gp=0x4000002c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000066f90 sp=0x4000066f70 pc=0xdc30b8
runtime.goparkunlock(...)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:441
runtime.forcegchelper()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:348 +0xb8 fp=0x4000066fd0 sp=0x4000066f90 pc=0xd8b978
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000066fd0 sp=0x4000066fd0 pc=0xdcb2b4
created by runtime.init.7 in goroutine 1
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:336 +0x24

goroutine 3 gp=0x4000003180 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000067760 sp=0x4000067740 pc=0xdc30b8
runtime.goparkunlock(...)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:441
runtime.bgsweep(0x4000092000)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgcsweep.go:316 +0x108 fp=0x40000677b0 sp=0x4000067760 pc=0xd71e18
runtime.gcenable.gowrap1()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:204 +0x28 fp=0x40000677d0 sp=0x40000677b0 pc=0xd65ad8
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000677d0 sp=0x40000677d0 pc=0xdcb2b4
created by runtime.gcenable in goroutine 1
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:204 +0x6c

goroutine 4 gp=0x4000003340 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x327a1c8?, 0x0?, 0x0?, 0x0?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000067f60 sp=0x4000067f40 pc=0xdc30b8
runtime.goparkunlock(...)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:441
runtime.(*scavengerState).park(0x4010860)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgcscavenge.go:425 +0x5c fp=0x4000067f90 sp=0x4000067f60 pc=0xd6f7ac
runtime.bgscavenge(0x4000092000)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgcscavenge.go:658 +0xac fp=0x4000067fb0 sp=0x4000067f90 pc=0xd6fd4c
runtime.gcenable.gowrap2()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:205 +0x28 fp=0x4000067fd0 sp=0x4000067fb0 pc=0xd65a78
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000067fd0 sp=0x4000067fd0 pc=0xdcb2b4
created by runtime.gcenable in goroutine 1
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:205 +0xac

goroutine 18 gp=0x4000104380 m=nil [finalizer wait]:
runtime.gopark(0x1f31120?, 0x12?, 0xc0?, 0x5?, 0x1000000000000?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000066590 sp=0x4000066570 pc=0xdc30b8
runtime.runfinq()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mfinal.go:196 +0x108 fp=0x40000667d0 sp=0x4000066590 pc=0xd64ad8
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000667d0 sp=0x40000667d0 pc=0xdcb2b4
created by runtime.createfing in goroutine 1
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mfinal.go:166 +0x80

goroutine 19 gp=0x4000104fc0 m=nil [chan receive]:
runtime.gopark(0x4000343720?, 0x400000e030?, 0x48?, 0x27?, 0xe950a8?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x40000626f0 sp=0x40000626d0 pc=0xdc30b8
runtime.chanrecv(0x4000100af0, 0x0, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/chan.go:664 +0x42c fp=0x4000062770 sp=0x40000626f0 pc=0xd5667c
runtime.chanrecv1(0x0?, 0x0?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/chan.go:506 +0x14 fp=0x40000627a0 sp=0x4000062770 pc=0xd56214
runtime.unique_runtime_registerUniqueMapCleanup.func2(...)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:1796
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:1799 +0x3c fp=0x40000627d0 sp=0x40000627a0 pc=0xd68cfc
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000627d0 sp=0x40000627d0 pc=0xdcb2b4
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:1794 +0x78

goroutine 22 gp=0x4000105880 m=nil [GC worker (idle)]:
runtime.gopark(0x2dd512d55a8?, 0x4000062f60?, 0x1?, 0x0?, 0x4000105880?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000062f10 sp=0x4000062ef0 pc=0xdc30b8
runtime.gcBgMarkWorker(0x4000101110)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:1423 +0xdc fp=0x4000062fb0 sp=0x4000062f10 pc=0xd67f6c
runtime.gcBgMarkStartWorkers.gowrap1()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:1339 +0x28 fp=0x4000062fd0 sp=0x4000062fb0 pc=0xd67e58
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000062fd0 sp=0x4000062fd0 pc=0xdcb2b4
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:1339 +0x140

goroutine 23 gp=0x40005561c0 m=nil [GC worker (idle)]:
runtime.gopark(0x2dd51289fc8?, 0x0?, 0x0?, 0x0?, 0x0?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000063710 sp=0x40000636f0 pc=0xdc30b8
runtime.gcBgMarkWorker(0x4000101110)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:1423 +0xdc fp=0x40000637b0 sp=0x4000063710 pc=0xd67f6c
runtime.gcBgMarkStartWorkers.gowrap1()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:1339 +0x28 fp=0x40000637d0 sp=0x40000637b0 pc=0xd67e58
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000637d0 sp=0x40000637d0 pc=0xdcb2b4
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/mgc.go:1339 +0x140

goroutine 5 gp=0x4000556380 m=nil [select]:
runtime.gopark(0x4000079d98?, 0x2?, 0x0?, 0x0?, 0x4000079d94?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000079a80 sp=0x4000079a60 pc=0xdc30b8
runtime.selectgo(0x4000079d98, 0x4000079d90, 0x0?, 0x0, 0x2?, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:351 +0x6c4 fp=0x4000079bb0 sp=0x4000079a80 pc=0xd9f624
github.com/aquasecurity/tracee/pkg/cmd/printer.startPrinter(0x4001cfe4d0, 0x4001b04310, 0x40005aa4d0, {0x32a2380, 0x4001daac90})
        /home/ubuntu/tracee/pkg/cmd/printer/broadcast.go:98 +0xb8 fp=0x4000079f90 sp=0x4000079bb0 pc=0x1a4e678
github.com/aquasecurity/tracee/pkg/cmd/printer.(*Broadcast).Init.gowrap1()
        /home/ubuntu/tracee/pkg/cmd/printer/broadcast.go:52 +0x38 fp=0x4000079fd0 sp=0x4000079f90 pc=0x1a4e2f8
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000079fd0 sp=0x4000079fd0 pc=0xdcb2b4
created by github.com/aquasecurity/tracee/pkg/cmd/printer.(*Broadcast).Init in goroutine 1
        /home/ubuntu/tracee/pkg/cmd/printer/broadcast.go:52 +0x1ec

goroutine 24 gp=0x4000003880 m=nil [select, locked to thread]:
runtime.gopark(0x40000647a0?, 0x2?, 0x98?, 0x46?, 0x400006478c?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000064610 sp=0x40000645f0 pc=0xdc30b8
runtime.selectgo(0x40000647a0, 0x4000064788, 0x0?, 0x0, 0x0?, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:351 +0x6c4 fp=0x4000064740 sp=0x4000064610 pc=0xd9f624
runtime.ensureSigM.func1()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/signal_unix.go:1085 +0x194 fp=0x40000647d0 sp=0x4000064740 pc=0xdbc954
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000647d0 sp=0x40000647d0 pc=0xdcb2b4
created by runtime.ensureSigM in goroutine 1
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/signal_unix.go:1068 +0xd8

goroutine 6 gp=0x4000556540 m=4 mp=0x400006d808 [syscall]:
runtime.notetsleepg(0x4566460, 0xffffffffffffffff)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/lock_futex.go:123 +0x34 fp=0x4000064f90 sp=0x4000064f60 pc=0xd5b9a4
os/signal.signal_recv()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/sigqueue.go:152 +0x30 fp=0x4000064fb0 sp=0x4000064f90 pc=0xdc5320
os/signal.loop()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/os/signal/signal_unix.go:23 +0x1c fp=0x4000064fd0 sp=0x4000064fb0 pc=0x11c3eac
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000064fd0 sp=0x4000064fd0 pc=0xdcb2b4
created by os/signal.Notify.func1.1 in goroutine 1
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/os/signal/signal.go:152 +0x28

goroutine 7 gp=0x4000556700 m=nil [select]:
runtime.gopark(0x40000657a8?, 0x2?, 0xc0?, 0x5?, 0x4000065794?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x400007be20 sp=0x400007be00 pc=0xdc30b8
runtime.selectgo(0x400007bfa8, 0x4000065790, 0x0?, 0x0, 0x0?, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:351 +0x6c4 fp=0x400007bf50 sp=0x400007be20 pc=0xd9f624
os/signal.NotifyContext.func1()
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/os/signal/signal.go:289 +0x68 fp=0x400007bfd0 sp=0x400007bf50 pc=0x11c3a68
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x400007bfd0 sp=0x400007bfd0 pc=0xdcb2b4
created by os/signal.NotifyContext in goroutine 1
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/os/signal/signal.go:288 +0x148

goroutine 25 gp=0x4000556e00 m=nil [select]:
runtime.gopark(0x4000287f50?, 0x2?, 0x54?, 0x34?, 0x4000287f14?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000287da0 sp=0x4000287d80 pc=0xdc30b8
runtime.selectgo(0x4000287f50, 0x4000287f10, 0x695234622b68?, 0x0, 0x3946624f45?, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:351 +0x6c4 fp=0x4000287ed0 sp=0x4000287da0 pc=0xd9f624
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x4000561030, {0x329f408, 0x40004040a0})
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0xc4 fp=0x4000287fa0 sp=0x4000287ed0 pc=0x130de44
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x30 fp=0x4000287fd0 sp=0x4000287fa0 pc=0x130dc80
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000287fd0 sp=0x4000287fd0 pc=0xdcb2b4
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 26 gp=0x4000556fc0 m=nil [select]:
runtime.gopark(0x4000417f50?, 0x2?, 0xe8?, 0xdd?, 0x4000417f14?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000417da0 sp=0x4000417d80 pc=0xdc30b8
runtime.selectgo(0x4000417f50, 0x4000417f10, 0x0?, 0x0, 0x0?, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:351 +0x6c4 fp=0x4000417ed0 sp=0x4000417da0 pc=0xd9f624
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x4000561060, {0x329f408, 0x40004040f0})
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0xc4 fp=0x4000417fa0 sp=0x4000417ed0 pc=0x130de44
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x30 fp=0x4000417fd0 sp=0x4000417fa0 pc=0x130dc80
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000417fd0 sp=0x4000417fd0 pc=0xdcb2b4
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 27 gp=0x4000557180 m=nil [select]:
runtime.gopark(0x400028df50?, 0x2?, 0x0?, 0x0?, 0x400028df14?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x400028dda0 sp=0x400028dd80 pc=0xdc30b8
runtime.selectgo(0x400028df50, 0x400028df10, 0x0?, 0x0, 0x0?, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:351 +0x6c4 fp=0x400028ded0 sp=0x400028dda0 pc=0xd9f624
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x4000561090, {0x329f408, 0x4000404140})
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0xc4 fp=0x400028dfa0 sp=0x400028ded0 pc=0x130de44
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x30 fp=0x400028dfd0 sp=0x400028dfa0 pc=0x130dc80
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x400028dfd0 sp=0x400028dfd0 pc=0xdcb2b4
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 9 gp=0x4000557340 m=nil [select]:
runtime.gopark(0x4000283f50?, 0x2?, 0x70?, 0x72?, 0x4000283f14?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000077da0 sp=0x4000077d80 pc=0xdc30b8
runtime.selectgo(0x4000077f50, 0x4000283f10, 0x303263?, 0x0, 0x657473614d2f6d6f?, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:351 +0x6c4 fp=0x4000077ed0 sp=0x4000077da0 pc=0xd9f624
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x400003c2a0, {0x329f408, 0x40001880f0})
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0xc4 fp=0x4000077fa0 sp=0x4000077ed0 pc=0x130de44
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x30 fp=0x4000077fd0 sp=0x4000077fa0 pc=0x130dc80
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000077fd0 sp=0x4000077fd0 pc=0xdcb2b4
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 10 gp=0x4000557500 m=nil [select]:
runtime.gopark(0x4000065f50?, 0x2?, 0xc0?, 0x5?, 0x4000065f14?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000065da0 sp=0x4000065d80 pc=0xdc30b8
runtime.selectgo(0x4000065f50, 0x4000065f10, 0x100000000000000?, 0x0, 0x40002af8c0?, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:351 +0x6c4 fp=0x4000065ed0 sp=0x4000065da0 pc=0xd9f624
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x400003c2d0, {0x329f408, 0x4000188140})
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0xc4 fp=0x4000065fa0 sp=0x4000065ed0 pc=0x130de44
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x30 fp=0x4000065fd0 sp=0x4000065fa0 pc=0x130dc80
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000065fd0 sp=0x4000065fd0 pc=0xdcb2b4
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x118

goroutine 31 gp=0x40005576c0 m=nil [IO wait]:
runtime.gopark(0x0?, 0x6?, 0xf?, 0x0?, 0x8000?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4002e61ad0 sp=0x4002e61ab0 pc=0xdc30b8
runtime.netpollblock(0x0?, 0xffffffff?, 0xff?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:575 +0x158 fp=0x4002e61b10 sp=0x4002e61ad0 pc=0xd83cc8
internal/poll.runtime_pollWait(0xefd6676b5ed0, 0x72)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/netpoll.go:351 +0xa0 fp=0x4002e61b40 sp=0x4002e61b10 pc=0xdc1a90
internal/poll.(*pollDesc).wait(0x400020ac00?, 0x4000496000?, 0x0)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x4002e61b70 sp=0x4002e61b40 pc=0xe4a008
internal/poll.(*pollDesc).waitRead(...)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x400020ac00, {0x4000496000, 0x8000, 0x8000})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/internal/poll/fd_unix.go:165 +0x1fc fp=0x4002e61c10 sp=0x4002e61b70 pc=0xe4b3fc
net.(*netFD).Read(0x400020ac00, {0x4000496000?, 0x4002e61c98?, 0xd5d86c?})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/net/fd_posix.go:55 +0x28 fp=0x4002e61c60 sp=0x4002e61c10 pc=0xecd378
net.(*conn).Read(0x4001ca6fa8, {0x4000496000?, 0x500000000?, 0xefd666d30cb8?})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/net/net.go:194 +0x34 fp=0x4002e61cb0 sp=0x4002e61c60 pc=0xede464
bufio.(*Reader).Read(0x40000ffe60, {0x40000ab2a0, 0x9, 0x40005c2030?})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/bufio/bufio.go:245 +0x1b0 fp=0x4002e61cf0 sp=0x4002e61cb0 pc=0xef8930
io.ReadAtLeast({0x328a000, 0x40000ffe60}, {0x40000ab2a0, 0x9, 0x9}, 0x9)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/io/io.go:335 +0xa0 fp=0x4002e61d40 sp=0x4002e61cf0 pc=0xe19620
io.ReadFull(...)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0x40000ab2a0, 0x9, 0x4001ee13b0?}, {0x328a000?, 0x40000ffe60?})
        /home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x58 fp=0x4002e61d90 sp=0x4002e61d40 pc=0x13d0518
golang.org/x/net/http2.(*Framer).ReadFrame(0x40000ab260)
        /home/ubuntu/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:501 +0x74 fp=0x4002e61e40 sp=0x4002e61d90 pc=0x13d1094
google.golang.org/grpc/internal/transport.(*http2Client).reader(0x4000172488, 0x40005aa7e0)
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1639 +0x158 fp=0x4002e61fb0 sp=0x4002e61e40 pc=0x1409d28
google.golang.org/grpc/internal/transport.NewHTTP2Client.gowrap4()
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:413 +0x2c fp=0x4002e61fd0 sp=0x4002e61fb0 pc=0x1400d3c
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4002e61fd0 sp=0x4002e61fd0 pc=0xdcb2b4
created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 28
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:413 +0x16ac

goroutine 8 gp=0x4000003dc0 m=nil [select]:
runtime.gopark(0x4000063ec8?, 0x2?, 0x78?, 0x3d?, 0x4000063e94?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x4000063d20 sp=0x4000063d00 pc=0xdc30b8
runtime.selectgo(0x4000063ec8, 0x4000063e90, 0x4000063ea8?, 0x0, 0x4001ca6fa8?, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:351 +0x6c4 fp=0x4000063e50 sp=0x4000063d20 pc=0xd9f624
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x40000cbe40, 0x1)
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:412 +0x144 fp=0x4000063ef0 sp=0x4000063e50 pc=0x13f7034
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x400020ae80)
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:575 +0x70 fp=0x4000063f50 sp=0x4000063ef0 pc=0x13f7890
google.golang.org/grpc/internal/transport.NewHTTP2Client.func6()
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:471 +0xb8 fp=0x4000063fd0 sp=0x4000063f50 pc=0x1400c38
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000063fd0 sp=0x4000063fd0 pc=0xdcb2b4
created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 28
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:469 +0x1b38

goroutine 11 gp=0x4000104540 m=nil [select]:
runtime.gopark(0x4000284750?, 0x2?, 0xc0?, 0x5?, 0x4000284714?)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:435 +0xc8 fp=0x40002845a0 sp=0x4000284580 pc=0xdc30b8
runtime.selectgo(0x4000284750, 0x4000284710, 0x726f2e676e616c6f?, 0x0, 0x7265746e692f6675?, 0x1)
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/select.go:351 +0x6c4 fp=0x40002846d0 sp=0x40002845a0 pc=0xd9f624
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x400003c300, {0x329f408, 0x4000188190})
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0xc4 fp=0x40002847a0 sp=0x40002846d0 pc=0x130de44
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1()
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x30 fp=0x40002847d0 sp=0x40002847a0 pc=0x130dc80
runtime.goexit({})
        /home/ubuntu/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40002847d0 sp=0x40002847d0 pc=0xdcb2b4
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
        /home/ubuntu/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x118

r0      0x0
r1      0x1a1d
r2      0x6
r3      0xefd6661bf180
r4      0xefd6af54eb88
r5      0x1
r6      0x20
r7      0xa72656666756220
r8      0x83
r9      0x45648d0
r10     0xefd6661be5d0
r11     0xefd6661be660
r12     0xefd6661be750
r13     0xefd6661be840
r14     0x0
r15     0x0
r16     0x1
r17     0xefd6af338444
r18     0x7
r19     0x1a1d
r20     0xefd6661bf180
r21     0x6
r22     0xefd6af547000
r23     0xefd6af409710
r24     0x0
r25     0x40000577a8
r26     0x400004c590
r27     0x10
r28     0x4000003c00
r29     0xefd6661be1a0
lr      0xefd6af3353e4
sp      0xefd6661be190
pc      0xefd6af3353f8
fault   0x0

@AndrewGMorgan
Copy link
Copy Markdown

I found a stack allocation thing that potentially affects arm64. My testing shows the same behavior with and without the change, but the change does seem to be what the kernel wants. I've tagged the psx package v1.2.76-rc1. You'll have to do a little surgery on package dependencies to get the cap package to use that, but if you are motivated... Else you can wait for v1.2.76 to be released. (Likely next month.)

@geyslan geyslan changed the title fix(libcap): bump libcap to v1.2.75 fix(libcap): bump libcap to v1.2.75 & libpsx to v1.2.76-rc1 Mar 31, 2025
@geyslan
Copy link
Copy Markdown
Member Author

geyslan commented Mar 31, 2025

@AndrewGMorgan Thanks for the new psx v1.2.76-rc1, I've bumped it to. Tests are underway, and I'll update you soon.

@geyslan
Copy link
Copy Markdown
Member Author

geyslan commented Mar 31, 2025

All aarch64 tests passed! 🎉 @AndrewGMorgan the change works. Again, thanks a million.

@geyslan
Copy link
Copy Markdown
Member Author

geyslan commented Mar 31, 2025

/fast-forward

@github-actions github-actions Bot merged commit fd21867 into aquasecurity:main Mar 31, 2025
@geyslan geyslan deleted the 4678-fix-libcap branch March 31, 2025 13:09
@AndrewGMorgan
Copy link
Copy Markdown

I just made the cap/v1.2.76 release, which includes this fix for the psx mechanism. Let me know if removing the -rc1 pinning workaround gives any issues.

geyslan added a commit to geyslan/tracee that referenced this pull request Apr 18, 2025
Even not required, this aligns libbpfgo that uses cap/psx v1.2.76, see: aquasecurity#4688.

This also brings USDT support.
geyslan added a commit that referenced this pull request Apr 18, 2025
Even not required, this aligns libbpfgo that uses cap/psx v1.2.76, see: #4688.

This also brings USDT support.
dependabot Bot pushed a commit that referenced this pull request Apr 18, 2025
Even not required, this aligns libbpfgo that uses cap/psx v1.2.76, see: #4688.

This also brings USDT support.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Go runtime stuck

3 participants