Hi, we're using Amazon Linux 2023 and can't use ddprof as it fails to load with
$ ddprof/bin/ddprof ls
<ERROR>Nov 27 22:56:59.045432 ddprof[1295]: Could not mmap memory for watcher #0: Invalid argument. Please increase kernel limits on pinned memory (ulimit -l). OR associate the IPC_LOCK capability to this process.
<ERROR>Nov 27 22:56:59.045477 ddprof[1295]: PERFMMAP: error in mmap operations at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/pevent_lib.cc:216
<ERROR>Nov 27 22:56:59.045484 ddprof[1295]: Forward error at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/pevent_lib.cc:255 - PERFMMAP: error in mmap operations
<ERROR>Nov 27 22:56:59.054679 ddprof[1295]: Could not mmap memory for watcher #0: Invalid argument. Please increase kernel limits on pinned memory (ulimit -l). OR associate the IPC_LOCK capability to this process.
<ERROR>Nov 27 22:56:59.054704 ddprof[1295]: PERFMMAP: error in mmap operations at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/pevent_lib.cc:216
<ERROR>Nov 27 22:56:59.054709 ddprof[1295]: Forward error at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/pevent_lib.cc:255 - PERFMMAP: error in mmap operations
<ERROR>Nov 27 22:56:59.054717 ddprof[1295]: Forward error at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/pevent_lib.cc:268 - PERFMMAP: error in mmap operations
<ERROR>Nov 27 22:56:59.054725 ddprof[1295]: Forward error at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/ddprof.cc:89 - PERFMMAP: error in mmap operations
<ERROR>Nov 27 22:56:59.054729 ddprof[1295]: Failed to initialize profiling
<ERROR>Nov 27 22:56:59.056723 ddprof-library[1293]: Unable to connect to socket: Unknown error
Note that ulimit and capabilities are correctly set. We can see the same issue when spinning up a recent Ubuntu 25.04 image. Running ddprof on images with older kernels works fine (e.g. Amazon Linux 2023 image from earlier in the year, or Ubuntu 24.04).
We think we've tracked the problem back to a kernel change that was done in the context of CVE-2025-38563, and think it's a problem in perfown_sz which overlaps perf event buffers.
Amazon quickly patched their images (see https://explore.alas.aws.amazon.com/CVE-2025-38563.html), and Canonical has done so too (see https://ubuntu.com/security/CVE-2025-38563), at least for Ubuntu 25.04.
This blocks us from using Datadog's profiler.
Hi, we're using Amazon Linux 2023 and can't use
ddprofas it fails to load withNote that ulimit and capabilities are correctly set. We can see the same issue when spinning up a recent Ubuntu 25.04 image. Running
ddprofon images with older kernels works fine (e.g. Amazon Linux 2023 image from earlier in the year, or Ubuntu 24.04).We think we've tracked the problem back to a kernel change that was done in the context of CVE-2025-38563, and think it's a problem in perfown_sz which overlaps perf event buffers.
Amazon quickly patched their images (see https://explore.alas.aws.amazon.com/CVE-2025-38563.html), and Canonical has done so too (see https://ubuntu.com/security/CVE-2025-38563), at least for Ubuntu 25.04.
This blocks us from using Datadog's profiler.