What makes perf_event special<\/strong> is its comprehensive approach. Rather than focusing on just one aspect of system performance, it offers a unified interface to collect data from multiple sources:<\/p>\n\n\n\n Now we know what perf_event is in Linux systems, but you might wonder, “Why should I learn about perf_event when there are plenty of monitoring tools available?” The answer is simple: power and precision.<\/strong><\/p>\n\n\n\n When you need to answer questions like “Why is this specific application slow?” or “What’s causing these mysterious latency spikes?”, generic monitoring tools often fall short. They might tell you that<\/em> something is wrong, but not why<\/em> or where<\/em>.<\/p>\n\n\n\n With perf_event, you can:<\/strong><\/p>\n\n\n\n I’ve personally used perf_event to solve performance mysteries that stumped an entire team\u2014like discovering a poorly optimized function that was causing cache misses and slowing down the entire application. The insights you gain from perf_event can transform the way you think about performance tuning.<\/p>\n\n\n\n While perf_event is the kernel subsystem<\/strong>, most users interact with it through the perf command-line tool, <\/strong>which provides a user-friendly interface to the perf_event functionality.<\/p>\n\n\n\n The perf tool <\/strong>is accessible from the command line and provides various subcommands for different performance analysis tasks.<\/p>\n\n\n\n Let’s look at some of the most useful perf subcommands:<\/strong><\/p>\n\n\n\n These commands form the foundation of your performance analysis toolkit. As you become more familiar with Perf, <\/strong>you’ll develop your\u00a0<\/strong>workflow b<\/span>y combining these tools to diagnose issues quickly.<\/p>\n\n\n\n Before diving into perf_event<\/strong>, you’ll need to install the perf tool.<\/strong> On most Linux distributions, it’s part of the linux-tools package.<\/p>\n\n\n\n On Ubuntu\/Debian systems:<\/strong><\/p>\n\n\n\n On Fedora\/RHEL\/CentOS:<\/strong><\/p>\n\n\n\n Once installed, verify it’s working:<\/strong><\/p>\n\n\n\n If you see a version number, you’re ready to start exploring!<\/p>\n\n\n\n Let’s start with the most straightforward and most useful perf <\/strong>command\u2014 For example<\/strong>, to measure performance statistics <\/strong>while running the This will output something like:<\/p>\n\n\n\n Even this simple output provides valuable insights:<\/p>\n\n\n\n With just this quick command, I can immediately see if my program is CPU-bound, memory-bound, or suffering from excessive context switching.<\/p>\n\n\n\n If you get the following error while running the perf stat command:<\/p>\n\n\n\n Then to solve it: Fix: Lower the To allow non-root users (or even root in some cases) to use Or, to allow full access:<\/p>\n\n\n\n Open the sysctl config: <\/strong><\/p>\n\n\n\n Add this line at the end of the file and save it:<\/strong><\/p>\n\n\n\n Apply the changes: <\/strong><\/p>\n\n\n\n While perf stat gives you a high-level overview, The basic workflow is to use perf record to capture performance data and then analyze it with perf report. This powerful combination allows you to see exactly where your system is spending time, down to the specific functions.<\/p>\n\n\n\n Let’s see it in action:<\/strong><\/p>\n\n\n\n The Why 99 Hz instead of a round number like 100<\/strong>? It’s a clever trick to avoid accidentally synchronizing with periodic system activities, which could skew your results.<\/p>\n\n\n\n When you run This shows the functions where your CPU time is being spent, sorted by “overhead<\/strong>” (the percentage of samples captured in that function). The beauty of this approach is that you’re seeing real-world usage, not theoretical bottlenecks.<\/p>\n\n\n\n I often describe perf record\/report as “putting your system under a microscope”\u2014it reveals details you simply can’t see with higher-level tools.<\/p>\n\n\n\n perf_events <\/strong>can trace a wide range of events, including hardware events from CPU performance counters, software events based on kernel counters, kernel tracepoints, user-level tracepoints, and dynamic events using kprobes and uprobes.<\/p>\n\n\n\n Here are some examples of more targeted perf commands:<\/p>\n\n\n\n Each of these commands provides visibility into a different aspect of system behavior. The key is understanding what you’re looking for and choosing the appropriate events to trace.<\/p>\n\n\n\n Dynamic tracing allows you to create custom instrumentation points anywhere in the kernel (using kprobes) or user-level applications (using uprobes). This capability lets you analyze what’s happening at very specific points in the code.<\/p>\n\n\n\n For example, to trace a specific kernel function:<\/p>\n\n\n\n If you have debug symbols installed, you can even probe specific lines within functions or access function arguments:<\/p>\n\n\n\n This level of flexibility is what makes perf_event such a powerful tool for performance debugging\u2014you can look at exactly what matters for your specific problem.<\/p>\n\n\n\n Let’s explore a real-world scenario where perf_event can save the day. Imagine you’re running a web server that suddenly experiences high latency. Traditional monitoring shows high CPU usage but doesn’t tell you why.<\/p>\n\n\n\n Here’s how I’d approach this with perf:<\/strong><\/p>\n\n\n\n While the text output of perf commands is informative, visualizing performance data can make patterns much more apparent. One of the most powerful visualization techniques for perf <\/strong>data is the flame graph<\/strong>.<\/p>\n\n\n\n Flame graphs<\/strong> display stack traces in a way that makes it immediately obvious where CPU time is being spent. Each rectangle represents a function in the stack, with the width proportional to how often it appears in the samples.<\/p>\n\n\n\n Download the FlameGraph tools: <\/strong><\/p>\n\n\n\n Collect stack traces with perf:<\/strong><\/p>\n\n\n\n Generate the flame graph: <\/strong><\/p>\n\n\n\n The resulting SVG file can be opened in any web browser.<\/strong> Functions consuming the most CPU time will be the widest bars, making bottlenecks visually obvious.<\/p>\n\n\n\n I’ve found flame graphs to be incredibly effective for communicating performance issues to team members who might not be familiar with performance analysis. A picture truly is worth a thousand perf reports!<\/p>\n\n\n\n While perf_event is incredibly powerful, it’s not without challenges:<\/p>\n\n\n\n perf_events <\/strong>has significant security implications because it can potentially expose sensitive information. Linux implements security controls based on process credentials, dividing them into privileged (root<\/strong>) and unprivileged processes.<\/p>\n\n\n\n By default, Linux restricts certain perf_event operations to privileged users.<\/strong> You can check the current settings:<\/p>\n\n\n\n Higher values impose more restrictions. You can temporarily lower this for testing, but be careful about security implications:<\/p>\n\n\n\n While perf is designed to be lightweight, tracing can still introduce overhead. This is especially true when:<\/p>\n\n\n\n Always be mindful of the overhead, especially on production systems. Start with targeted, low-frequency tracing and increase only as needed.<\/p>\n\n\n\n Some perf features depend on:<\/p>\n\n\n\n These dependencies can limit what’s possible on some systems, especially in virtualized environments where hardware access may be restricted.<\/p>\n\n\n\n As you become more comfortable with perf_event,<\/strong> here are some advanced topics worth exploring:<\/p>\n\n\n\n In newer Linux kernels (4.4+), perf has enhanced BPF support. This makes perf tracing programmable, transforming it from a mere counting and sampling tool to a fully programmable in-kernel tracer.<\/p>\n\n\n\n BPF programs can filter events in kernel space, drastically reducing overhead for complex analyses.<\/p>\n\n\n\n The This is invaluable for diagnosing CPU scheduling issues and latency problems.<\/p>\n\n\n\n Modern CPUs have extensive performance monitoring capabilities for their cache hierarchies:<\/p>\n\n\n\n Understanding cache behavior is crucial for optimizing high-performance applications.<\/p>\n\n\n\n\n
Why Should You Care About perf_event?<\/h2>\n\n\n\n
\n
The perf Command: Your Gateway to perf_event<\/h2>\n\n\n\n
\n
Getting Started: Installing perf<\/h2>\n\n\n\n
sudo apt install linux-tools-common linux-tools-generic<\/code><\/pre>\n\n\n\nsudo yum install perf<\/code><\/pre>\n\n\n\nperf --version<\/code><\/pre>\n\n\n\nPerf Stat: Your First Step into Performance Monitoring<\/h2>\n\n\n\n
perf stat<\/code>. This command measures key performance counters for a specified command or time period.<\/p>\n\n\n\ngzip<\/code> command:<\/p>\n\n\n\nperf stat gzip largefile<\/code><\/pre>\n\n\n\n Performance counter stats for 'gzip largefile':\n\n 1,920.15 msec task-clock # 0.991 CPUs utilized \n 13 context-switches # 0.007 K\/sec \n 0 cpu-migrations # 0.000 K\/sec \n 258 page-faults # 0.134 K\/sec \n 5,649,595,479 cles # 2.942 GHz \n 8,625,207,199 instructions # 1.53 insn per cycle \n 1,488,797,176 branches # 775.351 M\/sec \n 53,395,139 branch-misses # 3.59% of all branches \n\n 1.936842598 seconds time elapsed<\/code><\/pre>\n\n\n\n\n
Error:\nAccess to performance monitoring and observability operations is limited.\nConsider adjusting \/proc\/sys\/kernel\/perf_event_paranoid setting to open\naccess to performance monitoring and observability operations for processes\nwithout CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.\nMore information can be found at 'Perf events and tool security' document:\nhttps:\/\/www.kernel.org\/doc\/html\/latest\/admin-guide\/perf-security.html\nperf_event_paranoid setting is 4:\n -1: Allow use of (almost) all events by all users\n Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK\n>= 0: Disallow raw and ftrace function tracepoint access\n>= 1: Disallow CPU event access\n>= 2: Disallow kernel profiling\nTo make the adjusted perf_event_paranoid setting permanent preserve it\nin \/etc\/sysctl.conf (e.g. kernel.perf_event_paranoid = <setting>)<\/code><\/pre>\n\n\n\nperf_event_paranoid<\/code> Value<\/strong><\/p>\n\n\n\nperf<\/code>, you need to change the perf_event_paranoid<\/code><\/strong> setting.<\/p>\n\n\n\n\u2705 Temporary Fix (until reboot):<\/h3>\n\n\n\n
sudo sysctl -w kernel.perf_event_paranoid=1<\/code><\/pre>\n\n\n\nsudo sysctl -w kernel.perf_event_paranoid=-1<\/code><\/pre>\n\n\n\n\u2705 Permanent Fix:<\/h3>\n\n\n\n
sudo nano \/etc\/sysctl.conf<\/code><\/pre>\n\n\n\nkernel.perf_event_paranoid = -1<\/code><\/pre>\n\n\n\nsudo sysctl -p<\/code><\/pre>\n\n\n\nPerf Record & Report: Capturing and Analyzing Performance Data<\/h2>\n\n\n\n
perf record<\/code> and perf report<\/code> let you dive deeper by sampling what your system is doing over time.<\/p>\n\n\n\n# Sample CPU stack traces at 99 Hz for 30 seconds\nperf record -F 99 -a -g -- sleep 30\n\n# Analyze the collected data\nperf report<\/code><\/pre>\n\n\n\n-F 99<\/code> option sets the sampling frequency to 99 <\/strong>Hz (samples per second), -a<\/code> tells perf to monitor all CPUs, and -g<\/code> enables call graph recording so you can see complete stack traces.<\/p>\n\n\n\nperf report<\/code>, you’ll see something like:<\/p>\n\n\n\n![\"Analyze](\"https:\/\/linux.how2shout.com\/wp-content\/uploads\/2025\/05\/Analyze-the-collected-data-using-perf-report-command-1024x534.jpg\")
<\/figure>\n\n\n\nAdvanced Perf Usage: Tracing Specific Events<\/h2>\n\n\n\n
Tracing System Calls<\/h3>\n\n\n\n
# Count all system calls made by a command\nperf stat -e 'syscalls:sys_enter_*' command\n\n# Trace specific system calls with stack traces\nperf record -e 'syscalls:sys_enter_read' -ag<\/code><\/pre>\n\n\n\nAnalyzing Disk I\/O<\/h3>\n\n\n\n
# Trace all block I\/O events\nperf record -e block:* -a\n\n# Track disk I\/O latency\nperf record -e block:block_rq_issue -e block:block_rq_complete -a<\/code><\/pre>\n\n\n\nMonitoring Network Activity<\/h3>\n\n\n\n
# Trace outbound TCP connections\nperf record -e syscalls:sys_enter_connect -ag\n\n# Track socket buffer consumption\nperf record -e 'skb:consume_skb' -ag<\/code><\/pre>\n\n\n\nDynamic Tracing <\/h2>\n\n\n\n
# Add a dynamic probe to the tcp_sendmsg function\nperf probe --add tcp_sendmsg\n\n# Record traces of this function\nperf record -e probe:tcp_sendmsg -ag\n\n# Remove the probe when done\nperf probe --del tcp_sendmsg<\/code><\/pre>\n\n\n\n# Show available variables in the tcp_sendmsg function\nperf probe -V tcp_sendmsg\n\n# Add a probe that captures the 'size' argument\nperf probe --add 'tcp_sendmsg size'<\/code><\/pre>\n\n\n\nReal-World Example: Diagnosing a Performance Bottleneck<\/h2>\n\n\n\n
\n
perf top -p $(pgrep -f \"web-server\")<\/code> This shows me real-time usage. Let’s say I notice high activity in SSL\/TLS functions.<\/li>\n\n\n\nperf record -F 99 -p $(pgrep -f \"web-server\") -g -- sleep 60<\/code><\/li>\n\n\n\nperf report --sort comm,dso,symbol<\/code> This might reveal that most time is spent in a specific encryption function.<\/li>\n\n\n\nperf record -e probe:ssl_read -e probe:ssl_write -p $(pgrep -f \"web-server\")<\/code><\/li>\n<\/ol>\n\n\n\nPerformance Visualization with Flame Graphs<\/h2>\n\n\n\n
Example to create a flame graph:<\/h3>\n\n\n\n
git clone https:\/\/github.com\/brendangregg\/FlameGraph.git<\/code><\/pre>\n\n\n\nperf record -F 99 -a -g -- sleep 30<\/code><\/pre>\n\n\n\nperf script | FlameGraph\/stackcollapse-perf.pl | FlameGraph\/flamegraph.pl > profile.svg<\/code><\/pre>\n\n\n\n![\"Generate](\"https:\/\/linux.how2shout.com\/wp-content\/uploads\/2025\/05\/Generate-the-flame-graph-using-perf-command.jpg\")
<\/figure>\n\n\n\nChallenges and Considerations When Using perf_event<\/h2>\n\n\n\n
Security and Permissions<\/h3>\n\n\n\n
cat \/proc\/sys\/kernel\/perf_event_paranoid<\/code><\/pre>\n\n\n\nsudo sysctl kernel.perf_event_paranoid=1<\/code><\/pre>\n\n\n\nOverhead Considerations<\/h3>\n\n\n\n
\n
-F 1000<\/code> instead of -F 99<\/code>)<\/li>\n\n\n\nKernel and Hardware Dependencies<\/h3>\n\n\n\n
\n
Beyond Basics – That Is Worth Exploring<\/h2>\n\n\n\n
eBPF Integration<\/h3>\n\n\n\n
Scheduler Analysis with perf sched<\/h3>\n\n\n\n
perf sched<\/code> subcommand provides specialized tools for analyzing the kernel’s CPU scheduler behavior:<\/p>\n\n\n\n# Record scheduler events\nperf sched record -- sleep 1\n\n# Analyze scheduler latencies\nperf sched latency<\/code><\/pre>\n\n\n\nHardware Cache Events<\/h3>\n\n\n\n
# Analyze Level 1 data cache behavior\nperf stat -e L1-dcache-loads,L1-dcache-load-misses command<\/code><\/pre>\n\n\n\nFAQs About perf_event<\/h2>\n\n\n\n