{"@attributes":{"version":"2.0"},"channel":{"title":"Linux on Tanel Poder Blog","link":"https:\/\/tanelpoder.com\/categories\/linux\/","description":"Recent content in Linux on Tanel Poder Blog","generator":"Hugo -- gohugo.io","language":"en-us","lastBuildDate":"Tue, 09 Sep 2025 12:18:32 -0400","item":[{"title":"The xCapture and xtop eBPF tools are now in beta, with a demo dataset","link":"https:\/\/tanelpoder.com\/posts\/xcapture-xtop-beta\/","pubDate":"Tue, 09 Sep 2025 12:18:32 -0400","guid":"https:\/\/tanelpoder.com\/posts\/xcapture-xtop-beta\/","description":"Update: I just released xcapture v3.0.3 at P99 CONF, the first proper release of this whole concept! The 20-minute launch video is available for streaming on P99 CONF website.\nRepo: github.com\/tanelpoder\/0xtools\nThe title and update say it all, I think it’s time to call the xcapture and xtop toolset ready for use now.\nI plan to announce the production release (or at least a release candidate) at P99CONF this October."},{"title":"New Tool: xstack - Completely Passive eBPF Linux Stack Profiling Without Any Tracepoints","link":"https:\/\/tanelpoder.com\/posts\/xstack-passive-linux-stack-sampler-ebpf\/","pubDate":"Thu, 14 Aug 2025 17:01:35 -0400","guid":"https:\/\/tanelpoder.com\/posts\/xstack-passive-linux-stack-sampler-ebpf\/","description":"I just added another tool into my 0x.tools toolset:\nWith xstack, you can passively sample both kernel stacks and user stacks (as long as framepointers are present) with a pretty minimalistic tool, with no direct impact to your critical application processes!\nWhile xstack itself is just a data extraction tool, you probably want to summarize\/profile all those thread states and stack profiles. Here’s an example of piping xstack output to flamelens to immediately display a (terminal) flamegraph from the sampled data:"},{"title":"Dimensional Analysis of System Performance with eBPF & xtop: Top for Wall-Clock Time","link":"https:\/\/tanelpoder.com\/posts\/xtop-top-for-wall-clock-time\/","pubDate":"Tue, 12 Aug 2025 23:46:44 -0400","guid":"https:\/\/tanelpoder.com\/posts\/xtop-top-for-wall-clock-time\/","description":"If you are wondering how the dimensional eBPF data collection works under the hood, check out my earlier post about xcapture: eBPF performance analysis and the 0x.tools repo Summary xtop is like the Linux top tool, but extended with x-ray vision and ability to view your performance data from any chosen angle (that eBPF allows to access). This enables dimensional performance analysis on Linux and tools like top for wall-clock time and much more."},{"title":"Using Postgres pg_test_fsync tool for testing low latency writes","link":"https:\/\/tanelpoder.com\/posts\/using-pg-test-fsync-for-testing-low-latency-writes\/","pubDate":"Tue, 27 May 2025 20:31:34 -0400","guid":"https:\/\/tanelpoder.com\/posts\/using-pg-test-fsync-for-testing-low-latency-writes\/","description":"Here’s a useful tool for quickly testing whether a disk (or a cloud block store volume) is a good candidate for your database WAL\/redo logs and any other files that require low latency writes. The pg_test_fsync tool is bundled with standard Postgres packages, so no extra installation is needed. You don’t actually have to use Postgres as your database, this tool’s output is universally valuable for any workload requiring fast writes."},{"title":"New Tool: lsds - List All Linux Block Devices and Settings in One Place","link":"https:\/\/tanelpoder.com\/posts\/lsds-list-linux-block-devices-and-their-config\/","pubDate":"Fri, 09 May 2025 10:23:21 -0400","guid":"https:\/\/tanelpoder.com\/posts\/lsds-list-linux-block-devices-and-their-config\/","description":"When dealing with disks and I\/O things on Linux, you’d regularly run commands like lsblk, lsscsi, nvme list, etc. All of them tend to report a different set of information, so I ended up running multiple commands and correlating their output based on the device name or number.\nAnd then I had to run commands like these, to get extra info about the current OS-level configuration settings for specific disks:"},{"title":"Optimizing eBPF I\/O latency accounting when running 37M IOPS on a 384-CPU server","link":"https:\/\/tanelpoder.com\/posts\/optimizing-ebpf-biolatency-accounting\/","pubDate":"Tue, 29 Apr 2025 21:42:03 -0400","guid":"https:\/\/tanelpoder.com\/posts\/optimizing-ebpf-biolatency-accounting\/","description":"In this post I will introduce a much more efficient method for accounting block I\/O latencies with eBPF on Linux. In my stress test, the “new biolatency” accounting method had 59x lower CPU and probe latency overhead compared to the current biolatency approach.\nSo I couldn’t help it and ended up putting 21 NVMe SSDs into one of my homelab servers. 8 of them are PCIe5 and the remaining 13 are PCIe4."},{"title":"xCapture v3: Linux Performance Analysis with Modern eBPF and DuckDB","link":"https:\/\/tanelpoder.com\/posts\/xcapture-v3-alpha-ebpf-performance-analysis-with-duckdb\/","pubDate":"Wed, 23 Apr 2025 16:04:06 -0400","guid":"https:\/\/tanelpoder.com\/posts\/xcapture-v3-alpha-ebpf-performance-analysis-with-duckdb\/","description":"To capture your attention, this is what you get:\nTable of contents What is xCapture? xCapture with modern eBPF Modular and decoupled at multiple levels Dimensional performance data analysis with DuckDB Top wall-clock time used by all threads Regular expressions in the filter Syscalls against network socket file descriptors What is the performance overhead? Performance overhead of active tracking probes Userspace processing overhead Running xcapture What’s next? What is xCapture? 0x."},{"title":"When eBPF task->stack->pt_regs reads return garbage on the latest Linux kernels, blame Fred!","link":"https:\/\/tanelpoder.com\/posts\/ebpf-pt-regs-error-on-linux-blame-fred\/","pubDate":"Fri, 28 Feb 2025 19:57:27 -0500","guid":"https:\/\/tanelpoder.com\/posts\/ebpf-pt-regs-error-on-linux-blame-fred\/","description":"TLDR; Starting from Linux kernel version 6.9 on x86_64, there’s a new config option CONFIG_X86_FRED enabled and it adds 16 bytes to the starting point of a task’s kernel stack area, so you’ll need to account for this extra padding in your “raw” kernel stack & pt_regs lookup code.\nUpdate: It turns out that there’s a bpf_task_pt_regs() helper function available in newer kernel\/eBPF versions and it does the same offset math for you, if you can retrieve your task using the bpf_get_current_task_btf() BTF-enhanced helper that is data structure aware."},{"title":"TracepointArgs: List Any Linux Tracepoint with Their Arguments, Datatypes and Related Structs","link":"https:\/\/tanelpoder.com\/posts\/tpargs-list-tracepoint-arguments-and-structures\/","pubDate":"Thu, 30 Jan 2025 08:13:29 -0500","guid":"https:\/\/tanelpoder.com\/posts\/tpargs-list-tracepoint-arguments-and-structures\/","description":"This should be something useful for Linux kernel explorers and eBPF nerds!\nLast year I released a tool called syscallargs that walked through the Linux \/sys\/kernel\/debug\/tracing\/events\/syscalls directory tree and allowed you to query and list available system calls in your current system from the command line. And it printed out syscall argument names and their datatypes, so it has helped me avoid opening man-pages (or web-pages) a few times.\nLater I realized that since many other kinds of kernel events and tracepoints are also presented in the same \/sys directory, in the same format as syscall events, we can examine all of them using the same approach."},{"title":"Video: Testing the Silk Platform in 2024 Interview (12 minutes)","link":"https:\/\/tanelpoder.com\/posts\/testing-the-silk-platform-in-2024-interview\/","pubDate":"Thu, 28 Nov 2024 14:52:59 -0500","guid":"https:\/\/tanelpoder.com\/posts\/testing-the-silk-platform-in-2024-interview\/","description":"Here’s a 12-minute video of our chat with Silk VP of Product Tom O’Neill about my recent testing of the Silk Platform in Google Cloud.\nIn this interview we cover some high level points, conclusions and talk a little bit about the future. If you want to read all the technical details and some interesting references, see the links below:\nTesting the Silk Platform in 2024: Achieving 20 GiB\/s I\/O Throughput in a Single Cloud VM (2024) Testing The Silk Platform - Hands-On Technical Analysis of High-Performance I\/O in the Cloud (2021) NVIDIA 800Gbit network cards already on display (2024) 1."},{"title":"Testing the Silk Platform in 2024: Achieving 20 GiB\/s I\/O Throughput in a Single Cloud VM","link":"https:\/\/tanelpoder.com\/posts\/testing-the-silk-platform-in-2024\/","pubDate":"Tue, 05 Nov 2024 19:32:23 -0500","guid":"https:\/\/tanelpoder.com\/posts\/testing-the-silk-platform-in-2024\/","description":"Hands-on technical analysis of a novel data platform for high-performance block I\/O in the cloud, tested by Tanel Poder, a database consultant and a long-time computer performance nerd.\nIndex Background and motivation Scalable Architecture by Design Enterprise Features Testing Results I\/O Latency 1.3 Million IOPS Lessons Learned Summary Background and Motivation Back in 2021, my old friend Chris Buckel (@flashdba) asked me if I would like to test out the Silk Data Platform in the cloud and see how far I could push it."},{"title":"Tracking, not Tracing, Linux Thread Activity for Complete System Visibility (eBPF Summit)","link":"https:\/\/tanelpoder.com\/posts\/tracking-not-tracing-linux-thread-activity-with-ebpf\/","pubDate":"Thu, 12 Sep 2024 20:55:07 -0400","guid":"https:\/\/tanelpoder.com\/posts\/tracking-not-tracing-linux-thread-activity-with-ebpf\/","description":"Here’s my talk from eBPF Summit 2024.\nVideo (10 minutes) The 0x.tools Extended Task State Sampling approach provides a new observability signal - wall-clock time of all active threads’ activity in your system. You can see both total workload demand of your apps and also drill down deep into individual threads’ activity when needed. This method and toolset give you a single data source with many directly linked fields of information about what your application & OS threads are doing and why."},{"title":"Python Supports Profiling with Perf - How Does It Work?","link":"https:\/\/tanelpoder.com\/posts\/python-perf-profiling-support\/","pubDate":"Wed, 31 Jul 2024 01:14:56 -0400","guid":"https:\/\/tanelpoder.com\/posts\/python-perf-profiling-support\/","description":"Starting from Python 3.12, you can instruct Python to enable direct perf profiling support for your Python code. This feature currently works only on Linux, as it communicates the relationships between Python-level interpreted function calls and machine-code level return pointers (for stack unwinding) via perf map files. The Linux perf tool knows how to read these files and maps the return pointers in a thread’s stack to the corresponding Python-level symbols (similar to what JVM JIT code profiling with Perf maps does)."},{"title":"Can a User Process Consume Kernel Mode CPU Without a Syscall?","link":"https:\/\/tanelpoder.com\/posts\/user-process-consuming-kernel-cpu-without-a-syscall\/","pubDate":"Wed, 10 Jul 2024 09:33:13 -0400","guid":"https:\/\/tanelpoder.com\/posts\/user-process-consuming-kernel-cpu-without-a-syscall\/","description":"Yes!\nVarious traps like page faults may cause your user process to be switched into kernel codepath even when the process is minding its own business in the userspace. It just needs to do something that causes such a trap, like touch a new page of memory in its virtual memory address space, that hasn’t been “fully materialized” yet in the kernel memory structures.\nFor example, after starting up an Oracle database instance that uses a large amount of shared memory, there’s one process that burns CPU for a while:"},{"title":"Next Big Thing: X-Ray Vision for Linux Systems with eBPF","link":"https:\/\/tanelpoder.com\/posts\/next-big-thing\/","pubDate":"Fri, 21 Jun 2024 20:14:02 -0400","guid":"https:\/\/tanelpoder.com\/posts\/next-big-thing\/","description":"I recently published what I have been working on for the past 5 years.\nNo, this is not about AI or a yet another startup - I’m still a big fan of NI (natural intelligence) and what I reveal is open source and free for everyone to use. I’m not gonna sell you any products, only ideas!\nThis is a new approach and a supporting toolset for understanding system behavior and its performance across all your applications and data center & cloud footprint."},{"title":"Syscallargs: List All Linux System Calls With Their Arguments","link":"https:\/\/tanelpoder.com\/posts\/list-linux-system-call-arguments-with-syscallargs\/","pubDate":"Thu, 13 Jun 2024 23:50:24 -0400","guid":"https:\/\/tanelpoder.com\/posts\/list-linux-system-call-arguments-with-syscallargs\/","description":"This blog post is not about tracing program system calls, but about programmatically extracting all system calls and their accepted arguments in your current system. If you want to read more about tracing system calls, here’s my tutorial using strace for troubleshooting a real life scenario: Troubleshooting Linux SSH Login Delay - Why does logging in always take 10 seconds?1\nUpdate: I have also created a similar program tracepointargs for listing any Linux tracepoint with their arguments, datatypes and related structs."},{"title":"RTX4090 converted into an eGPU","link":"https:\/\/tanelpoder.com\/posts\/rtx4090-egpu\/","pubDate":"Fri, 07 Jun 2024 06:59:31 -0400","guid":"https:\/\/tanelpoder.com\/posts\/rtx4090-egpu\/","description":"One evening back in January I finally had enough of thermal issues within my homelab server. You know, every time the computer fans make more noise than I think they should, I can’t help but investigate!\nAlso, the RTX4090 is so thick that it takes 3 PCIe slots worth of space on a typical motherboard - and you also need space for the airflow! I was using the same machine that achieved 11M IOPS & 66 GB\/s IO on a Single ThreadRipper Workstation back in 2020."},{"title":"MySQL Now Shows its Thread Names at OS Level for Better Troubleshooting","link":"https:\/\/tanelpoder.com\/posts\/mysql-now-shows-thread-names-at-os-level\/","pubDate":"Thu, 17 Mar 2022 14:19:40 -0400","guid":"https:\/\/tanelpoder.com\/posts\/mysql-now-shows-thread-names-at-os-level\/","description":"Starting from v8.0.27, the MySQL daemon shows its thread names at the OS level too, for better troubleshooting. The MySQL 8.0.27 release notes say this:\nTo assist monitoring and troubleshooting, the Performance Schema instrumentation is now used to export names of instrumented threads to the operating system. This enables utilities that display thread names, such as debuggers and the Unix ps command, to display distinct mysqld thread names rather than \u201cmysqld\u201d."},{"title":"Do Not Use '>' in Your Command Prompt (and How to Stay Safe in Shell)","link":"https:\/\/tanelpoder.com\/posts\/how-to-stay-safe-in-shell\/","pubDate":"Tue, 16 Mar 2021 16:30:29 -0500","guid":"https:\/\/tanelpoder.com\/posts\/how-to-stay-safe-in-shell\/","description":"Over the years of troubleshooting performance problems in the Unix\/Linux world, I have seen multiple cases where a regularly used command line tool in a customer server just stops working for some reason. The tool just returns immediately, doing absolutely nothing. No output printed, no coredumps and the exit code is zero (success!).\nThis article walks you through a couple of such incidents and in the end I explain how I avoid accidentally doing bad stuff in production in general."},{"title":"Is Your Linux Version Hiding Interrupt CPU Usage From You?","link":"https:\/\/tanelpoder.com\/posts\/linux-hiding-interrupt-cpu-usage\/","pubDate":"Mon, 15 Feb 2021 00:00:35 -0500","guid":"https:\/\/tanelpoder.com\/posts\/linux-hiding-interrupt-cpu-usage\/","description":"TL;DR: Some Linux distros and even different kernel versions within a distro have disabled IRQ time accounting. In such case, your monitoring tools will report zero time spent in hardware interrupt handlers (shown as %hi, %irq, hiq, etc). It’s easy to check how your kernel is behaving by looking at \/proc\/stat and you can still measure interrupt CPU usage using perf and a little creativity.\nIndex Index Problem Explanation Kernel configuration options Measuring IRQ time with perf Different distros and kernel versions Summary Here’s a reminder that your Linux kernel may be not breaking down hardware interrupt CPU usage separately in its \/proc\/stat CPU time reporting, depending on the kernel’s compile-time settings."},{"title":"Video: High Performance Block I\/O on Linux","link":"https:\/\/tanelpoder.com\/posts\/high-performance-block-io-on-linux\/","pubDate":"Sun, 07 Feb 2021 15:58:57 -0500","guid":"https:\/\/tanelpoder.com\/posts\/high-performance-block-io-on-linux\/","description":"Here’s the High Performance Block IO on Linux hacking session video.\nIt’s based on my recent article about achieving 11M IOPS & 66 GB\/s IO on a single ThreadRipper workstation.\nEnjoy!\nTowards the end of this video, I also explain how I got up to 11.5M IOPS without having to keep CPUs 100% busy, with batched I\/O submission and completion checks I reduced the CPU usage pretty noticeably. Also, with properly configuring I\/O polling queues in the Linux kernel, I got close to 10M IOPS by using just 10 CPUs (leaving almost 70% of CPU time idle)."},{"title":"Achieving 11M IOPS & 66 GiB\/s IO on a Single ThreadRipper Workstation","link":"https:\/\/tanelpoder.com\/posts\/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation\/","pubDate":"Fri, 29 Jan 2021 07:40:33 -0500","guid":"https:\/\/tanelpoder.com\/posts\/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation\/","description":"TL;DR Modern disks are so fast that system performance bottleneck shifts to RAM access and CPU. With up to 64 cores, PCIe 4.0 and 8 memory channels, even a single-socket AMD ThreadRipper Pro workstation makes a hell of a powerful machine - if you do it right!\nUpdate 1: Since 2021, networks have gotten faster and now it is possible to have high-speed reliable remote I\/O even in cloud VMs!"},{"title":"Measuring Java JVM thread activity in Linux with task state sampling","link":"https:\/\/tanelpoder.com\/posts\/measuring-java-jvm-thread-level-activity-on-linux\/","pubDate":"Sat, 28 Nov 2020 17:52:09 -0500","guid":"https:\/\/tanelpoder.com\/posts\/measuring-java-jvm-thread-level-activity-on-linux\/","description":"FYI: I will be speaking about similar topics at P99Conf in Oct 2023 (online, free event)\nThere are plenty of JVM profilers out there, but before attaching with a profiler, you can get a high-level overview by just sampling the Linux \/proc\/PID\/comm and \/proc\/PID\/stat fields to see which JVMs threads are most active and whether they’re burning CPU or are blocked by some I\/O.\nWhen creating threads in Java, you can name them using the following syntax: Thread t = new Thread("MyThread") or thread."},{"title":"High System Load with Low CPU Utilization on Linux?","link":"https:\/\/tanelpoder.com\/posts\/high-system-load-low-cpu-utilization-on-linux\/","pubDate":"Fri, 20 Nov 2020 11:31:58 -0500","guid":"https:\/\/tanelpoder.com\/posts\/high-system-load-low-cpu-utilization-on-linux\/","description":"In this post I will show you how to break down Linux system load by the load contributor or reason. You can drill down into the “linux system load in thousands” and “high system load, but low CPU utilization” problem patterns too.\nIntroduction - terminology Troubleshooting high system load on Linux Drilling down deeper - WCHAN Drilling down deeper - kernel stack How to troubleshoot past problems Summary Further reading Introduction - Terminology The system load metric aims to represent the system “resource demand” as just a single number."},{"title":"SchedLat: a Low Tech Script for Measuring Process CPU Scheduling Latency on Linux","link":"https:\/\/tanelpoder.com\/posts\/schedlat-low-tech-script-for-measuring-cpu-scheduling-latency-on-linux\/","pubDate":"Wed, 26 Feb 2020 23:18:52 -0500","guid":"https:\/\/tanelpoder.com\/posts\/schedlat-low-tech-script-for-measuring-cpu-scheduling-latency-on-linux\/","description":"As you may know, I like to use low tech scripts when possible, ideally such ones that don’t even require you to be root. I prefer simple and “boring” tools as a starting point simply because of the production reality at my customers in traditional enterprise IT. Systems where RHEL6 (and clones) seem to be the most common distros (with Linux kernel 2.6.32) and it’s not too unusual to see RHEL5 occasionally either."},{"title":"pSnapper: Linux Process Snapper Page and Videos","link":"https:\/\/tanelpoder.com\/posts\/linux-process-snapper-videos\/","pubDate":"Wed, 03 Apr 2019 16:03:59 -0400","guid":"https:\/\/tanelpoder.com\/posts\/linux-process-snapper-videos\/","description":"I have created a new page & uploaded videos of my Linux Process Snapper hacking session:\nResources Homepage: https:\/\/tp.dev\/psnapper GitHub: https:\/\/github.com\/tanelpoder\/psnapper Videos Video 1: Exploring \/proc Filesystem & System Calls Hacking Session Video 2: Linux Process Snapper Demo Slide(s) The only slide I showed during the hacking session is this:\nAs I didn’t get to cover some planned Linux tracing topics today, I guess this means we’ll have another hacking session some day!"},{"title":"Tutorial: Troubleshooting Linux SSH Login Delay - Why does logging in always take 10 seconds?","link":"https:\/\/tanelpoder.com\/posts\/troubleshooting-linux-ssh-logon-delay-always-takes-10-seconds\/","pubDate":"Thu, 07 Mar 2019 12:16:58 -0500","guid":"https:\/\/tanelpoder.com\/posts\/troubleshooting-linux-ssh-logon-delay-always-takes-10-seconds\/","description":"As I’m delivering my Linux Troubleshooting training soon, I am going to blog about some typical issues and techniques we’ll troubleshoot in the class too.\nI’ll start from a relatively simple problem - logging in to a server via SSH always takes 10 seconds. The delay seems to be pretty constant, there don’t seem to be major network problems and the server is not overloaded. Yet, remote logins always take 10 seconds."},{"title":"RAM is the new disk \u2013 and how to measure its performance \u2013 Part 3 \u2013 CPU Instructions & Cycles","link":"https:\/\/tanelpoder.com\/2015\/11\/30\/ram-is-the-new-disk-and-how-to-measure-its-performance-part-3-cpu-instructions-cycles\/","pubDate":"Mon, 30 Nov 2015 06:45:58 +0000","guid":"https:\/\/tanelpoder.com\/2015\/11\/30\/ram-is-the-new-disk-and-how-to-measure-its-performance-part-3-cpu-instructions-cycles\/","description":"

If you haven\u2019t read the previous parts of this series yet, here are the links:<\/em> [ Part 1<\/a> | Part 2<\/a> ].<\/p>\n

A Refresher<\/h3>\n

In the\u00a0first part<\/a>\u00a0of this series I said\u00a0that RAM access is\u00a0the slow component of a modern in-memory database engine and for performance you\u2019d want to reduce RAM access as much as possible.\u00a0Reduced memory traffic thanks to the\u00a0new columnar data formats is\u00a0the most important enabler for the awesome In-Memory processing performance and SIMD is just icing on the cake.<\/p>\n

In the second part<\/a> I also showed how to measure the CPU efficiency of your (Oracle) process using a Linux perf stat<\/code> command. How well your\u00a0applications\u00a0actually utilize your CPU execution units depends on many factors. The biggest factor is your process\u2019es cache efficiency that depends on the CPU cache size and your application\u2019s memory access patterns. Regardless of\u00a0what the OS CPU accounting tools like top<\/code> or vmstat<\/code> may show you, your \u201c100% busy\u201d CPUs may actually spend a significant amount\u00a0of their cycles internally idle, with a\u00a0stalled pipeline, waiting for some event (like a memory line arrival from RAM) to happen.<\/p>\n

Luckily there are\u00a0plenty of tools for measuring what\u2019s actually going on inside the CPUs, thanks to modern processors\u00a0having CPU Performance Counters (CPC) built in to them.<\/p>\n

A key derived metric for understanding CPU-efficiency is the IPC<\/strong> (instructions per cycle). Years ago people were actually talking about the inverse metric CPI<\/strong> (cycles per instruction) as on average it took more than one CPU cycle to complete an instruction\u2019s execution\u00a0(again, due to the abovementioned reasons like memory stalls). However, thanks to today\u2019s superscalar processors with out-of-order execution on a modern CPU\u2019s\u00a0multiple execution units \u2013 and with large CPU caches \u2013 a\u00a0well-optimized application can execute multiple instructions per a single CPU cycle, thus\u00a0it\u2019s more natural to use the IPC (instructions-per-cycle) metric. With IPC, higher\u00a0is better.<\/p>"},{"title":"RAM is the new disk \u2013 and how to measure its performance \u2013 Part 2 \u2013 Tools","link":"https:\/\/tanelpoder.com\/2015\/09\/21\/ram-is-the-new-disk-and-how-to-measure-its-performance-part-2-tools\/","pubDate":"Mon, 21 Sep 2015 08:20:18 +0000","guid":"https:\/\/tanelpoder.com\/2015\/09\/21\/ram-is-the-new-disk-and-how-to-measure-its-performance-part-2-tools\/","description":"

[ part 1<\/a> | part 2 |\u00a0part 3<\/a> ]<\/p>\n

In the previous article<\/a> I explained that the main requirement for high-speed in-memory data scanning is column-oriented storage format for in-memory data. SIMD instruction processing is just icing on the cake. Let’s dig deeper. This is a long post, you’ve been warned.<\/p>\n

Test Environment<\/h2>\n

I will cover full test results in the next article in this series. First, let’s look into the test setup, environment\u00a0and what tools I used for peeking inside CPU hardware.<\/p>\n

I was running the tests on a relatively old machine with 2 CPU sockets, with 6-core CPUs in each socket (2s12c24t):<\/p>\n

$ egrep \"MHz|^model name\" \/proc\/cpuinfo | sort | uniq -c\n     24 cpu MHz\t\t: 2926.171<\/strong>\n     24 model name\t: Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz\n<\/strong><\/pre>\n
The CPUs support SSE4.2 SIMD extensions (but not the newer AVX stuff):<\/p>\n
$ grep ^flags \/proc\/cpuinfo | egrep \"avx|sse|popcnt\" | sed 's\/ \/\\n\/g' | egrep \"avx|sse|popcnt\" | sort | uniq\npopcnt\nsse\nsse2\nsse4_1\nsse4_2<\/strong>\nssse3\n<\/pre>\n
Even though the \/proc\/cpuinfo above shows the\u00a0CPU clock frequency as 2.93GHz, these CPUs have Intel Turboboost feature that allows some cores run at up to 3.33GHz frequency when\u00a0not all cores are fully busy and the\u00a0CPUs aren’t too hot.<\/p>\n
Indeed, the turbostat<\/em> command below shows that the CPU core executing\u00a0my Oracle process was running at 3.19GHz frequency:<\/p>\n
# turbostat -p sleep 1\npk cor CPU    %c0  GHz<\/strong>  TSC SMI    %c1    %c3    %c6 CTMP   %pc3   %pc6\n             6.43 3.02 2.93   0  93.57   0.00   0.00   59   0.00   0.00\n 0   0   0   4.49 3.19 2.93   0  95.51   0.00   0.00   46   0.00   0.00\n 0   1   1  10.05 3.19 2.93   0  89.95   0.00   0.00   50\n 0   2   2   2.48 3.19 2.93   0  97.52   0.00   0.00   45\n 0   8   3   2.05 3.19 2.93   0  97.95   0.00   0.00   44\n 0   9   4   0.50 3.20 2.93   0  99.50   0.00   0.00   50\n 0  10   5 100.00 3.19<\/span><\/strong> 2.93   0   0.00   0.00   0.00   59\n 1   0   6   6.25 2.23 2.93   0  93.75   0.00   0.00   44   0.00   0.00\n 1   1   7   3.93 2.04 2.93   0  96.07   0.00   0.00   43\n 1   2   8   0.82 2.15 2.93   0  99.18   0.00   0.00   44\n 1   8   9   0.41 2.48 2.93   0  99.59   0.00   0.00   41\n 1   9  10   0.99 2.35 2.93   0  99.01   0.00   0.00   43\n 1  10  11   0.76 2.36 2.93   0  99.24   0.00   0.00   44\n\n<\/pre>\n
I will come back to this CPU frequency turbo-boosting later when explaining\u00a0some performance metrics.<\/p>\n
I ran the experiments in Oct\/Nov 2014, so used a relatively early Oracle 12.1.0.2.1 version with a bundle patch (19189240) for in-memory stuff.<\/p>\n
The test was deliberately very simple as I was researching raw in-memory scanning and filtering speed and was not looking into join\/aggregation performance. I was running the query below with different hints and parameters to change access path options:<\/p>\n
SELECT COUNT(cust_valid) FROM customers_nopart c WHERE cust_id > 0\n<\/pre>"},{"title":"RAM is the new disk \u2013 and how to measure its performance \u2013 Part 1 \u2013 Introduction","link":"https:\/\/tanelpoder.com\/2015\/08\/09\/ram-is-the-new-disk-and-how-to-measure-its-performance-part-1\/","pubDate":"Sun, 09 Aug 2015 23:26:55 +0000","guid":"https:\/\/tanelpoder.com\/2015\/08\/09\/ram-is-the-new-disk-and-how-to-measure-its-performance-part-1\/","description":"
[ part 1 | part 2<\/a> | part 3<\/a> ]<\/p>\n
RAM is the new disk<\/strong>, at least in the In-Memory computing world.<\/p>\n
No, I am not talking about Flash here, but Random Access Memory \u2013 RAM as in SDRAM. I\u2019m by far not the first one to say it. Jim Gray<\/a>\u00a0wrote this in 2006:\u00a0\u201cTape is dead, disk is tape, flash is disk, RAM locality is king\u201d<\/em> (presentation<\/a>).<\/p>\n
Also, I\u2019m\u00a0not\u00a0going to talk\u00a0about how RAM is faster than disk (everybody knows that), but in fact how RAM is the slow<\/em> component of\u00a0an in-memory processing\u00a0engine.<\/p>\n
I will use Oracle\u2019s In-Memory column store and the hardware performance counters<\/em> in modern CPUs for drilling down into\u00a0the low-level hardware performance metrics about CPU efficiency and memory access.<\/p>\n
But let\u2019s first get started by looking a few years into past into the old-school disk IO and index based SQL performance bottlenecks :)<\/p>"},{"title":"What the heck are the \/dev\/shm\/JOXSHM_EXT_x files on Linux?","link":"https:\/\/tanelpoder.com\/2014\/05\/09\/what-the-heck-are-the-devshmjoxshm_ext_x-files-on-linux\/","pubDate":"Fri, 09 May 2014 20:12:01 +0000","guid":"https:\/\/tanelpoder.com\/2014\/05\/09\/what-the-heck-are-the-devshmjoxshm_ext_x-files-on-linux\/","description":"
There was an interesting question in Oracle-L<\/a> about the JOXSHM_EXT_* files in \/dev\/shm directory on Linux. Basically something like this:<\/p>\n
$ ls -l \/dev\/shm\/*<\/strong> | head\n-rwxrwx--- 1 oracle dba 4096 Apr 18 10:16 \/dev\/shm\/JOXSHM_EXT_<\/strong><\/span>0_LIN112_1409029\n-rwxrwx--- 1 oracle dba 4096 Apr 18 10:16 \/dev\/shm\/JOXSHM_EXT_100_LIN112_1409029\n-rwxrwx--- 1 oracle dba 4096 Apr 18 10:16 \/dev\/shm\/JOXSHM_EXT_101_LIN112_1409029\n-rwxrwx--- 1 oracle dba 4096 Apr 18 10:23 \/dev\/shm\/JOXSHM_EXT_102_LIN112_1409029\n-rwxrwx--- 1 oracle dba 4096 Apr 18 10:23 \/dev\/shm\/JOXSHM_EXT_103_LIN112_1409029\n-rwxrwx--- 1 oracle dba 36864 Apr 18 10:23 \/dev\/shm\/JOXSHM_EXT_104_LIN112_1409029\n...<\/pre>\n
There are a few interesting MOS articles about these files and how\/when to get rid of those (don\u2019t remove any files before reading the notes!), but none of these articles explain why these JOXSHM (and PESHM) files are needed at all:<\/p>\n