{"generator":"Jekyll","link":[{"@attributes":{"href":"https:\/\/richardstartin.github.io\/feed.xml","rel":"self","type":"application\/atom+xml"}},{"@attributes":{"href":"https:\/\/richardstartin.github.io\/","rel":"alternate","type":"text\/html"}}],"updated":"2023-11-13T10:24:21+00:00","id":"https:\/\/richardstartin.github.io\/feed.xml","title":"Richard Startin\u2019s Blog","entry":[{"title":"How Async-Profiler\u2019s crash handler works","link":{"@attributes":{"href":"https:\/\/richardstartin.github.io\/posts\/async-profiler-crash-handler","rel":"alternate","type":"text\/html","title":"How Async-Profiler\u2019s crash handler works"}},"published":"2023-11-12T00:00:00+00:00","updated":"2023-11-12T00:00:00+00:00","id":"https:\/\/richardstartin.github.io\/posts\/async-profiler-crash-handler","content":"

Raw pointers in C\/C++ open up entire classes of error that are practically unimaginable in higher level languages.\nSo why does anybody use them at all?\nUnfortunately, it\u2019s impossible to write a profiler without getting close to some of the sharp edges of unsafe memory. \nThe async-profiler code base contains a lot of low level tricks, and it\u2019s worth studying how some of them work.<\/p>\n\n

This post is about how async-profiler safely dereferences arbitrary pointers without crashing the JVM, and why it needs to do that sometimes.<\/p>\n\n

    \n
  1. Signals and signal handlers<\/a><\/li>\n
  2. Frame pointer unwinding<\/a><\/li>\n
  3. SafeAccess::load<\/code><\/a><\/li>\n
  4. Replacing crash handlers<\/a><\/li>\n<\/ol>\n\n

    Signals and signal handlers<\/h1>\n\n

    Linux allows threads to be interrupted by signals, which are identified by a signal number.\nSignals can be sent to threads via a number of syscalls, such as tgkill<\/code><\/a>, or indirectly via profiling services like itimer<\/a> or perf_event_open<\/a>.\nWhen a thread transitions between the kernel and user space, such as when the thread is scheduled to run on the CPU, the kernel checks if there are any pending signals for the thread.\nIf there is a pending signal, and the signal isn\u2019t blocked by a signal mask<\/a>, and there is a signal handler<\/em> installed for the signal, then the thread is halted and the signal handler is invoked.\nA signal based sampling profiler consists of a mechanism for choosing threads and sending signals (usually SIGPROF<\/code>) to them, then a signal handler for collecting information about the state of the thread (e.g. unwinding the stack).\nSignals and signal handlers are used for a number of other things, such as killing the process (SIGKILL<\/code>), or signalling that something has gone wrong (SIGSEGV<\/code>, SIGABRT<\/code>, SIGBUS<\/code>).\nIf you just want your program not to crash when performing memory unsafe operations, you can install signal handlers for error signals and swallow them, though this will cause other problems without care.<\/p>\n\n

    Frame pointer unwinding<\/h1>\n\n

    You don\u2019t need to handle segfaults if your code can\u2019t segfault, so why not just use a programming language with a compiler which verifies memory safety?\nUnfortunately, you wouldn\u2019t have much of a profiler if you did that (without unsafe blocks, anyway).\nConsider the simplest stack unwinding approach using frame pointers.<\/p>\n\n

    Signal handlers are invoked with a ucontext<\/code> which consists of a program counter, a frame pointer, and a stack pointer.<\/p>\n\n

    The program counter (pc<\/code>) is the address of the code currently being executed.\nIt can be used to find the currently executing function by binary searching a compiled binary\u2019s text section (where the code is found) or the JVM\u2019s code heap for JIT compiled code.\nIt is enough to determine the currently executing instruction (which is how instruction profiling works).\nHowever, the program counter is not enough to determine the caller of the interrupted function.<\/p>\n\n

    The frame pointer (fp<\/code>) determines the start address of the stack frame, which contains function parameters and local variables.\nThe stack frame is like a snapshot of the registers before calling a function, so that when the called function returns, the calling context can be restored.\nObtaining the frame pointer to the calling frame from the callee frame is known as unwinding<\/em>.\nIf<\/em> a binary is compiled with frame pointers, it\u2019s really simple: on x64, the rbp<\/code> register contains the caller\u2019s frame pointer.\nSo frame pointer unwinding consists of the following (on x64, the registers differ on aarch64):<\/p>\n\n

      \n
    1. Record the program counter from the ucontext<\/code> (this can be used later to determine things like function name, line number, and the instruction)<\/li>\n
    2. Read the caller\u2019s frame pointer from rbp<\/code><\/li>\n
    3. Check termination (this could be if the frame pointer is zero, if the frame pointer is outside the bounds of the stack, or in async-profiler if a Java frame has been found).<\/li>\n
    4. Restore the caller frame from the caller\u2019s frame pointer (dereference the frame pointer).<\/li>\n
    5. Record the program counter (see point 1)<\/li>\n
    6. Go to step 2<\/li>\n<\/ol>\n\n

      This is about as fast and simple as stack unwinding gets, but the problem is lots of binaries are compiled without frame pointers.\nIf a binary is compiled without frame pointers, rbp<\/code> contains arbitrary data and isn\u2019t safe to dereference.\nDepending on the exact value stored in the register, dereferencing it may produce a bogus frame but will probably segfault (generate a SIGSEGV<\/code> signal).\nI am unaware of any way to determine that an arbitrary binary was compiled with frame pointers or not, so the only way to determine whether rbp<\/code> is a frame pointer or not is to dereference it and hopefuly get a frame.\nThis puts frame pointer unwinding outside the realm of compile time memory safety checks: either mitigate at runtime or don\u2019t do it.<\/p>\n\n

      SafeAccess::load<\/code><\/h1>\n\n

      Async-profiler uses the following mechanism:<\/p>\n

        \n
      1. Always dereference untrusted data via a function called SafeAccess::load<\/code>.<\/li>\n
      2. Install a SIGSEGV<\/code> handler.<\/li>\n
      3. Check in the SIGSEGV<\/code> handler if the SIGSEGV<\/code> signal\u2019s pc<\/code> came from SafeAccess::load<\/code>. If it did, patch things up as if the function returned zero, otherwise pass the signal on.<\/li>\n<\/ol>\n\n

        There are some details which are worth paying attention to in the function declaration (source<\/a>):<\/p>\n\n

           #ifdef __clang__\n<\/span>   #  define NOINLINE __attribute__((noinline))\n<\/span>   #else\n<\/span>   #  define NOINLINE __attribute__((noinline,noclone))\n<\/span>   #endif\n<\/span>   ...<\/span>\n   NOINLINE<\/span> __attribute__<\/span>((<\/span>aligned<\/span>(<\/span>16<\/span>)))<\/span>\n    static<\/span> void<\/span>*<\/span> load<\/span>(<\/span>void<\/span>**<\/span> ptr<\/span>)<\/span> {<\/span>\n        return<\/span> *<\/span>ptr<\/span>;<\/span>\n    }<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
        Firstly, the function can\u2019t be inlined because of the compiler attributes applied to it, because if it were inlined, it wouldn\u2019t be possible to determine whether the SIGSEGV<\/code>\u2019s pc<\/code> came from the function or not.\nNext, the function is aligned so that a pc<\/code> can be determined to be within the function\u2019s code\u2019s address range easily.<\/p>\n\n
        In the signal handler installed when starting the profiler, the following code executes:<\/p>\n\n
            uintptr_t<\/span> length<\/span> =<\/span> SafeAccess<\/span>::<\/span>skipLoad<\/span>(<\/span>pc<\/span>);<\/span>\n    if<\/span> (<\/span>length<\/span> ><\/span> 0<\/span>)<\/span> {<\/span>\n        \/\/ Skip the fault instruction, as if it successfully loaded NULL<\/span>\n        frame<\/span>.<\/span>pc<\/span>()<\/span> +=<\/span> length<\/span>;<\/span>\n        frame<\/span>.<\/span>retval<\/span>()<\/span> =<\/span> 0<\/span>;<\/span>\n        return<\/span>;<\/span>\n    }<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
        This code figures out how far to skip ahead to simulate returning from the function, having set the return value to zero.\nThe way the number of instructions to skip is determined is a little cryptic (only for x64 below):<\/p>\n\n
            static<\/span> uintptr_t<\/span> skipLoad<\/span>(<\/span>uintptr_t<\/span> pc<\/span>)<\/span> {<\/span>\n        if<\/span> ((<\/span>pc<\/span> -<\/span> (<\/span>uintptr_t<\/span>)<\/span>load<\/span>)<\/span> <<\/span> 16<\/span>)<\/span> {<\/span>\n            return<\/span> *<\/span>(<\/span>u16<\/span>*<\/span>)<\/span>pc<\/span> ==<\/span> 0x8b48<\/span> ?<\/span> 3<\/span> :<\/span> 0<\/span>;<\/span>  \/\/ mov rax, [reg]<\/span>\n        }<\/span>\n        return<\/span> 0<\/span>;<\/span>\n    }<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
        Checking if the pc<\/code> is in the address range of the load<\/code> function is straightforward enough, but what on earth is 0x8b48<\/code>?\nDereferencing a pc<\/code> gives a sequence of bytes corresponding to the instructions. \nCasting to a u16*<\/code> truncates to the most significant 2 bytes, which has the effect of ignoring the register the value would be loaded into.\nThe 0x8b48<\/code> check verifies that the pc<\/code> is pointing at a mov<\/code> operation, otherwise something has gone wrong.\nFinally, 3 bytes need to be skipped over: 2 for the load, and one for the destination register.<\/p>\n\n
        Replacing crash handlers<\/h1>\n\n
        The JVM has its own SIGSEGV<\/code> handler already, in fact, it\u2019s quite important because it\u2019s part of the uncommon trap mechanism<\/a> for deoptimisation of speculative optimisations.\nThis means Async-profiler must delegate to the JVM\u2019s SIGSEGV<\/code> handler in order not to break the JVM.\nIt does this by returning a function pointer to the old handler<\/a> when installing its own handler<\/a>, and then delegates to it if the pc<\/code> didn\u2019t come from SafeAccess::load<\/code>.\nThe JVM doesn\u2019t exactly like signals it handles being overridden like this, given the central importance of SIGSEGV<\/code> signals to the lifecycle of JIT compiled code, and running with -Xcheck:jni<\/code> will spit out angry warning messages about it. \nThese messages are annoying but much better than simply dropping samples when the JVM is not executing Java code, because Java code can call native code, and this is sometimes where performance problems lie.<\/p>","author":{"name":{}},"category":{"@attributes":{"term":"profiling"}},"summary":"Raw pointers in C\/C++ open up entire classes of error that are practically unimaginable in higher level languages. So why does anybody use them at all? Unfortunately, it\u2019s impossible to write a profiler without getting close to some of the sharp edges of unsafe memory. The async-profiler code base contains a lot of low level tricks, and it\u2019s worth studying how some of them work."},{"title":"Understanding Request Latency with Profiling","link":{"@attributes":{"href":"https:\/\/richardstartin.github.io\/posts\/wallclock-profiler","rel":"alternate","type":"text\/html","title":"Understanding Request Latency with Profiling"}},"published":"2023-09-05T00:00:00+00:00","updated":"2023-09-05T00:00:00+00:00","id":"https:\/\/richardstartin.github.io\/posts\/wallclock-profiler","content":"
        It can be hard to figure out why response times are high in Java applications.\nIn my experience, people either apply a process of elimination to a set of recent commits, or might sometimes use profiles of the system to explain changes in metrics.\nMaking guesses about recent commits can be frustrating for a number of reasons, but mostly because even if you pinpoint the causal change, you still might not know why<\/em> it was a bad change and are left in limbo.\nIn theory, using a profiler makes root cause analysis a part of the triage process, so adopting continuous profiling should make this whole process easier, but using profilers can be frustrating because you\u2019re using the wrong type<\/em> of profile for analysis.\nLots of profiling data focuses on CPU time, but the cause of your latency problem may be related to time spent off CPU instead. \nThis post is about Datadog\u2019s Java wallclock profiler, which I worked on last year, and explores how to improve request latency without making any code changes, or even seeing the code for that matter.<\/p>\n\n
        \n  
        This is a personal blog and everything written below is my own personal opinion.<\/p>\n<\/blockquote>\n\n
          \n  
        1. From metrics to profiles<\/a><\/li>\n  
        2. async-profiler<\/a><\/li>\n  
        3. Wallclock profiling<\/a>    
            \n      
          1. Sampling strategy<\/a><\/li>\n      
          2. Tracer managed thread filter<\/a><\/li>\n    <\/ol>\n  <\/li>\n  
          3. Latency investigation<\/a>    
              \n      
            1. Increasing the heap size<\/a><\/li>\n      
            2. Increasing the number of vCPUs<\/a><\/li>\n    <\/ol>\n  <\/li>\n  
            3. What about cost?<\/a><\/li>\n<\/ol>\n\n
              From metrics to profiles<\/h2>\n\n
              Suppose you have metrics which show you that request latency is too high for a particular endpoint.\nHere is the endpoint level latency timeseries for a slow endpoint GET \/allocation-stalls<\/code> (the name may create a sense of foreboding) in a very contrived demo Java application.<\/p>\n\n
              \"High<\/p>\n\n
              The response time is about 3-4s, which we would like to improve somehow without undertaking an R&D project.\nUnless you have detailed tracing, you might start looking at profiles to do this.\nOpenJDK has a built-in profiler called Java Flight Recorder (JFR), which Datadog has used historically for collecting CPU profiles from Java services.\nHere\u2019s what a flamegraph of JFR\u2019s jdk.ExecutionSample<\/code> events looks like after being processed by Datadog\u2019s profiling backend:<\/p>\n\n
              \"JFR<\/p>\n\n
              To the far left there is some JVM activity derived from JFR\u2019s GC events. \nNext to that there are some stack traces for Jetty\u2019s routing (see JettyHandler.doHandle()<\/code>).\nTo the right are stack traces for handling the requests in another thread pool (see ThreadPoolExecutor$Worker.run()<\/code>).\nThere\u2019s not much context beyond the frame names, and the most we have to go on is JavaServer.lambda$allocationStall$28()<\/code> which we can probably guess is related to handling GET \/allocation-stalls<\/code> requests by its name.\nJava code can be very generic and it\u2019s not always straightforward to map frames to context by frame names alone, though in this case it\u2019s easy enough. \nExpanding it gets us to where I think most people turn away from profiling, assuming it was as easy to find the relevant frame as in this contrived example:<\/p>\n\n
              \"JFR<\/p>\n\n
              This points to random number generation being the bottleneck, which you aren\u2019t likely to speed up.\nYou may be able to reduce the number of random numbers generated, but in this contrived example, this would require renegotiating requirements.\nPart of the problem is that this flamegraph doesn\u2019t show the complete picture. \nWithout having the full picture, we can\u2019t be sure we know what the bottleneck is, and might just be looking at the frame we happen to measure<\/em> the most.\nThis is where async-profiler comes in.<\/p>\n\n
              async-profiler<\/h2>\n\n
              Some time ago, Datadog started experimenting with using async-profiler<\/a> instead of JFR\u2019s execution sampler in its continuous profiling product.\nThere were two primary motivations for doing this:<\/p>\n\n
                \n  
              1. It\u2019s better\n    
                  \n      
                1. async-profiler has a much better CPU profiler than JFR, primarily because it can use CPU time (or other perf events) to schedule samples, whereas JFR uses wall time and a state filter.<\/li>\n      
                2. async-profiler has several approaches to unwinding native parts of the stack, whereas JFR does not unwind stacks with native frames.<\/li>\n      
                3. async-profiler reports failed samples so even if it\u2019s not possible to get a full stack trace, it doesn\u2019t distort the weights of other samples.<\/li>\n    <\/ol>\n  <\/li>\n  
                4. We could modify the source code in a fork\n    
                    \n      
                  1. It\u2019s easy to make changes to the source code and get those changes shipped in a timely manner.<\/li>\n      
                  2. JFR doesn\u2019t allow any contextualisation of samples, which makes presenting the data it reports in useful and engaging ways challenging. It was quite easy to extend async-profiler to support arbitrary, multi-dimensional labeling, whereas the process to get this functionality into JFR would be long-winded.<\/li>\n    <\/ol>\n  <\/li>\n<\/ol>\n\n
                    So async-profiler\u2019s excellent codebase gave us a starting point for implementing some of the features we had in mind.<\/p>\n\n
                    Below is a flamegraph of the same application as above recorded with Datadog\u2019s customisation of async-profiler. \nNotice the ZGC activity to the left and compare it to the GC activity we could infer from JFR above, there\u2019s more detail and it\u2019s more obvious that the service has a GC problem.<\/p>\n\n
                    \"async-profiler<\/p>\n\n
                    Having access to the source code made it possible to add new fields to event types, such as correlation identifiers.\nDecorating samples with span ids makes it possible to link each sample to the endpoint being served, rather than guess based on the frame name (which only gets harder outside of contrived examples).\nThis makes it possible to filter the flamegraph by endpoint instead of guessing based on the frame names:<\/p>\n\n
                    \"async-profiler<\/p>\n\n
                    The green frames to the left, which are JNI calls into the lz4-java library, are missing from the JFR flamegraph because JFR won\u2019t unwind stacks when the JVM is executing native code.\nIn this particular case, this doesn\u2019t actually distort the flamegraph much<\/em>, because LZ4 compression is very fast and doesn\u2019t need as much CPU time as the random number generation we already know about, but if it were the other way around the JFR flamegraph would be very misleading.<\/p>\n\n
                    Even though the flamegraph now shows a complete picture of CPU time, it still points towards optimising random number generation, which we can\u2019t do.\nThe problem is now that CPU time isn\u2019t always enough to understand latency.<\/p>\n\n
                    Wallclock profiling<\/h2>\n\n
                    As used in Datadog\u2019s profiler, async-profiler\u2019s CPU profiler uses CPU time to schedule sampling, which means time spent off-CPU will not be sampled. \nIn Java Thread<\/code> terms, a CPU profiler can only sample threads in the RUNNABLE<\/code> state, and ignores threads in the BLOCKED<\/code>, TIMED_WAITING<\/code>, and WAITING<\/code> states, but a wallclock profiler<\/em> can sample threads in any state.\nasync-profiler also has a wallclock profiler, but we found we needed to tailor it to our requirements.<\/p>\n\n
                    Sampling strategy<\/h3>\n\n
                    The first issue we encountered was with overhead (for our own particular requirements, don\u2019t be discouraged from using async-profiler\u2019s wallclock profiler in a different context).\nCPU profiling is always very lightweight because the overhead is proportional to the number of cores, whereas for wallclock profiling, overhead scales with the number of threads.\nasync-profiler\u2019s wallclock profiler samples sets of up to 8 threads round-robin, and to keep the sampling interval consistent as the number of threads changes, will adjust the sampling interval to as low 100 microseconds.\nUnless the user provides a thread filter expression to control how many threads to sample, the overhead of this approach can be quite high.\nUsing a predefined thread filter assumes the user knows where the problems are ahead of time, and they might do when doing offline\/non-continuous profiling.\nIt can also produce quite large recordings: in 60s we might record as many as 4.8 million samples, which, at about 64 bytes per JFR event, corresponds to about 250MiB.<\/p>\n\n
                    We wanted smaller recordings and needed to be very conservative about overhead, so we changed the way threads are sampled. \nWe implemented reservoir sampling of threads so we could take a uniform sample of a fixed number of threads at a constant sampling interval. \nThis gives a much more manageable upper bound on overhead and recording size: in 60s we can sample 16 threads at 100Hz and record at most 96,000 samples, which is just under 6MiB of JFR events.\nThis approach reintroduces the problem async-profiler solves with its interval adjustment when the number of threads changes over time because this changes the probability of sampling a thread (the reservoir is fixed size), but this can be solved by recording the number of live threads each sample interval and upscaling later.<\/p>\n\n
                    Tracer managed thread filter<\/h3>\n\n
                    High thread counts also can also reduce sample relevance, without a thread filter. \nFor latency analysis it would be ideal to bias towards threads on the critical path. \nThere are two kinds of noncritical threads for typical applications:<\/p>\n\n
                      \n  
                    1. idle threads in over-provisioned thread pools, each waiting for items to appear in a queue.<\/li>\n  
                    2. background threads such as metrics reporters.<\/li>\n<\/ol>\n\n
                      In the screenshot below, at least half of the samples are of the 200 workers (which is too many for this application) waiting for work to arrive on a LinkedBlockingQueue<\/code>.<\/p>\n\n
                      \"all<\/p>\n\n
                      This isn\u2019t very useful information: while fine-tuning the size of this thread pool would be generally beneficial, the threads aren\u2019t blocking the threads processing requests.<\/p>\n\n
                      What\u2019s worse is when the samples are rendered as a timeline of a particular trace, the coverage is quite sparse (the grey bars are PARKED<\/code> walltime samples, the blue bars are CPU samples).<\/p>\n\n
                      \"all<\/p>\n\n
                      Sampling idle or background activity reduces the coverage of traces.<\/p>\n\n
                      Both of these problems can be solved by only sampling threads which have an active trace context set up by the tracer.\nFirstly, tracing is designed to record request latency, so if there is a trace context associated with a thread for a time period, there is a good chance the work is being done to complete a request.\nThanks to context propagation, which propagates trace context into thread pools, any thread which ends up working to complete the request can be sampled.\nSecondly, biasing sampling towards traced activity helps focus<\/em> the profiler on explaining trace latency, so we have the situation where the tracer can tell you how long a request took, and there\u2019s a good chance that there will be wallclock samples to explain why it took so long.\nWhenever we have too much work to do, we can do a better job by prioritising some tasks over others, and it\u2019s the same for sampling threads.<\/p>\n\n
                      This results in a flamegraph focused on latency-critical work:<\/p>\n\n
                      \"traced<\/p>\n\n
                      and when we look at the timeline of a particular trace, we have much better coverage (i.e. more grey bars for more threads)<\/p>\n\n
                      \"filtered<\/p>\n\n
                      This is implemented by repurposing async-profiler\u2019s thread filter, so that it is only applied to the wallclock profiler, and is controlled by the tracer. \nThe current thread is added to a bitset of thread ids whenever the current thread has a trace associated with it, and removed from the bitset whenever the thread no longer has a trace associated with it.\nIf you have used a tracing API, you will be familiar with spans<\/em> (logical operations within traces) and scopes<\/em> (activations of spans on a thread). \nThe tracer pushes a scope to a stack whenever a span is activated on a thread, so that any subsequent spans can refer to the activated span as a parent.\nModifying the bitset each time a scope is pushed or popped from the scope stack is the most obvious implementation, but would lead to very high overhead for certain async workloads.\nInstead, the bitset is updated only when the scope stack becomes empty or non-empty, which corresponds to whether there is some trace associated with the thread or not, which almost always reduces the number of bitset updates and never increases it.\nFinally, the sampler thread implements reservoir sampling over the bitset of traced threads, rather than over all threads.<\/p>\n\n
                      Latency investigation<\/h2>\n\n
                      The wallclock profiler reports stack traces for any blocking off-CPU activity, which should tell you where<\/em> the blocking operation happened, and why<\/em> the blocking operation occurred by looking at the ancestor frames.\nIt should have been possible to figure out why the request latency is so high from the screenshots so far, but here\u2019s the timeline for a particular request trace again:<\/p>\n\n
                      \"filtered<\/p>\n\n
                      The long grey bars, with durations of about a second, all contain the frame ZPageAllocator::alloc_page_stall()<\/code>, which blocks on PosixSemaphore::wait()<\/code>, with causal ancestor frame ByteBuffer.allocate(int)<\/code>.\nIn the flamegraph scoped to the endpoint, we can see that about 50% of the samples under JavaServer.lambda$allocationStall$29()<\/code> are allocation stalls, so the CPU profile missed roughly half the activity:<\/p>\n\n
                      \"\"<\/p>\n\n
                      Allocation stalls happen with ZGC, when the application threads try to allocate more memory than is currently available.\nThe application thread blocks until the requested allocation can be performed, which incurs latency. \nZGC is effectively applying backpressure on the application threads, throttling concurrent access to memory as if it were a bounded queue, and it does this rather than OOM which is the only alternative.\nAllocation stalls are a symptom of an undersized heap, and there are two choices for mitigation: allocate a larger heap, or make the application allocate less.<\/p>\n\n
                      In this contrived case, there are 30 tasks allocating a pair of 50 MiB ByteBuffer<\/code>s in a threadpool of 10 threads, which translates to 3GiB per request and 1000MiB allocated at any time during the request, with only 1GiB allocated to the heap.\nHaving identified and understood allocation stalls, we can see the heap is too small for the workload.\nWe can\u2019t reduce allocation pressure by reducing the size of the buffers without renegotiating requirements, and we can\u2019t reduce it by object pooling without blocking on the pool (we have less memory than we need) and our buffer pool might be less efficient than ZGC.<\/p>\n\n
                      \n  
                      There is also a JFR event jdk.ZAllocationStall<\/code> which could be used instead of a wallclock profiler to determine that the heap is too small, but it does not collect stacktraces.<\/p>\n<\/blockquote>\n\n
                      Increasing the heap size<\/h3>\n\n
                      Solving this problem should be a simple case of increasing the heap size so more memory can be allocated without stalling.\nThe service should then become CPU bound, and it was already established that random number generation can\u2019t be accelerated, so we wouldn\u2019t be able to remove this bottleneck without redefining the workload.<\/p>\n\n
                      The heap size was increased to 6GiB.\nLooking at the wallclock profile, we can see a change in thread state:<\/p>\n\n
                      \"thread<\/p>\n\n
                      \"thread<\/p>\n\n
                      Before, in the top image, about half the time was in the RUNNABLE<\/code> state and half in the PARKED<\/code> state, afterwards the split is 3:1.\nThe allocation stalls have gone, and none of the PARKED<\/code> time is under the JavaServer.lambda$allocationStall$29()<\/code> frame: the allocation stalls have disappeared.\nLooking at a trace of a particular request, the grey bars which represented parking have gone, and we only see the blue CPU samples:<\/p>\n\n
                      \"timeline<\/p>\n\n
                      It\u2019s clear the way the JVM executes the code has changed as a result of being given more resources, but it hasn\u2019t actually reduced latency.<\/p>\n\n
                      \"latency<\/p>\n\n
                      Increasing the number of vCPUs<\/h3>\n\n
                      The clue about what to fix next is in the timeline for the request: the CPU samples are very sparse in time and across the threads.<\/p>\n\n
                      \"timeline<\/p>\n\n
                      We can also see a strange discrepancy between the amount of wall time sampled in the RUNNABLE<\/code> state and the amount of sampled CPU time:<\/p>\n\n
                      \"cpu\n\"wall<\/p>\n\n
                      If the workload is now CPU bound, as we expect it to be, and<\/em> it\u2019s actually executing, then the CPU time and the wall time should be the same, but there\u2019s 18 seconds of RUNNABLE<\/code> wall time and less than 5s of CPU time.\nThough we don\u2019t have a direct way to show it, this indicates that the threads are competing to get scheduled on too few vCPUs, and if we could allocate more CPU to the container, the responses should become CPU bound.\nAs can be seen in the timeline, the workload is parallelised over 10 threads, but only 2 vCPUs were allocated. \nIncreasing the number of vCPUs to 10, finally, reduces latency dramatically:<\/p>\n\n
                      \"latency<\/p>\n\n
                      Though they will never correspond exactly, the sampled RUNNABLE<\/code> wall time and CPU times have converged:\n\"Wall\n\"CPU<\/p>\n\n
                      and the timeline for a particular trace shows 10 threads with denser blue CPU samples:\n\"timeline\"<\/p>\n\n
                      What about cost?<\/h2>\n\n
                      This probably isn\u2019t very satisfying: the cost of the running service has been increased dramatically.\nHowever, the aim was to reduce request latency for a contrived workload that isn\u2019t amenable to optimisations.\nWhen code is at its efficient frontier (or effectively<\/em> at its efficient frontier without, say, incurring huge engineering costs), there is a tradeoff between latency and cost.\nIf you don\u2019t have an SLA, you can just run this workload on 2vCPUs and a 1GiB heap for very little cost, and though ZGC will throttle allocations and it will be slow, things will basically work.\nIf you do have an SLA, and optimisation is impractical or impossible, and you can\u2019t negotiate requirements, you\u2019ll need to incur costs and allocate more compute resources.\nIt\u2019s quite rare for a Java service to be at its efficient frontier, and there\u2019s usually low hanging fruit surfaced by continuous profiling which can sometimes reduce latency and cost simultaneously.<\/p>","author":{"name":{}},"category":{"@attributes":{"term":"profiling"}},"summary":"It can be hard to figure out why response times are high in Java applications. In my experience, people either apply a process of elimination to a set of recent commits, or might sometimes use profiles of the system to explain changes in metrics. Making guesses about recent commits can be frustrating for a number of reasons, but mostly because even if you pinpoint the causal change, you still might not know why it was a bad change and are left in limbo. In theory, using a profiler makes root cause analysis a part of the triage process, so adopting continuous profiling should make this whole process easier, but using profilers can be frustrating because you\u2019re using the wrong type of profile for analysis. Lots of profiling data focuses on CPU time, but the cause of your latency problem may be related to time spent off CPU instead. This post is about Datadog\u2019s Java wallclock profiler, which I worked on last year, and explores how to improve request latency without making any code changes, or even seeing the code for that matter."},{"title":"Using JFR and JMC to root cause a performance bug in JMC caused by a typo in JFR","link":{"@attributes":{"href":"https:\/\/richardstartin.github.io\/posts\/jmc-hanging","rel":"alternate","type":"text\/html","title":"Using JFR and JMC to root cause a performance bug in JMC caused by a typo in JFR"}},"published":"2022-12-18T00:00:00+00:00","updated":"2022-12-18T00:00:00+00:00","id":"https:\/\/richardstartin.github.io\/posts\/jmc-hanging","content":"
                      When you open a profile in JMC, it normally takes a few seconds, but there are some profiles that JMC struggles to load.\nThis happens when a profile contains events which violate an assumption made in JMC\u2019s parser: events on the same thread are almost always disjoint in their durations.\nWhen JMC parses a JFR file, it splits the events of the same type emitted on the same thread into lanes so that events within a lane are disjoint.\nJMC can handle overlapping events, but assumes this essentially never happens, and exhibits quadratic scaling when events overlap.<\/p>\n\n
                      The good news is that unless you hack it into the JFR file generation in your own customisation of async-profiler, it\u2019s practically impossible to generate events that violate this assumption.\nHowever, if you run this code:<\/p>\n\n
                          public<\/span> static<\/span> void<\/span> main<\/span>(<\/span>String<\/span>...<\/span> args<\/span>)<\/span> throws<\/span> Exception<\/span> {<\/span>\n        if<\/span> (<\/span>args<\/span>.<\/span>length<\/span> <<\/span> 1<\/span>)<\/span> {<\/span>\n            System<\/span>.<\/span>err<\/span>.<\/span>println<\/span>(<\/span>\"provide a directory to write the JFR file to\"<\/span>);<\/span>\n            System<\/span>.<\/span>exit<\/span>(<\/span>1<\/span>);<\/span>\n        }<\/span>\n        try<\/span>(<\/span>Recording<\/span> recording<\/span> =<\/span> new<\/span> Recording<\/span>()){<\/span>\n            recording<\/span>.<\/span>setName<\/span>(<\/span>\"slow\"<\/span>);<\/span>\n            recording<\/span>.<\/span>setDumpOnExit<\/span>(<\/span>true<\/span>);<\/span>\n            recording<\/span>.<\/span>setDestination<\/span>(<\/span>getOrCreateDirectory<\/span>(<\/span>args<\/span>[<\/span>0<\/span>]).<\/span>resolve<\/span>(<\/span>\"slow.jfr\"<\/span>));<\/span>\n            recording<\/span>.<\/span>enable<\/span>(<\/span>\"jdk.FileWrite\"<\/span>);<\/span>\n            recording<\/span>.<\/span>start<\/span>();<\/span>\n            generateFileEvents<\/span>();<\/span>\n            \n        }<\/span>\n    }<\/span>\n\n    private<\/span> static<\/span> void<\/span> generateFileEvents<\/span>()<\/span> throws<\/span> IOException<\/span> {<\/span>\n        Path<\/span> directory<\/span> =<\/span> Files<\/span>.<\/span>createTempDirectory<\/span>(<\/span>\"files\"<\/span>);<\/span>\n        for<\/span> (<\/span>int<\/span> i<\/span> =<\/span> 0<\/span>;<\/span> i<\/span> <<\/span> 100<\/span> *<\/span> 1000<\/span>;<\/span> i<\/span>++)<\/span> {<\/span>\n            try<\/span> (<\/span>RandomAccessFile<\/span> file<\/span> =<\/span> new<\/span> RandomAccessFile<\/span>(<\/span>directory<\/span>.<\/span>resolve<\/span>(<\/span>i<\/span> +<\/span> \"\"<\/span>).<\/span>toFile<\/span>(),<\/span> \"rw\"<\/span>))<\/span> {<\/span>\n               file<\/span>.<\/span>write<\/span>((<\/span>\"data\"<\/span> +<\/span> i<\/span>).<\/span>getBytes<\/span>(<\/span>StandardCharsets<\/span>.<\/span>UTF_8<\/span>));<\/span>\n            }<\/span>\n        }<\/span>\n    }<\/span>\n\n    private<\/span> static<\/span> Path<\/span> getOrCreateDirectory<\/span>(<\/span>String<\/span> dir<\/span>)<\/span> throws<\/span> IOException<\/span> {<\/span>\n        Path<\/span> destination<\/span> =<\/span> Paths<\/span>.<\/span>get<\/span>(<\/span>dir<\/span>);<\/span>\n        if<\/span> (!<\/span>Files<\/span>.<\/span>exists<\/span>(<\/span>destination<\/span>))<\/span> {<\/span>\n            Files<\/span>.<\/span>createDirectory<\/span>(<\/span>destination<\/span>);<\/span>\n        }<\/span>\n        return<\/span> destination<\/span>;<\/span>\n    }<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
                      And then open the .jfr file in JMC, you will see this for quite some time (well over 30s):<\/p>\n\n
                      \"Slow\"<\/p>\n\n
                      Fortunately JMC itself can be profiled as it is loading a profile, using JFR with the command:<\/p>\n\n
                      jcmd <pid of JMC> JFR.start settings=profile filename=jmcrecording.jfr duration=60s\n<\/code><\/pre><\/div><\/div>\n\n
                      After 60s a profile will be dumped containing execution samples of whatever JMC spent its time doing.\nLoading the profile of JMC itself and navigating to Method Profiling<\/em>:<\/p>\n\n
                      \"Method<\/p>\n\n
                      This shows you a flamegraph of CPU samples:<\/p>\n\n
                      \"Cause\"<\/p>\n\n
                      There are over 6000 samples, which is a good number for a JFR CPU profile, which takes samples at about 100Hz.\nIf most activity is during loading the profile is on the same thread, at 100Hz, having 6000 samples means we got one sample every 10ms so don\u2019t have a lot of failed samples, which JFR doesn\u2019t otherwise account for.\nAssuming there is no JNI activity, this means the profile should be reasonably accurate.\nIt\u2019s tempting to scroll to the bottom and look for the widest frame and optimise it.\nThe obvious candidate here are the calls to ImpreciseScaleFactor.targetNumber<\/code>, outlined in black below:<\/p>\n\n
                      \"Red<\/p>\n\n
                      Though there is a little fat to trim in this method, this is a red herring; optimising this method would be a waste of time and it would take an heroic effort to move the needle by focusing here.\nCPU sampling can\u2019t differentiate between a method being slow (requiring constant factors improvements) and being called too often (requiring algorithmic improvements).\nUnfortunately, JFR offers no way to perform a differential diagnosis to rule out constant factors and focus on algorithmic issues.<\/p>\n\n
                      Once you have identified a bottleneck, the easiest way to differentiate between a constant factors and algorithmic issue is honestly just to read the code for a few of the frames towards the bottom of the stacktrace.\nSometimes, this is too complicated, and I have resorted to inserting counter probes in these methods with bytebuddy (but I recently discovered async-profiler can do this<\/a> and I have adopted this approach in preference to instrumentation during analyses).\nIf you have a superlinear algorithm, you see high hit counts at the leaves, and low hit counts in the interior nodes.\nIn any case, reading the code for DisjointBuilder.add<\/code> (outlined in black below):<\/p>\n\n
                      \"DisjointBuilder\"<\/p>\n\n
                      reveals what looks like an optimisation:<\/p>\n\n
                          \/\/ at 77580fe1b483b4daf8f46938c297185b01f32304<\/span>\n\tpublic<\/span> void<\/span> add<\/span>(<\/span>T<\/span> e<\/span>)<\/span> {<\/span>\n\t\tIQuantity<\/span> start<\/span> =<\/span> startAccessor<\/span>.<\/span>getMember<\/span>(<\/span>e<\/span>);<\/span>\n\t\tIQuantity<\/span> end<\/span> =<\/span> endAccessor<\/span>.<\/span>getMember<\/span>(<\/span>e<\/span>).<\/span>in<\/span>(<\/span>start<\/span>.<\/span>getUnit<\/span>());<\/span>\n\t\tif<\/span> (<\/span>noLanes<\/span> ==<\/span> 0<\/span>)<\/span> {<\/span>\n\t\t\taddToNewLane<\/span>(<\/span>e<\/span>,<\/span> start<\/span>,<\/span> end<\/span>);<\/span>\n\t\t}<\/span> else<\/span> if<\/span> (!<\/span>lanes<\/span>[<\/span>0<\/span>].<\/span>accept<\/span>(<\/span>e<\/span>,<\/span> start<\/span>,<\/span> end<\/span>))<\/span> {<\/span>\n\t\t\tint<\/span> changedLane<\/span> =<\/span> addToOtherLane<\/span>(<\/span>e<\/span>,<\/span> start<\/span>,<\/span> end<\/span>);<\/span>\n\t\t\tsortLanes<\/span>(<\/span>changedLane<\/span>);<\/span>\n\t\t}<\/span>\n\t}<\/span>\n\n\tprivate<\/span> void<\/span> sortLanes<\/span>(<\/span>int<\/span> fromIndex<\/span>)<\/span> {<\/span>\n\t\t\/\/ Sorting the lanes by descending end time<\/span>\n\t\tfor<\/span> (<\/span>int<\/span> i<\/span> =<\/span> fromIndex<\/span>;<\/span> i<\/span> ><\/span> 0<\/span>;<\/span> i<\/span>--)<\/span> {<\/span>\n\t\t\tif<\/span> (<\/span>lanes<\/span>[<\/span>i<\/span>].<\/span>end<\/span>.<\/span>compareTo<\/span>(<\/span>lanes<\/span>[<\/span>i<\/span> -<\/span> 1<\/span>].<\/span>end<\/span>)<\/span> ><\/span> 0<\/span>)<\/span> {<\/span>\n\t\t\t\tDisjointArray<\/span><<\/span>T<\/span>><\/span> tmp<\/span> =<\/span> lanes<\/span>[<\/span>i<\/span> -<\/span> 1<\/span>];<\/span>\n\t\t\t\tlanes<\/span>[<\/span>i<\/span> -<\/span> 1<\/span>]<\/span> =<\/span> lanes<\/span>[<\/span>i<\/span>];<\/span>\n\t\t\t\tlanes<\/span>[<\/span>i<\/span>]<\/span> =<\/span> tmp<\/span>;<\/span>\n\t\t\t}<\/span>\n\t\t}<\/span>\n\t}<\/span>\n\n\tprivate<\/span> int<\/span> addToOtherLane<\/span>(<\/span>T<\/span> e<\/span>,<\/span> IQuantity<\/span> start<\/span>,<\/span> IQuantity<\/span> end<\/span>)<\/span> {<\/span>\n\t\t\/\/ Try with the other existing lanes<\/span>\n\t\tfor<\/span> (<\/span>int<\/span> i<\/span> =<\/span> 1<\/span>;<\/span> i<\/span> <<\/span> noLanes<\/span>;<\/span> i<\/span>++)<\/span> {<\/span>\n\t\t\tif<\/span> (<\/span>lanes<\/span>[<\/span>i<\/span>].<\/span>accept<\/span>(<\/span>e<\/span>,<\/span> start<\/span>,<\/span> end<\/span>))<\/span> {<\/span>\n\t\t\t\treturn<\/span> i<\/span>;<\/span>\n\t\t\t}<\/span>\n\t\t}<\/span>\n\t\treturn<\/span> addToNewLane<\/span>(<\/span>e<\/span>,<\/span> start<\/span>,<\/span> end<\/span>);<\/span>\n\t}<\/span>\n\n\tprivate<\/span> int<\/span> addToNewLane<\/span>(<\/span>T<\/span> e<\/span>,<\/span> IQuantity<\/span> start<\/span>,<\/span> IQuantity<\/span> end<\/span>)<\/span> {<\/span>\n\t\tif<\/span> (<\/span>noLanes<\/span> >=<\/span> lanes<\/span>.<\/span>length<\/span>)<\/span> {<\/span>\n\t\t\tlanes<\/span> =<\/span> Arrays<\/span>.<\/span>copyOf<\/span>(<\/span>lanes<\/span>,<\/span> (<\/span>lanes<\/span>.<\/span>length<\/span> *<\/span> 3<\/span>)<\/span> \/<\/span> 2<\/span> +<\/span> 2<\/span>);<\/span>\n\t\t}<\/span>\n\t\tlanes<\/span>[<\/span>noLanes<\/span>]<\/span> =<\/span> new<\/span> DisjointArray<\/span><>(<\/span>e<\/span>,<\/span> start<\/span>,<\/span> end<\/span>);<\/span>\n\t\treturn<\/span> noLanes<\/span>++;<\/span>\n\t}<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
                      lanes[0].accept<\/code> is obviously expected to be the common case (it happens whenever the events are disjoint).\nIts implementation appears to be a constant time operation too, assuming the comparator isn\u2019t bizarre:<\/p>\n\n
                          boolean<\/span> accept<\/span>(<\/span>T<\/span> e<\/span>,<\/span> IQuantity<\/span> start<\/span>,<\/span> IQuantity<\/span> end<\/span>)<\/span> {<\/span>\n        if<\/span> (<\/span>size<\/span> >=<\/span> array<\/span>.<\/span>length<\/span>)<\/span> {<\/span>\n            int<\/span> newCapacity<\/span> =<\/span> array<\/span>.<\/span>length<\/span> <<\/span> 100<\/span> ?<\/span> array<\/span>.<\/span>length<\/span> *<\/span> 4<\/span> :<\/span> (<\/span>array<\/span>.<\/span>length<\/span> *<\/span> 3<\/span>)<\/span> \/<\/span> 2<\/span> +<\/span> 1<\/span>;<\/span>\n            array<\/span> =<\/span> Arrays<\/span>.<\/span>copyOf<\/span>(<\/span>array<\/span>,<\/span> newCapacity<\/span>);<\/span>\n        }<\/span>\n        if<\/span> (<\/span>start<\/span>.<\/span>compareTo<\/span>(<\/span>this<\/span>.<\/span>end<\/span>)<\/span> >=<\/span> 0<\/span>)<\/span> {<\/span>\n            array<\/span>[<\/span>size<\/span>++]<\/span> =<\/span> e<\/span>;<\/span>\n            this<\/span>.<\/span>end<\/span> =<\/span> end<\/span>;<\/span>\n            return<\/span> true<\/span>;<\/span>\n        }<\/span>\n        return<\/span> false<\/span>;<\/span>\n    }<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
                      The method addToOtherLane<\/code> shows up in the flamegraph (outlined in black):<\/p>\n\n
                      \"DisjointBuilder\"<\/p>\n\n
                      So we must be going down the other branch:<\/p>\n\n
                          int<\/span> changedLane<\/span> =<\/span> addToOtherLane<\/span>(<\/span>e<\/span>,<\/span> start<\/span>,<\/span> end<\/span>);<\/span>\n    sortLanes<\/span>(<\/span>changedLane<\/span>);<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
                      addToOtherLane<\/code> does a linear search for the first lane for which the last event ends before the event being added.\nThis can obviously be improved by replacing the linear search with binary search, since the sort order of the lanes by end time is maintained.\nThis doesn\u2019t describe all of the samples though, and it seems strange that so many samples should be in MemberAccessorToolkit$2.getMember<\/code>, and where is sortLanes<\/code>?<\/p>\n\n
                      \"MemberAccessToolkit\"<\/p>\n\n
                      This only makes sense if sortLanes<\/code> (which is a quadratic bubble sort) has been inlined.\nKnowing that the lanes are maintained in sort order, when a lane needs to move, it\u2019s possible to find where it needs to move to in logarithmic time, and then perform a linear time copy.\nThis makes the algorithm linear in the number of lanes.<\/p>\n\n
                      \"Fixed\"<\/p>\n\n
                      Erik Gahlin pointed out on the issue that jdk.FileWrite<\/code> events making profiles slow to load must be because of a bug in JFR.\nIt turned out there has been a bug in JFR since JDK15, introduced here<\/a>.\nThe problematic change is here:<\/p>\n\n
                           @SuppressWarnings<\/span>(<\/span>\"deprecation\"<\/span>)<\/span>\n     @JIInstrumentationMethod<\/span>\n     public<\/span> void<\/span> write<\/span>(<\/span>byte<\/span> b<\/span>[])<\/span> throws<\/span> IOException<\/span> {<\/span>\n-<\/span>        FileWriteEvent<\/span> event<\/span> =<\/span> FileWriteEvent<\/span>.<\/span>EVENT<\/span>.<\/span>get<\/span>();<\/span>\n-<\/span>        if<\/span> (!<\/span>event<\/span>.<\/span>isEnabled<\/span>())<\/span> {<\/span>\n+<\/span>        EventHandler<\/span> handler<\/span> =<\/span> Handlers<\/span>.<\/span>FILE_WRITE<\/span>;<\/span>\n+<\/span>        if<\/span> (!<\/span>handler<\/span>.<\/span>isEnabled<\/span>())<\/span> {<\/span>\n             write<\/span>(<\/span>b<\/span>);<\/span>\n             return<\/span>;<\/span>\n         }<\/span>\n+<\/span>        long<\/span> bytesWritten<\/span> =<\/span> 0<\/span>;<\/span>\n+<\/span>        long<\/span> start<\/span> =<\/span> 0<\/span>;<\/span>\n         try<\/span> {<\/span>\n-<\/span>            event<\/span>.<\/span>begin<\/span>();<\/span>\n+<\/span>            start<\/span> =<\/span> EventHandler<\/span>.<\/span>timestamp<\/span>();<\/span>\n             write<\/span>(<\/span>b<\/span>);<\/span>\n-<\/span>            event<\/span>.<\/span>bytesWritten<\/span> =<\/span> b<\/span>.<\/span>length<\/span>;<\/span>\n+<\/span>            bytesWritten<\/span> =<\/span> b<\/span>.<\/span>length<\/span>;<\/span>\n         }<\/span> finally<\/span> {<\/span>\n-<\/span>            event<\/span>.<\/span>path<\/span> =<\/span> path<\/span>;<\/span>\n-<\/span>            event<\/span>.<\/span>commit<\/span>();<\/span>\n-<\/span>            event<\/span>.<\/span>reset<\/span>();<\/span>\n+<\/span>            long<\/span> duration<\/span> =<\/span> EventHandler<\/span>.<\/span>timestamp<\/span>();<\/span> \/\/ <-- bug here<\/span>\n+<\/span>            if<\/span> (<\/span>handler<\/span>.<\/span>shouldCommit<\/span>(<\/span>duration<\/span>))<\/span> {<\/span>\n+<\/span>                handler<\/span>.<\/span>write<\/span>(<\/span>start<\/span>,<\/span> duration<\/span>,<\/span> path<\/span>,<\/span> bytesWritten<\/span>);<\/span>\n+<\/span>            }<\/span>\n         }<\/span>\n     }<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
                      The duration should have the start time subtracted from it. \nThis means that these events will generally overlap because end = start + duration<\/code> makes a nonsense value. \nThe problem is compounded because the faulty duration will always exceed the latency threshold, which means these events flood JFR files to the extent they stress the parsing code.<\/p>","author":{"name":{}},"category":[{"@attributes":{"term":"java"}},{"@attributes":{"term":"profiling"}}],"summary":"When you open a profile in JMC, it normally takes a few seconds, but there are some profiles that JMC struggles to load. This happens when a profile contains events which violate an assumption made in JMC\u2019s parser: events on the same thread are almost always disjoint in their durations. When JMC parses a JFR file, it splits the events of the same type emitted on the same thread into lanes so that events within a lane are disjoint. JMC can handle overlapping events, but assumes this essentially never happens, and exhibits quadratic scaling when events overlap."},{"title":"Evaluating Equality Predicates","link":{"@attributes":{"href":"https:\/\/richardstartin.github.io\/posts\/range-bitmap-equality-queries","rel":"alternate","type":"text\/html","title":"Evaluating Equality Predicates"}},"published":"2022-12-18T00:00:00+00:00","updated":"2022-12-18T00:00:00+00:00","id":"https:\/\/richardstartin.github.io\/posts\/range-bitmap-equality-queries","content":"
                        \n  
                      1. How to use it?<\/a><\/li>\n  
                      2. How does it work?<\/a><\/li>\n<\/ol>\n\n
                        How to use it?<\/h3>\n\n
                        This post extends the example set up in Evaluating Range Predicates<\/a> and Counting over Range Predicates<\/a>, which centre around a large collection of Transaction<\/code> objects:<\/p>\n\n
                          public<\/span> static<\/span> final<\/span> class<\/span> Transaction<\/span> {<\/span>\n    private<\/span> final<\/span> int<\/span> quantity<\/span>;<\/span>\n    private<\/span> final<\/span> long<\/span> price<\/span>;<\/span>\n    private<\/span> final<\/span> long<\/span> timestamp<\/span>;<\/span>\n\n    public<\/span> Transaction<\/span>(<\/span>int<\/span> quantity<\/span>,<\/span> long<\/span> price<\/span>,<\/span> long<\/span> timestamp<\/span>)<\/span> {<\/span>\n      this<\/span>.<\/span>quantity<\/span> =<\/span> quantity<\/span>;<\/span>\n      this<\/span>.<\/span>price<\/span> =<\/span> price<\/span>;<\/span>\n      this<\/span>.<\/span>timestamp<\/span> =<\/span> timestamp<\/span>;<\/span>\n    }<\/span>\n\n    public<\/span> int<\/span> getQuantity<\/span>()<\/span> {<\/span>\n      return<\/span> quantity<\/span>;<\/span>\n    }<\/span>\n\n    public<\/span> long<\/span> getPrice<\/span>()<\/span> {<\/span>\n      return<\/span> price<\/span>;<\/span>\n    }<\/span>\n\n    public<\/span> long<\/span> getTimestamp<\/span>()<\/span> {<\/span>\n      return<\/span> timestamp<\/span>;<\/span>\n    }<\/span>\n  }<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
                        Let\u2019s find all the transactions with a certain quantity in the most obvious way possible, using the stream API.<\/p>\n\n
                            transactions<\/span>.<\/span>stream<\/span>()<\/span>\n         .<\/span>filter<\/span>(<\/span>transaction<\/span> -><\/span> transaction<\/span>.<\/span>quantity<\/span> ==<\/span> qty<\/span>)<\/span>\n         .<\/span>forEach<\/span>(<\/span>this<\/span>::<\/span>processTransaction<\/span>);<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
                        On my laptop, it takes about 3ms to select about 100 transactions from 1M.<\/p>\n\n
                        \n\n  \n    \n      \n        \n      \n        
                        Benchmark<\/th>\n        Mode<\/th>\n        Threads<\/th>\n        Samples<\/th>\n        Score<\/th>\n        Score Error (99.9%)<\/th>\n        Unit<\/th>\n        Param: minPrice<\/th>\n        Param: minQuantity<\/th>\n        Param: size<\/th>\n      <\/tr>\n    <\/thead>\n    
                        stream<\/td>\n        avgt<\/td>\n        1<\/td>\n        5<\/td>\n        2849.444527<\/td>\n        30.581160<\/td>\n        us\/op<\/td>\n        100<\/td>\n        1<\/td>\n        1000000<\/td>\n      <\/tr>\n    <\/tbody>\n  <\/table>\n\n<\/div>\n\n
                        With a RangeBitmap<\/code>, these same transactions can be found with the already existing between<\/code> method:<\/p>\n\n
                            RoaringBitmap<\/span> matchesQuantity<\/span> =<\/span> qtyIndex<\/span>.<\/span>between<\/span>(<\/span>qty<\/span> -<\/span> minQty<\/span>,<\/span> qty<\/span> -<\/span> minQty<\/span>);<\/span>\n    matchesQuantity<\/span>.<\/span>forEach<\/span>((<\/span>IntConsumer<\/span>)<\/span> i<\/span> -><\/span> processTransaction<\/span>(<\/span>transactions<\/span>.<\/span>get<\/span>(<\/span>i<\/span>)));<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
                        \n  
                        as explained in Evaluating Range Predicates<\/a>, the quantities in the index have been anchored to the smallest value in the population, as a size optimisation.<\/p>\n<\/blockquote>\n\n
                        This performs quite a lot better, selecting the same transactions in under 300us, which is a 10x improvement.<\/p>\n\n
                        \n\n  \n    \n      \n        \n      \n        \n        
                        Benchmark<\/th>\n        Mode<\/th>\n        Threads<\/th>\n        Samples<\/th>\n        Score<\/th>\n        Score Error (99.9%)<\/th>\n        Unit<\/th>\n        Param: minPrice<\/th>\n        Param: minQuantity<\/th>\n        Param: size<\/th>\n      <\/tr>\n    <\/thead>\n    
                        stream<\/td>\n        avgt<\/td>\n        1<\/td>\n        5<\/td>\n        2849.444527<\/td>\n        30.581160<\/td>\n        us\/op<\/td>\n        100<\/td>\n        1<\/td>\n        1000000<\/td>\n      <\/tr>\n      
                        between<\/td>\n        avgt<\/td>\n        1<\/td>\n        5<\/td>\n        296.488277<\/td>\n        0.703309<\/td>\n        us\/op<\/td>\n        100<\/td>\n        1<\/td>\n        1000000<\/td>\n      <\/tr>\n    <\/tbody>\n  <\/table>\n\n<\/div>\n\n
                        This can be rewritten using eq<\/code>:<\/p>\n\n
                            RoaringBitmap<\/span> matchesQuantity<\/span> =<\/span> qtyIndex<\/span>.<\/span>eq<\/span>(<\/span>qty<\/span> -<\/span> minQty<\/span>);<\/span>\n    matchesQuantity<\/span>.<\/span>forEach<\/span>((<\/span>IntConsumer<\/span>)<\/span> i<\/span> -><\/span> processTransaction<\/span>(<\/span>transactions<\/span>.<\/span>get<\/span>(<\/span>i<\/span>)));<\/span>\n<\/code><\/pre><\/div><\/div>\n\n
                        The new eq<\/code> method doesn\u2019t need to do as much work as between<\/code>, which needs to maintain and combine two bitsets during the scan over the RangeBitmap<\/code>, whereas eq<\/code> only needs one.\nThis means we can get a good speedup from eq<\/code> to select the same transactions:<\/p>\n\n
                        \n\n  \n    \n      \n        \n      \n        \n        \n        
                        Benchmark<\/th>\n        Mode<\/th>\n        Threads<\/th>\n        Samples<\/th>\n        Score<\/th>\n        Score Error (99.9%)<\/th>\n        Unit<\/th>\n        Param: minPrice<\/th>\n        Param: minQuantity<\/th>\n        Param: size<\/th>\n      <\/tr>\n    <\/thead>\n    
                        stream<\/td>\n        avgt<\/td>\n        1<\/td>\n        5<\/td>\n        2849.444527<\/td>\n        30.581160<\/td>\n        us\/op<\/td>\n        100<\/td>\n        1<\/td>\n        1000000<\/td>\n      <\/tr>\n      
                        between<\/td>\n        avgt<\/td>\n        1<\/td>\n        5<\/td>\n        296.488277<\/td>\n        0.703309<\/td>\n        us\/op<\/td>\n        100<\/td>\n        1<\/td>\n        1000000<\/td>\n      <\/tr>\n      
                        eq<\/td>\n        avgt<\/td>\n        1<\/td>\n        5<\/td>\n        183.913329<\/td>\n        1.994362<\/td>\n        us\/op<\/td>\n        100<\/td>\n        1<\/td>\n        1000000<\/td>\n      <\/tr>\n    <\/tbody>\n  <\/table>\n\n<\/div>\n\n
                        It\u2019s worth making a comparison with an inverted index over quantity<\/code> (so a bitmap of transaction positions per quantity) now.\nThe inverted index would always win for speed on this query, but can take up a lot more space, and the inverted index would almost always lose for range queries.\nA range encoded inverted index (a mapping from quantity to bitmap of the positions of all transactions with a smaller quantity) will generally beat RangeBitmap<\/code> for speed at the cost of space.\nThis makes a RangeBitmap<\/code> suitable for equality queries eiter on a high cardinality attribute or as a fallback better than scanning when range queries are more common.<\/p>\n\n
                        Inequality filters tend not to be very selective, so benchmarking the evaluation would be dominated by the time to scan the results, but the neq<\/code> method is present for API symmetry.<\/p>\n\n
                        In the same set of changes there are also methods to push a RoaringBitmap<\/code> context down into eq<\/code> and neq<\/code> queries, which behaves like an intersection. \nRather than producing a large bitmap and then intersecting it with a small context bitmap, the context bitmap is used to potentially skip over large sections of the RangeBitmap<\/code>.\nThere are also eqCardinality<\/code> and neqCardinality<\/code> methods, which produce counts rather than bitmaps, as described for range counts<\/a>.<\/p>\n\n
                        How does it work?<\/h3>\n\n
                        RangeBitmap<\/code> has a simple two-dimensional structure.\nThe first dimension is the rows in the order the values were added in.\nThe second dimension is the binary representation of each value added, after range encoding, and there are only as many of these columns as there are significant bits in the largest value added.<\/p>\n\n
                        The layout is essentially the bit-transposition of the values added, striped in bands of $2^{16}$ rows, with a little bit of metadata to help traversals. \nEach $2^{16}$ rows are bucketed into a horizontal slice<\/em> which consists of $64 - clz(max)$ RoaringBitmap<\/code> containers, one for each bit of the inputs, and a mask of size $64 - clz(max)$ bits, indicating the presence of a container. \nIf the $n$th value added to the RangeBitmap<\/code> does not have bit $i$ set, the $i$th container in the $n\/2^{16}$th slice will have no bit set.\nIf all $2^{16}$ values in the slice have bit $i$ unset, the mask will have bit $i$ unset and no container, which is an optimisation to avoid storing empty containers.<\/p>\n\n
                        If the following values (with their binary representation in brackets) are added in sequence<\/p>\n
                        42 (101010)\n24 (011000)\n9  (001001)\n27 (011011)\n<\/code><\/pre><\/div><\/div>\n\n
                        The values are first negated to range encode them (see here<\/a> for explanation).<\/p>\n\n
                        010101\n100111\n110110\n100100\n<\/code><\/pre><\/div><\/div>\n
                        There will be one horizontal slice with four rows in it, a 6 bit mask, and 5 containers per slice because the 4th bit is present in all the values in the slice.\nThis looks something like this, if stored as plain bitsets:<\/p>\n\n
                        110111 1100 0110 1111 1010 0110 \n<\/code><\/pre><\/div><\/div>\n\n
                        There are three kinds of RoaringBitmap<\/code> containers for different densities - sparse, dense, and run length encoded.\nContainers with only four values would always be sparse, for the sake of explanation, assume these are just the first four values in a much larger slice.\nEncoded as containers, this looks like:<\/p>\n\n
                        110111 [4,2] {2,4} [0,4] 1010 {2,4}\n<\/code><\/pre><\/div><\/div>\n\n
                        Where the containers in braces have been represented as arrays of 16 bit values, the ones in square brackets are run length encoded, and the binary numbers are just bitsets.<\/p>\n\n
                        To evaluate queries against RangeBitmap<\/code>, the slices need to be iterated over in ascending order.\nFor each slice, the containers need to be extracted and combined, using a different combination algorithm for each relation.\nThe equality combination algorithm is very simple:<\/p>\n
                      I have just implemented support<\/a> for (in)equality queries against a RangeBitmap<\/code>, a succinct data structure in the RoaringBitmap<\/code> library which supports range queries.\nRangeBitmap<\/code> was designed to support range queries in Apache Pinot (more details here<\/a>) but this enhancement would allow a range index to be used as a fallback for (in)equality queries in case nothing better is available.\nSupporting (in)equality queries allows a RangeBitmap<\/code> to be used as a kind of compact inverted index, trading space for time, capable of supporting high cardinality gracefully.
                      \nSince RangeBitmap<\/code> supports memory mapping from files, I think that it could be used for data engineering beyond Apache Pinot.<\/p>\n\n
                      These changes were implemented in JMC-7950<\/a>, which will be released in as part of JMC 9, and the issue is more or less fixed.\nIf you open the profile generated by the code at the top of the post in JMC 9, it will still be slower to load than you might have hoped, but it will be several times faster.\nUsing JFR to take a profile of JMC loading the same profile after the change produces a much smaller number of samples (JFR also records much less CPU time) and shows different stack traces:<\/p>\n\n