For off heap buffers (direct or mmap'd), we currently use Lbuffer library if the size is more than 2GB (which is supported for raw forward index by the v3 and v4 writer).

We have seen production issues on at least 2 occasions where a race condition in the library (map followed by unmap and then map) can lead to SIGSEGV where the address the code is trying to access is 3GB beyond the total virtual address space mapped for Pinot segments on the host where crash was observed.

Not easily reproducible but has happened twice in production.

Also, the library is no longer being maintained by the original author so may be there is benefit in writing our own allocator for large buffers.

TODO - add a link to the race condition issue and add more details from the production issues we observed.