At the moment we are evaluating/comparing the performance/efficiency of storage of different Map implementations in resource constrained environments. Our application generates the records stored into two Maps of ~700K and ~400K each. The keys are short strings, the values are small Java objects with custom (optimal) serializers, manually processing each field.

We are comparing 4 different Map implementations (all latest versions for Java 7):

1. Koloboke HashObjObjMap (updatable)
2. Chronicle Map ChronicleMapBuilder.of(String.class, Record.class).create()
3. MapDB BTreeMap (with packed B-trees)
4. MapDB HTreeMap (more like a regular hash map)

To analyze memory consumption we've instrumented our test with calls to Runtime.getRuntime().[free|total|max]Memory() for heap and sun.misc.SharedSecrets.getJavaNioAccess(). getDirectBufferPool().get[Count|MemoryUsed|TotalCapacity]]() for direct. Additionally we've recorded the Java process's RSS size at the end of each tests.

First, Chronicle Map did the best, beating out even Koloboke in both time to completion (by ~5%) and in the process RSS size (by ~15%). MapDB maps fell somewhere in the middle by RSS size with BTreeMap being much slower (by ~40%, owing to BTree restructurings) and HTreeMap being just slightly slower (~5%) than Koloboke. The heap usage (after forced full GC) looked as expected in all cases.

What seemed unusual is that sun.misc methods for estimating direct memory produced rather strange results. The numbers were essentially the same (negligible) for Koloboke and Chronicle, while for MapDB grew to represent what reasonably seems like the actual size of the data (~60M for BTree, ~90M for HTree to ~200M for Koloboke on the fully shrunk heap with the map and not much else).

After seeing these results I wonder what would be the proper way to estimate actual consumption of direct memory by the Chronicle Map. RSS might not be a good metric as it could get over-inflated by garbage (at least in the case of our producer application - we had ~1.5G, ~1.3G and ~1.4G for Koloboke, Chronicle and MapDBs). I did find that Chronicle uses unsafe.allocateMemory vs. ByteBuffer.allocateDirect, but in any case I'd like to figure out how to sanity-check our memory usage.

Additionally, for mmap'd files is there a way to tell JVM/Linux how much of those should be loaded into RAM? In case of multiple files is it possible to prioritize caching of one file over another?