It is possible to have a leveled fold that can be passed a list of buckets. In this case, a snapshot of the penciller will be taken which has an unfiltered L0 cache. There will then be multiple folds requested in series from that snapshot, each one filtering the L0 cache.
To be clear, the penciller is a PID (leveled_penciller gen_server) which has a L0 cache in its loop state. The snapshot is a new PID (leveled_penciller gen_server) which is initiated with the same L0 cache as in the original PID. The list bucket process then sends multiple messages to the new (snapshot) PID, each one will require an extract to be taken from the L0 cache.
There is an intriguing problem here introduced in OTP 22.
The L0 cache can be large - say 30K+ objects (although each object is small). When the cache is larger, each query against the snapshot takes longer to be set-up. This time isn't being spent inside the actual handle_call, it is being spent sending the gen_server:call/3 and receiving the response (a fold function to use).
With a cache of 30K objects, in OTP 22, this time to setup each fold can be 80-100ms. However, running the same code in OTP 21, the time to setup each fold is around 1ms - so OTP 22 is 100 x slower for this operation.
This delta is concerning. The only placed where it has so far been observed is in kv_index_tictactree with the ct tests:
mockvnode_SUITE/loadexchangeandrebuild_stbucketso and mockvnode_SUITE/loadexchangeandrebuild_tuplebucketso
These tests use the segment-ordered backend, which may query a single snapshot > 1K times (especially is small trees are used for the exchange).
This can be mitigated (use large trees, and smaller penciller cache sizes for leveled_so backends). However, what if there is a general problem with the performance of processes which have a large amount of data in the loop state? Is there a general need to reconsider optimisation for leveled in OTP 22 +?
It is possible to have a leveled fold that can be passed a list of buckets. In this case, a snapshot of the penciller will be taken which has an unfiltered L0 cache. There will then be multiple folds requested in series from that snapshot, each one filtering the L0 cache.
To be clear, the penciller is a PID (leveled_penciller gen_server) which has a L0 cache in its loop state. The snapshot is a new PID (leveled_penciller gen_server) which is initiated with the same L0 cache as in the original PID. The list bucket process then sends multiple messages to the new (snapshot) PID, each one will require an extract to be taken from the L0 cache.
There is an intriguing problem here introduced in OTP 22.
The L0 cache can be large - say 30K+ objects (although each object is small). When the cache is larger, each query against the snapshot takes longer to be set-up. This time isn't being spent inside the actual handle_call, it is being spent sending the
gen_server:call/3and receiving the response (a fold function to use).With a cache of 30K objects, in OTP 22, this time to setup each fold can be 80-100ms. However, running the same code in OTP 21, the time to setup each fold is around 1ms - so OTP 22 is 100 x slower for this operation.
This delta is concerning. The only placed where it has so far been observed is in kv_index_tictactree with the ct tests:
mockvnode_SUITE/loadexchangeandrebuild_stbucketso and mockvnode_SUITE/loadexchangeandrebuild_tuplebucketso
These tests use the segment-ordered backend, which may query a single snapshot > 1K times (especially is
smalltrees are used for the exchange).This can be mitigated (use large trees, and smaller penciller cache sizes for leveled_so backends). However, what if there is a general problem with the performance of processes which have a large amount of data in the loop state? Is there a general need to reconsider optimisation for leveled in OTP 22 +?