Add lazyfree-lazy-user-flush config to control default behavior of FLUSH[ALL|DB], SCRIPT FLUSH#8258
Conversation
src/lazyfree.c
Outdated
There was a problem hiding this comment.
i don't think this condition is sufficient.
in all other places a lazy free is dependent on a user input or a config, and this threshold just defines if it worth while to send the job to the other thread or keep it in the main thread just because the list is short.
i.e the user first needs to decide if he's willing to accept the async freeing side effects (not atomic), or not.
so to be consistent with that, we should either add a check for lazyfree_lazy_server_del or add an ASYNC argument for SCRIPT FLUSH
There was a problem hiding this comment.
Agree with you, I think both of your suggestions should be taken:
- Provide
SCRIPT FLUSH ASYNCfor users to actively trigger. SCRIPT FLUSHjudges whether to release asynchronously according tolazyfree_lazy_server_del.
There was a problem hiding this comment.
on a second thought, i'm actually not sure lazyfree-lazy-server-del is the right config, maybe it should be lazyfree-lazy-user-del or neither.
looking at what FLUSHB does, it should be neither.
There was a problem hiding this comment.
I don't think we should use lazyfree-lazy-server-del or lazyfree-lazy-user-del, since they are already defined and don't mean this.
I also don't think we should be adding ASYNC into command arguments. It's not really something you should be deciding on a case by case basis, and it adds overhead to client developers to support and us to document.
I think we should introduce a general new flag for lazyfree-policy, that covers every other lazy free usage not covered by an existing flag. I think it would even be nice to update the default value for the other lazy-free configs to add a third option, use-default, which would use the lazyfree-policy value unless an explicit value was set.
There was a problem hiding this comment.
I think we should introduce a general new flag for lazyfree-policy, that covers every other lazy free usage not covered by an existing flag.
I agree with your suggestion of adding a new flag, for example: lazyfree-lazy-server-free. However, it is more difficult for us to find the remaining code that is not covered by lazyfree. The problem of scriptingRelease is encountered in our production environment, but there are many places where dictRelease is called, see [here](https:// github.com/redis/redis/search?q=dictRelease) . In addition to dictRelease, there are listRelease, raxFree, etc.
I think it would even be nice to update the default value for the other lazy-free configs to add a third option, use-default, which would use the lazyfree-policy value unless an explicit value was set.
Here I have a little doubt, for example: you mean that lazyfree-lazy-eviction will have three optional values: yes,no,use-default? If use-default is set, then the rest of the lazyfree configuration is this value?
There was a problem hiding this comment.
Hi guys. I have some questions:
freeLuaScriptsAsyncThis function name metions the behavior will be a async way. But we still calculate the dict size. And if dictsize not reachLAZYFREE_THRESHOLD 64. It will become a sync way. In other ways,scriptingReleaserequire aasyncarg to callfreeLuaScriptsAsync. But insidefreeLuaScriptsAsync, it could be a sync way- so do
freeObjAsyncfunction - and also the comment of
freeTrackingRadixTreeAsyncseems wrong
There is not problem with those questions. Just my thinking
There was a problem hiding this comment.
@enjoy-binbin Yes, LAZYFREE_THRESHOLD controls whether it is really async free.
About freeTrackingRadixTreeAsync comment, it's wrong, it will direct async free and do not judge object's size.
/* Free an object, if the object is huge enough, free it in async way. */
void freeTrackingRadixTreeAsync(rax *tracking) {
atomicIncr(lazyfree_objects,tracking->numele);
bioCreateLazyFreeJob(lazyFreeTrackingTable,1,tracking);
}
and another problem is we should use tracking->numnodes(Every macro node in the Stream is one allocation) instead of tracking->numele.
Now we have two points to fix:
1, use tracking->numnodes instead of tracking->numele
2. Judge size to async free or fix comments.
@oranagra @madolson WDYT?
There was a problem hiding this comment.
Received a reply so soon. cool. I am willing to correct it if needed
And the mean time. Do we need to be consistent before and after method behavior?
There was a problem hiding this comment.
freeXxxxAsync doesn't necessarily means it'll be released by the thread.
it means it will be potentially async (which has implication on memory being reclaimed in time or not), but it doesn't mean it'll always be done by the thread.
even the UNLINK function doesn't always release asynchronously, it considers that and makes a efficiency decision (it will be inefficient to do it async if you just need to release a few pointers).
so i don't see a problem about the initial complain on freeLuaScriptsAsync, or maybe i didn't understand it.
i agree that freeTrackingRadixTreeAsync and also freeSlotsToKeysMapAsync need to be extended with a check for LAZYFREE_THRESHOLD (against numnodes, not numele).
please note that not every rax structure that we release this way needs to look at numnodes. if the rax was holding any points in it's elements that need to be released too, then we should look at numele or nuele+numnodes, but in this case but these rax structures are holding only keys (no values), so it only needs to release the nodes.
regarding lazyfree_objects, it is a statistics metric, not a real counter of released allocations. it seems to denote the number of objects / elements released, not actual allocations.
However, when we release a list object (composed of a linked list of ziplists), we increment it only by one (not the number of elements in the list, and not the number of linked-list nodes), so for from that point of view, we can increment it only by one here too. (if we do that, we need to update the call to atomicDecr too).
Again, just a metric, and i don't have any major argument for incrementing by either [1, numele, numnodes].
There was a problem hiding this comment.
1e8ebea to
55452bf
Compare
afb7435 to
1b30052
Compare
6654e7f to
99a4ed3
Compare
…LL|DB], SCRIPT FLUSH commands are lazyfree the current behavior is as follows: FLUSH[ALL|DB],SCRIPT FLUSH: Determine sync or async according to the value of lazyfree-lazy-user-flush. FLUSH[ALL|DB],SCRIPT FLUSH ASYNC: Always flushes the database in an async manner. FLUSH[ALL|DB],SCRIPT FLUSH SYNC: Always flushes the database in a sync manner.
99a4ed3 to
66210fa
Compare
Co-authored-by: Oran Agra <[email protected]>
1fbd6ec
|
@redis/core-team please approve or comment. Lets split the lazy configs to 3 groups:
|
Co-authored-by: Itamar Haber <[email protected]>
|
@yangbodong22011 Can you create PR for the doc update? I think that is the only thing blocking the merge now. |
|
docs PR: redis/redis-doc#1493 |
…USH[ALL|DB], SCRIPT FLUSH (redis#8258) * Adds ASYNC and SYNC arguments to SCRIPT FLUSH * Adds SYNC argument to FLUSHDB and FLUSHALL * Adds new config to control the default behavior of FLUSHDB, FLUSHALL and SCRIPT FLUASH. the new behavior is as follows: * FLUSH[ALL|DB],SCRIPT FLUSH: Determine sync or async according to the value of lazyfree-lazy-user-flush. * FLUSH[ALL|DB],SCRIPT FLUSH ASYNC: Always flushes the database in an async manner. * FLUSH[ALL|DB],SCRIPT FLUSH SYNC: Always flushes the database in a sync manner.
the new behavior is as follows:
value of lazyfree-lazy-user-flush.