-
Notifications
You must be signed in to change notification settings - Fork 23.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
improve performance for scan command when matching data type #12395
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@judeng hi, let's continue with it? thanks. |
@sundb Thank you, long time no see. I haven't been online in a long time, I'd like fix conflicts in this week. |
I failed to sign the CLA, the page always return 403 error, I will try it later |
@judeng glad to see you again, thanks. |
sundb
approved these changes
Jul 15, 2024
@judeng it's merged, thanks. |
sundb
pushed a commit
to sundb/redis
that referenced
this pull request
Aug 29, 2024
…2395) Move the TYPE filtering to the scan callback so that avoided the `lookupKey` operation. This is the follow-up to redis#12209 . In this thread we introduced two breaking changes: 1. we will not attempt to do lazy expire (delete) a key that was filtered by not matching the TYPE (like we already do for MATCH pattern). 2. when the specified key TYPE filter is an unknown type, server will reply a error immediately instead of doing a full scan that comes back empty handed.
Merged
YaacovHazan
added a commit
that referenced
this pull request
Sep 12, 2024
### New Features in binary distributions - 7 new data structures: JSON, Time series, Bloom filter, Cuckoo filter, Count-min sketch, Top-k, t-digest - Redis scalable query engine (including vector search) ### Potentially breaking changes - #12272 `GETRANGE` returns an empty bulk when the negative end index is out of range - #12395 Optimize `SCAN` command when matching data type ### Bug fixes - #13510 Fix `RM_RdbLoad` to enable AOF after RDB loading is completed - #13489 `ACL CAT` - return module commands - #13476 Fix a race condition in the `cache_memory` of `functionsLibCtx` - #13473 Fix incorrect lag due to trimming stream via `XTRIM` command - #13338 Fix incorrect lag field in `XINFO` when tombstone is after the `last_id` of the consume group - #13470 On `HDEL` of last field - update the global hash field expiration data structure - #13465 Cluster: Pass extensions to node if extension processing is handled by it - #13443 Cluster: Ensure validity of myself when loading cluster config - #13422 Cluster: Fix `CLUSTER SHARDS` command returns empty array ### Modules API - #13509 New API calls: `RM_DefragAllocRaw`, `RM_DefragFreeRaw`, and `RM_RegisterDefragCallbacks` - defrag API to allocate and free raw memory ### Performance and resource utilization improvements - #13503 Avoid overhead of comparison function pointer calls in listpack `lpFind` - #13505 Optimize `STRING` datatype write commands - #13499 Optimize `SMEMBERS` command - #13494 Optimize `GEO*` commands reply - #13490 Optimize `HELLO` command - #13488 Optimize client query buffer - #12395 Optimize `SCAN` command when matching data type - #13529 Optimize `LREM`, `LPOS`, `LINSERT`, and `LINDEX` commands - #13516 Optimize `LRANGE` and other commands that perform several writes to client buffers per call - #13431 Avoid `used_memory` contention when updating from multiple threads ### Other general improvements - #13495 Reply `-LOADING` on replica while flushing the db ### CLI tools - #13411 redis-cli: Fix wrong `dbnum` showed after the client reconnected ### Notes - No backward compatibility for replication or persistence. - Additional distributions, upgrade paths, features, and improvements will be introduced in upcoming pre-releases. - With the GA release of 8.0 we will deprecate Redis Stack.
YaacovHazan
pushed a commit
to YaacovHazan/redis
that referenced
this pull request
Jan 13, 2025
…edis#12395)" Although the commit #7f0a7f0a6 improves the performance of the SCAN command, we can't accept it as we should avoid a breaking change for non-critical bug fixes. This reverts commit 7f0a7f0.
YaacovHazan
pushed a commit
to YaacovHazan/redis
that referenced
this pull request
Jan 13, 2025
…edis#12395)" Although the commit #7f0a7f0a6 improves the performance of the SCAN command, we can't accept it as we should avoid breaking changes for non-critical bug fixes. This reverts commit 7f0a7f0.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Notice: this change was reverted in 8afb72a
we concluded we can't accept it as we should avoid breaking changes for non-critical bug fixes.
Move the TYPE filtering to the scan callback so that avoided the
lookupKey
operation. This is the follow-up to #12209 . In this thread we introduced two breaking changes:Benchmark result:
For different scenarios, we obtained about 48% more performance improvement: