Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve performance for scan command when matching data type #12395

Merged
merged 2 commits into from
Aug 20, 2024

Conversation

judeng
Copy link
Contributor

@judeng judeng commented Jul 8, 2023

Notice: this change was reverted in 8afb72a
we concluded we can't accept it as we should avoid breaking changes for non-critical bug fixes.

Move the TYPE filtering to the scan callback so that avoided the lookupKey operation. This is the follow-up to #12209 . In this thread we introduced two breaking changes:

  1. we will not attempt to do lazy expire (delete) a key that was filtered by not matching the TYPE (like we already do for MATCH pattern).
  2. when the specified key TYPE filter is an unknown type, server will reply a error immediately instead of doing a full scan that comes back empty handed.

Benchmark result:
For different scenarios, we obtained about 48% more performance improvement:

  • scene 1: All keys are volatile keys
scan type match 50% match 100% unmatched
unstable 2665 2924
pr 3954 7463
imporovement 48.37% 155.23%
  • scene 2: All keys are persistent keys
scan type 50% unmatched 100% unmatched
unstable 3229 4117
pr 5175 10985
imporovement 60.27% 166.82%

@oranagra oranagra added the breaking-change This change can potentially break existing application label Jul 9, 2023
@CLAassistant
Copy link

CLAassistant commented Mar 24, 2024

CLA assistant check
All committers have signed the CLA.

@sundb
Copy link
Collaborator

sundb commented Jun 4, 2024

@judeng hi, let's continue with it? thanks.

@judeng
Copy link
Contributor Author

judeng commented Jun 27, 2024

@sundb Thank you, long time no see. I haven't been online in a long time, I'd like fix conflicts in this week.

@judeng
Copy link
Contributor Author

judeng commented Jun 27, 2024

I failed to sign the CLA, the page always return 403 error, I will try it later

@sundb
Copy link
Collaborator

sundb commented Jun 27, 2024

@judeng glad to see you again, thanks.

@sundb
Copy link
Collaborator

sundb commented Jul 16, 2024

@sundb sundb merged commit 7f0a7f0 into redis:unstable Aug 20, 2024
13 checks passed
@sundb
Copy link
Collaborator

sundb commented Aug 20, 2024

@judeng it's merged, thanks.

sundb pushed a commit to sundb/redis that referenced this pull request Aug 29, 2024
…2395)

Move the TYPE filtering to the scan callback so that avoided the
`lookupKey` operation. This is the follow-up to redis#12209 . In this thread
we introduced two breaking changes:
1. we will not attempt to do lazy expire (delete) a key that was
filtered by not matching the TYPE (like we already do for MATCH
pattern).
2. when the specified key TYPE filter is an unknown type, server will
reply a error immediately instead of doing a full scan that comes back
empty handed.
@YaacovHazan YaacovHazan mentioned this pull request Sep 11, 2024
YaacovHazan added a commit that referenced this pull request Sep 12, 2024
### New Features in binary distributions

- 7 new data structures: JSON, Time series, Bloom filter, Cuckoo filter,
Count-min sketch, Top-k, t-digest
- Redis scalable query engine (including vector search)

### Potentially breaking changes

- #12272 `GETRANGE` returns an empty bulk when the negative end index is
out of range
- #12395 Optimize `SCAN` command when matching data type

### Bug fixes

- #13510 Fix `RM_RdbLoad` to enable AOF after RDB loading is completed
- #13489 `ACL CAT` - return module commands
- #13476 Fix a race condition in the `cache_memory` of `functionsLibCtx`
- #13473 Fix incorrect lag due to trimming stream via `XTRIM` command
- #13338 Fix incorrect lag field in `XINFO` when tombstone is after the
`last_id` of the consume group
- #13470 On `HDEL` of last field - update the global hash field
expiration data structure
- #13465 Cluster: Pass extensions to node if extension processing is
handled by it
- #13443 Cluster: Ensure validity of myself when loading cluster config
- #13422 Cluster: Fix `CLUSTER SHARDS` command returns empty array

### Modules API

- #13509 New API calls: `RM_DefragAllocRaw`, `RM_DefragFreeRaw`, and
`RM_RegisterDefragCallbacks` - defrag API to allocate and free raw
memory

### Performance and resource utilization improvements

- #13503 Avoid overhead of comparison function pointer calls in listpack
`lpFind`
- #13505 Optimize `STRING` datatype write commands
- #13499 Optimize `SMEMBERS` command
- #13494 Optimize `GEO*` commands reply
- #13490 Optimize `HELLO` command
- #13488 Optimize client query buffer
- #12395 Optimize `SCAN` command when matching data type
- #13529 Optimize `LREM`, `LPOS`, `LINSERT`, and `LINDEX` commands
- #13516 Optimize `LRANGE` and other commands that perform several
writes to client buffers per call
- #13431 Avoid `used_memory` contention when updating from multiple
threads

### Other general improvements

- #13495 Reply `-LOADING` on replica while flushing the db

### CLI tools

- #13411 redis-cli: Fix wrong `dbnum` showed after the client
reconnected

### Notes

- No backward compatibility for replication or persistence.
- Additional distributions, upgrade paths, features, and improvements
will be introduced in upcoming pre-releases.
- With the GA release of 8.0 we will deprecate Redis Stack.
YaacovHazan pushed a commit to YaacovHazan/redis that referenced this pull request Jan 13, 2025
…edis#12395)"

Although the commit #7f0a7f0a6 improves the performance of the SCAN command,
we can't accept it as we should avoid a breaking change for non-critical bug fixes.

This reverts commit 7f0a7f0.
YaacovHazan pushed a commit to YaacovHazan/redis that referenced this pull request Jan 13, 2025
…edis#12395)"

Although the commit #7f0a7f0a6 improves the performance of the SCAN command,
we can't accept it as we should avoid breaking changes for non-critical bug fixes.

This reverts commit 7f0a7f0.
YaacovHazan pushed a commit that referenced this pull request Feb 5, 2025
…12395)"

Although the commit #7f0a7f0a6 improves the performance of the SCAN command,
we can't accept it as we should avoid breaking changes for non-critical bug fixes.

This reverts commit 7f0a7f0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change This change can potentially break existing application
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants