Fix crash in LSET command when replacing small list items with larger ones, creating listpacks larger than 4GB#12955
Merged
oranagra merged 5 commits intoredis:unstablefrom Feb 6, 2024
Merged
Conversation
enjoy-binbin
approved these changes
Jan 16, 2024
Contributor
enjoy-binbin
left a comment
There was a problem hiding this comment.
did not CR, but great work, thanks for finding this
Collaborator
Author
|
@enjoy-binbin in fact it was found by @imchuncai. |
Collaborator
Author
|
I'd actually prefer to continue ##12659, which is a better solution to the problem of over-sized nodes due to LSET commands. |
oranagra
approved these changes
Feb 6, 2024
Member
oranagra
left a comment
There was a problem hiding this comment.
LGTM.
just to confirm, this pile of new code in quicklistReplaceEntry is reachable with other tests we have (ones not using large-memory).
i.e. if we had some off by one bug inserting the new entry in the wrong index, would it be detected?
Collaborator
Author
|
@oranagra yes, current tests without |
sundb
added a commit
to sundb/redis
that referenced
this pull request
Feb 7, 2024
sundb
added a commit
to sundb/redis
that referenced
this pull request
Feb 8, 2024
oranagra
pushed a commit
that referenced
this pull request
Feb 8, 2024
Fix two crash introducted by #12955 When a quicklist node can't be inserted and split, we eventually merge the current node with its neighboring nodes after inserting, and compress the current node and its siblings. 1. When the current node is merged with another node, the current node may become invalid and can no longer be used. Solution: let `_quicklistMergeNodes()` return the merged nodes. 3. If the current node is a LZF quicklist node, its recompress will be 1. If the split node can be merged with a sibling node to become head or tail, recompress may cause the head and tail to be compressed, which is not allowed. Solution: always recompress to 0 after merging.
roggervalf
pushed a commit
to roggervalf/redis
that referenced
this pull request
Feb 11, 2024
… 4GB (redis#12955) Fix redis#12864 The main reason for this crash is that when replacing a element of a quicklist packed node with lpReplace() method, if the final size is larger than 4GB, lpReplace() will fail and returns NULL, causing `node->entry` to be incorrectly set to NULL. Since the inserted data is not a large element, we can't just replace it like a large element, first quicklistInsertAfter() and then quicklistDelIndex(), because the current node may be merged and invalidated in quicklistInsertAfter(). The solution of this PR: When replacing a node fails (listpack exceeds 4GB), split the current node, create a new node to put in the middle, and try to merge them. This is the same as inserting a large element. In the worst case, its size will not exceed 4GB.
roggervalf
pushed a commit
to roggervalf/redis
that referenced
this pull request
Feb 11, 2024
…edis#13040) Fix two crash introducted by redis#12955 When a quicklist node can't be inserted and split, we eventually merge the current node with its neighboring nodes after inserting, and compress the current node and its siblings. 1. When the current node is merged with another node, the current node may become invalid and can no longer be used. Solution: let `_quicklistMergeNodes()` return the merged nodes. 3. If the current node is a LZF quicklist node, its recompress will be 1. If the split node can be merged with a sibling node to become head or tail, recompress may cause the head and tail to be compressed, which is not allowed. Solution: always recompress to 0 after merging.
YaacovHazan
pushed a commit
to YaacovHazan/redis
that referenced
this pull request
May 16, 2024
… 4GB (redis#12955) Fix redis#12864 The main reason for this crash is that when replacing a element of a quicklist packed node with lpReplace() method, if the final size is larger than 4GB, lpReplace() will fail and returns NULL, causing `node->entry` to be incorrectly set to NULL. Since the inserted data is not a large element, we can't just replace it like a large element, first quicklistInsertAfter() and then quicklistDelIndex(), because the current node may be merged and invalidated in quicklistInsertAfter(). The solution of this PR: When replacing a node fails (listpack exceeds 4GB), split the current node, create a new node to put in the middle, and try to merge them. This is the same as inserting a large element. In the worst case, its size will not exceed 4GB. (cherry picked from commit 1f00c95)
YaacovHazan
pushed a commit
to YaacovHazan/redis
that referenced
this pull request
May 16, 2024
…edis#13040) Fix two crash introducted by redis#12955 When a quicklist node can't be inserted and split, we eventually merge the current node with its neighboring nodes after inserting, and compress the current node and its siblings. 1. When the current node is merged with another node, the current node may become invalid and can no longer be used. Solution: let `_quicklistMergeNodes()` return the merged nodes. 3. If the current node is a LZF quicklist node, its recompress will be 1. If the split node can be merged with a sibling node to become head or tail, recompress may cause the head and tail to be compressed, which is not allowed. Solution: always recompress to 0 after merging. (cherry picked from commit 1e8dc1d)
Merged
YaacovHazan
pushed a commit
that referenced
this pull request
May 19, 2024
… 4GB (#12955) Fix #12864 The main reason for this crash is that when replacing a element of a quicklist packed node with lpReplace() method, if the final size is larger than 4GB, lpReplace() will fail and returns NULL, causing `node->entry` to be incorrectly set to NULL. Since the inserted data is not a large element, we can't just replace it like a large element, first quicklistInsertAfter() and then quicklistDelIndex(), because the current node may be merged and invalidated in quicklistInsertAfter(). The solution of this PR: When replacing a node fails (listpack exceeds 4GB), split the current node, create a new node to put in the middle, and try to merge them. This is the same as inserting a large element. In the worst case, its size will not exceed 4GB. (cherry picked from commit 1f00c95)
YaacovHazan
pushed a commit
that referenced
this pull request
May 19, 2024
Fix two crash introducted by #12955 When a quicklist node can't be inserted and split, we eventually merge the current node with its neighboring nodes after inserting, and compress the current node and its siblings. 1. When the current node is merged with another node, the current node may become invalid and can no longer be used. Solution: let `_quicklistMergeNodes()` return the merged nodes. 3. If the current node is a LZF quicklist node, its recompress will be 1. If the split node can be merged with a sibling node to become head or tail, recompress may cause the head and tail to be compressed, which is not allowed. Solution: always recompress to 0 after merging. (cherry picked from commit 1e8dc1d)
funny-dog
pushed a commit
to funny-dog/redis
that referenced
this pull request
Sep 17, 2025
… 4GB (redis#12955) Fix redis#12864 The main reason for this crash is that when replacing a element of a quicklist packed node with lpReplace() method, if the final size is larger than 4GB, lpReplace() will fail and returns NULL, causing `node->entry` to be incorrectly set to NULL. Since the inserted data is not a large element, we can't just replace it like a large element, first quicklistInsertAfter() and then quicklistDelIndex(), because the current node may be merged and invalidated in quicklistInsertAfter(). The solution of this PR: When replacing a node fails (listpack exceeds 4GB), split the current node, create a new node to put in the middle, and try to merge them. This is the same as inserting a large element. In the worst case, its size will not exceed 4GB.
funny-dog
pushed a commit
to funny-dog/redis
that referenced
this pull request
Sep 17, 2025
…edis#13040) Fix two crash introducted by redis#12955 When a quicklist node can't be inserted and split, we eventually merge the current node with its neighboring nodes after inserting, and compress the current node and its siblings. 1. When the current node is merged with another node, the current node may become invalid and can no longer be used. Solution: let `_quicklistMergeNodes()` return the merged nodes. 3. If the current node is a LZF quicklist node, its recompress will be 1. If the split node can be merged with a sibling node to become head or tail, recompress may cause the head and tail to be compressed, which is not allowed. Solution: always recompress to 0 after merging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #12864
The main reason for this crash is that when replacing a element of a quicklist packed node with lpReplace() method,
if the final size is larger than 4GB, lpReplace() will fail and returns NULL, causing
node->entryto be incorrectly set to NULL.Since the inserted data is not a large element, we can't just replace it like a large element, first quicklistInsertAfter()
and then quicklistDelIndex(), because the current node may be merged and invalidated in quicklistInsertAfter().
The solution of this PR:
When replacing a node fails (listpack exceeds 4GB), split the current node, create a new node to put in the middle, and try to merge them.
This is the same as inserting a large element.
In the worst case, its size will not exceed 4GB.