Skip to content

Fix some wrong server.dirty increments#8140

Merged
oranagra merged 10 commits intoredis:unstablefrom
sundb:fix_spop_add_dirty
Dec 15, 2020
Merged

Fix some wrong server.dirty increments#8140
oranagra merged 10 commits intoredis:unstablefrom
sundb:fix_spop_add_dirty

Conversation

@sundb
Copy link
Collaborator

@sundb sundb commented Dec 6, 2020

  1. Fix wrong server.drity increment in spopWithCountCommand. If the count of spop is larger than the size of set, server.dirty should only increase the count of set, not the count.
  2. Fix wrong server.dirty increment in hsetCommand, pfaddCommand, ltrimCommand.

oranagra
oranagra previously approved these changes Dec 6, 2020
Copy link
Member

@oranagra oranagra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sundb do you see any reason not to delete the two other increments to server.dirty in this function?

@oranagra oranagra added the state:to-be-merged The PR should be merged soon, even if not yet ready, this is used so that it won't be forgotten label Dec 6, 2020
@sundb sundb changed the title Fix wrong server dirty addition in spop Fix wrong server dirty increments in spop Dec 7, 2020
@sundb
Copy link
Collaborator Author

sundb commented Dec 7, 2020

@sundb do you see any reason not to delete the two other increments to server.dirty in this function?

Ths, These two increments really seem unnecessary.

@sundb sundb changed the title Fix wrong server dirty increments in spop Fix wrong server.dirty increments in spop and remove redundancy increment Dec 7, 2020
@sundb sundb changed the title Fix wrong server.dirty increments in spop and remove redundancy increment Fix some wrong server.dirty increments Dec 7, 2020
@oranagra
Copy link
Member

oranagra commented Dec 7, 2020

@itamarhaber @guybe7 please confirm that i'm not missing anything.
not sure why in some places we used to increment the dirty with the amount of changed fields, and in others just increment once for the changed key.

oranagra
oranagra previously approved these changes Dec 7, 2020
@sundb
Copy link
Collaborator Author

sundb commented Dec 7, 2020

@oranagra
I've organized some case:

  1. add, update, delete N times with or without dbDelete, server.dirty += N
  2. dbDelete only, no add, update, delete, server.dirty += 1
  3. dbAdd and insert, server.dirty += 1

@itamarhaber
Copy link
Member

Sorry, I can't confirm although it looks ok. I believe, in the sense that's my current understanding, that as long as dirty is larger than the number of keys modified, it should be ok... but belief isn't something you build software with :)

@oranagra
Copy link
Member

@sundb one last thing before i merge this, did you go over all the commands that change multiple elements, and these were the only two that needed such a fix?

@sundb
Copy link
Collaborator Author

sundb commented Dec 10, 2020

@sundb one last thing before i merge this, did you go over all the commands that change multiple elements, and these were the only two that needed such a fix?

These days in the hospital, and so home I went through all the codes.

@itamarhaber
Copy link
Member

@sundb I hope all is well with you and yours.

@sundb
Copy link
Collaborator Author

sundb commented Dec 13, 2020

@itamarhaber Thanks.

@sundb
Copy link
Collaborator Author

sundb commented Dec 15, 2020

@oranagra I have gone through all the code that involves server.dirty modification, I think there should be ok.

@oranagra oranagra merged commit 7993780 into redis:unstable Dec 15, 2020
@sundb sundb deleted the fix_spop_add_dirty branch December 16, 2020 07:46
linxiang-dev pushed a commit to linxiang-dev/redis that referenced this pull request Dec 27, 2020
Fix wrong server dirty increment in
* spopWithCountCommand
* hsetCommand
* ltrimCommand
* pfaddCommand

Some didn't increment the amount of fields (just one per command).
Others had excessive increments.
JackieXie168 pushed a commit to JackieXie168/redis that referenced this pull request Mar 2, 2021
Fix wrong server dirty increment in
* spopWithCountCommand
* hsetCommand
* ltrimCommand
* pfaddCommand

Some didn't increment the amount of fields (just one per command).
Others had excessive increments.
@enjoy-binbin
Copy link
Contributor

I think it might be better to unify, in the future.
There is a case i can tell.

We use list as a MQ in some simple scenarios. Like:

pipeline.lrange('log:stat:xxxx_log', 0, args.chunk - 1)
pipeline.ltrim('log:stat:xxxx_log', args.chunk, -1)

The changes in server.dirty += (ltrim + rtrim);
Will cause the trigger of bgsave to not meet my expectations.
(Will be easily triggered by server_cron)

Because I always thought that changes in save are about keys.... in redis.conf we can see:

# Unless specified otherwise, by default Redis will save the DB:
#   * After 3600 seconds (an hour) if at least 1 key changed
#   * After 300 seconds (5 minutes) if at least 100 keys changed
#   * After 60 seconds if at least 10000 keys changed
#
# You can set these explicitly by uncommenting the three following lines.
#
# save 3600 1
# save 300 100
# save 60 10000

Although this seems reasonable, the list has indeed changed so many times.
And We may also want to persist for this situation

@oranagra
Copy link
Member

I'm not 100% certain what's the right thing and what was the intention.
I think it is likely that the documentation got outdated (left there from a time there were no complex data types).
on the other hand, some of the pre-existing lines like server.dirty += count in 9feee42 are very explicit.
that's why we assumed that the other places that incremented just once where simply overlooked.

@enjoy-binbin
Copy link
Contributor

# Unless specified otherwise, by default Redis will save the DB:
#   * After 3600 seconds (an hour) if at least 1 key changed
#   * After 300 seconds (5 minutes) if at least 100 keys changed
#   * After 60 seconds if at least 10000 keys changed

i want to fix the outdated doc (about the key word)
I thought about using elements, but it is also not right, because in FLUSHDB, server.dirty += dictSize(db)
Not to mention ++ in some other places. so maybe change it to at least 1 change from the last save

and i suppose we can also mention the rdb_changes_since_last_save (user can see it in INFO command).

long long dirty; /* Changes to DB from the last save */

@oranagra
Copy link
Member

i agree, the word "key" is outdated.
maybe "if at least 1 change was performed"?

enjoy-binbin added a commit to enjoy-binbin/redis that referenced this pull request Jan 24, 2022
For some complex data types, server.dirty actually counts
the number of elements that have been changed.
And in FLUSHDB or FLUSHALL, we count the number of keys.

So the word "key" is not strictly correct and is outdated.
Some discussion can be seen at redis#8140.
oranagra pushed a commit that referenced this pull request Jan 24, 2022
For some complex data types, server.dirty actually counts
the number of elements that have been changed.
And in FLUSHDB or FLUSHALL, we count the number of keys.

So the word "key" is not strictly correct and is outdated.
Some discussion can be seen at #8140.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:to-be-merged The PR should be merged soon, even if not yet ready, this is used so that it won't be forgotten

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants