Skip to content

EXISTS should not alter LRU, OBJECT should not reveal expired keys#8016

Merged
oranagra merged 2 commits intoredis:unstablefrom
guybe7:exists_idletime
Nov 18, 2020
Merged

EXISTS should not alter LRU, OBJECT should not reveal expired keys#8016
oranagra merged 2 commits intoredis:unstablefrom
guybe7:exists_idletime

Conversation

@guybe7
Copy link
Collaborator

@guybe7 guybe7 commented Nov 4, 2020

Bug was intoduced by #5021.

Before that commit, dbExists was used instead of
lookupKeyRead (which affects access time)

Bug was intoduced by redis#5021.

Before that commit, dbExists was used instead of
lookupKeyRead (which affects access time)
@guybe7
Copy link
Collaborator Author

guybe7 commented Nov 4, 2020

@soloestoy FYI

oranagra
oranagra previously approved these changes Nov 4, 2020
@oranagra oranagra added the state:to-be-merged The PR should be merged soon, even if not yet ready, this is used so that it won't be forgotten label Nov 4, 2020
@guybe7
Copy link
Collaborator Author

guybe7 commented Nov 4, 2020

speaking of which, maybe other commands should not affect access time? OBJECT?
@redis/core-team

@itamarhaber
Copy link
Member

I don't object to cleaning the stables and locating more commands that violate this principle, but unless introduced by accident like in this case, we have to consider the (skewed) use case of someone actually doing OBJECT to touch a key. For these we may want to wait for 6.2.

@soloestoy
Copy link
Contributor

Have a question about LOOKUP_NOTOUCH flag usage, I just grep it and found all commands now use it: TTL/SCAN/DEBUG and EXISTS and maybe module open a key with REDISMODULE_OPEN_KEY_NOTOUCH.

I understand the DEBUG command should not touch access time, but others seem like normal commands and users use them not for debugging, so I'm confused when to use LOOKUP_NOTOUCH flag.

@guybe7
Copy link
Collaborator Author

guybe7 commented Nov 5, 2020

@soloestoy yes, that's a grey area in Redis... Maybe it's time to formalize it: I suggest that any command that doesn't access the data of a key should not affect LRU/LFU. Except TOUCH.

@oranagra
Copy link
Member

oranagra commented Nov 5, 2020

I think SCAN (checking that the key didn't expire) and EXIST are kinda obvious that they should not touch the LRU.
OBJECT and DEBUG are introspection commands, so they shouldn't touch the LRU either.

Note that OBJECT uses dictFind right now, but i think it better use lookupKey(LOOKUP_NOTOUCH) (if they key already expired it should not be handled).

I think DEBUG DIGEST-VALUE should be changed to be like DEBUG OBJECT, and use dictFind directly.

Not sure about TTL and EXPIRE (which i suppose should behave similarly in that respect), maybe TTL should be changed to touch the LRU.

@oranagra oranagra added state:major-decision Requires core team consensus release-notes indication that this issue needs to be mentioned in the release notes labels Nov 5, 2020
@soloestoy
Copy link
Contributor

BTW, besides the access time, if key doesn't exist lookupKeyReadWithFlags would call notifyKeyspaceEvent to send keymiss message, so if we take a command as introspection, I think we should not use lookupKeyReadWithFlags.

@oranagra
Copy link
Member

oranagra commented Nov 5, 2020

So we can add a flag named LOOKUP_NONOTIFY and use it in OBJECT (in addition to the changes i listed in my previous message).

We can also add a LOOKUP_NOEXPIRE and then we can let DEBUG command use lookupKey (with all 3 flags set), but i'm not sure what's the point in that.

madolson
madolson previously approved these changes Nov 9, 2020
Copy link
Contributor

@madolson madolson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

itamarhaber
itamarhaber previously approved these changes Nov 11, 2020
@guybe7 guybe7 dismissed stale reviews from itamarhaber, madolson, and oranagra via 6d2bcf2 November 11, 2020 12:09
@guybe7
Copy link
Collaborator Author

guybe7 commented Nov 11, 2020

@redis/core-team pushed another commit fixing DEBUG digest-value (to handle expired keys) and OBJECT (to avoid handling expired keys)

the only thing left to decide is whether EXPIRE command(s) affect LRU/LFU
my gut says "yes" but i don't have any justification.. maybe except that some users can actually be affected if we decide that EXPIRE doesn't change access time

@itamarhaber
Copy link
Member

I agree with @guybe7's gut

@oranagra
Copy link
Member

[deleted my last message since it stated incorrect facts, here's the updated one]

i tend to agree.. (keep EXPIRE as it is, which is to "thouch" the LRU/LFU).
let's try to group all commands that access keys into 4 categories

  1. commands reading from the key data
  2. commands writing to the key data
  3. commands reading from the key metadata (EXISTS, TYPE, TTL, OBJECT ENCODING, OBJECT IDLETIME) - note that there was a test that confirms this for TTL and TYPE, but not for EXISTS.
  4. commands writing to the key metadata (EXPIRE)
  • it's clear that groups 1 and 2 should "touch" the key LRU/LFU.
  • in the past (and again after the fix this PR makes to an accidental change), group 3 (TYPE, EXIST, TTL) would not "touch" the LRU.
  • regarding group 4 (the EXPIRE), i have mixed feelings. till now it did "touch" the LRU, and i'm not sure i wanna change that.

So this means that if we agree, this PR is ready to be merged.
the summary of the changes is:

  • EXIST changed to touch the LRU (revert of an unintended change)
  • DEBUG DIGEST-VALUE changed to handle expired keys (like DEBUG OBJECT)
  • OBJECT changed to avoid handling logically expired keys

@redis/core-team please approve. or state what you think about EXPIRE's behavior.

@itamarhaber
Copy link
Member

Agreed about not changing the existing touchy EXPIRE for now.

Copy link
Collaborator

@yossigo yossigo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree about EXPIRE touching the key. I think it makes sense for metadata introspection to not touch LRU, but metadata modification should touch it.

@oranagra oranagra changed the title EXISTS should not alter the last access time of a key EXISTS should not alter LRU, OBJECT should not reveal expired keys on replica Nov 18, 2020
@oranagra oranagra merged commit f8ae991 into redis:unstable Nov 18, 2020
@oranagra oranagra changed the title EXISTS should not alter LRU, OBJECT should not reveal expired keys on replica EXISTS should not alter LRU, OBJECT should not reveal expired keys Nov 18, 2020
@oranagra
Copy link
Member

Note: the commit comment that was merged was wrong, in the sense that it indicated that the effect of the change in OBJECT is only on replica, but in fact the change affects master too.

JackieXie168 pushed a commit to JackieXie168/redis that referenced this pull request Nov 19, 2020
… replica (redis#8016)

The bug was introduced by redis#5021 which only attempted avoid EXIST on an
already expired key from returning 1 on a replica.

Before that commit, dbExists was used instead of
lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)

Other than that, this commit fixes OBJECT to also come empty handed on
expired keys in replica.

And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
the key regardless of it's expired state)
@oranagra oranagra mentioned this pull request Jan 11, 2021
oranagra pushed a commit that referenced this pull request Jan 12, 2021
… replica (#8016)

The bug was introduced by #5021 which only attempted avoid EXIST on an
already expired key from returning 1 on a replica.

Before that commit, dbExists was used instead of
lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU)

Other than that, this commit fixes OBJECT to also come empty handed on
expired keys in replica.

And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from
the key regardless of it's expired state)

(cherry picked from commit f8ae991)
@oranagra oranagra mentioned this pull request Jan 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes indication that this issue needs to be mentioned in the release notes state:major-decision Requires core team consensus state:to-be-merged The PR should be merged soon, even if not yet ready, this is used so that it won't be forgotten

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants