Skip to content

Overhead is the allocated size of the AOF buffer, not its length#5453

Merged
antirez merged 1 commit intoredis:unstablefrom
damz:pr/aof-buffer-evict
Oct 24, 2018
Merged

Overhead is the allocated size of the AOF buffer, not its length#5453
antirez merged 1 commit intoredis:unstablefrom
damz:pr/aof-buffer-evict

Conversation

@damz
Copy link
Contributor

@damz damz commented Oct 16, 2018

freeMemoryGetNotCountedMemory() uses sdslen(server.aof_buf) when trying to remove memory that is not used as part of the main database, where it should use sdsalloc instead. This causes AOF to induce eviction although it is unecessary, and could end up resulting in OOM errors being returned to the client if there is not enough evictable keys available.

We were trying to debug why Redis was returning a OOM error on a write spike on a old 3.2.11 server with AOF enabled, and code review led us to this. We haven't confirmed that it solves the problem. I'm uploading this for discussion.

@antirez
Copy link
Contributor

antirez commented Oct 17, 2018

This looks correct to me @damz and indeed may be source of issues. Btw in your investigation, you actually found the following?

  1. Write spike
  2. AOF buffer enlarged significantly before being flushed
  3. This in turn causing maxmemory to be reached.
  4. ??? that's the part that is odd ??? freeMemoryIfNeeded() not able to reclaim memory?
  5. OOM error returned.

I don't understand 4, what was the maxmemory policy? Anyway the fix is needed, but technically with an evicting memory policy we should just see a mass-eviction of keys because of this bug, and not an error reported to the client.

CCing @oranagra @soloestoy FYI.

@soloestoy
Copy link
Contributor

For point 2

AOF buffer enlarged significantly before being flushed

As I know there are two scenarios can lead to that:

  1. Lua script or MULTI/EXEC contains too much write commands.

  2. KEYS command trigger too much keys expired.

@damz
Copy link
Contributor Author

damz commented Oct 17, 2018

I don't understand 4, what was the maxmemory policy? Anyway the fix is needed, but technically with an evicting memory policy we should just see a mass-eviction of keys because of this bug, and not an error reported to the client.

@antirez In this case, the policy was maxmemory-policy volatile-lru and there was no volatile keys left.

@yyevgenii
Copy link

Anyway the fix is needed, but technically with an evicting memory policy we should just see a mass-eviction of keys because of this bug, and not an error reported to the client.

Actually we see both: the eviction keys counter is pretty high (tens thousands) and time to time we're getting the "OOM command not allowed" error.

@soloestoy
Copy link
Contributor

KEYS command trigger too much keys expired.

@antirez maybe we can disable key expiration in KEYS command, just a suggestion #5470.

@antirez
Copy link
Contributor

antirez commented Oct 24, 2018

@soloestoy merged your PR. Also merging @damz PR. Thanks both.

@antirez antirez merged commit a2131f9 into redis:unstable Oct 24, 2018
JackieXie168 pushed a commit to JackieXie168/redis that referenced this pull request Dec 17, 2018
Overhead is the allocated size of the AOF buffer, not its length
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants