Memory cache issue #207
Description
Description
Sometime during high load we are facing OOM in keydb kubernetes pods, but before failure keydb memory usage not even close to maxmemory limit(in our case 8Gb). We are using no-eviction policy because of specific data in memory that shouldn't be cleared, so we are giving as much memory as possible. What we have mentioned that we have really huge system cache usage during high load. For example, when we have keydb used_memory_rss_human:3.73G, real system memory usage could be more than 5gb(5477M), where cache is 1,6Gb(1661095936). As result, when we have huge load, cache could be 2x bigger than used_memory_rss_human and we get OOM. Any ideas what can cause so huge memory usage?
My keydb-memory config:
Memory
used_memory:3841953104
used_memory_human:3.58G
used_memory_rss:4006895616
used_memory_rss_human:3.73G
used_memory_peak:7434135544
used_memory_peak_human:6.92G
used_memory_peak_perc:51.68%
used_memory_overhead:10334966
used_memory_startup:9088888
used_memory_dataset:3831618138
used_memory_dataset_perc:99.97%
allocator_allocated:3842460296
allocator_active:3916775424
allocator_resident:4032458752
total_system_memory:67557625856
total_system_memory_human:62.92G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:8589934592
maxmemory_human:8.00G
maxmemory_policy:noeviction
allocator_frag_ratio:1.02
allocator_frag_bytes:74315128
allocator_rss_ratio:1.03
allocator_rss_bytes:115683328
rss_overhead_ratio:0.99
rss_overhead_bytes:-25563136
mem_fragmentation_ratio:1.04
mem_fragmentation_bytes:164942576
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:85930
mem_clients_normal:103116
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
Memory usage at the same time range:
echo $(( $(cat /sys/fs/cgroup/memory/memory.usage_in_bytes) / ( 1024 * 1024 ) ))M
5477M
max usage:
echo $(( $(cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes) / ( 1024 * 1024 ) ))M
8191M
echo
bash: cache 1661095936
rss 3999059968
rss_huge 0
shmem 0
mapped_file 0
dirty 540672
writeback 0
swap 0
pgpgin 1407052779
pgpgout 1405670648
pgfault 255782109
pgmajfault 0
pgmajfault_s 0
pgmajfault_a 0
pgmajfault_f 0
inactive_anon 0
active_anon 3965030400
inactive_file 1695313920
active_file 806912
unevictable 0
hierarchical_memory_limit 8589934592
hierarchical_memsw_limit 8589934592
total_cache 1661095936
total_rss 3999059968
total_rss_huge 0
total_shmem 0
total_mapped_file 0
total_dirty 540672
total_writeback 0
total_swap 0
total_pgpgin 1407052779
total_pgpgout 1405670648
total_pgfault 255782109
total_pgmajfault 0
total_pgmajfault_s 0
total_pgmajfault_a 0
total_pgmajfault_f 0
total_inactive_anon 0
total_active_anon 3965030400
total_inactive_file 1695313920
total_active_file 806912
total_unevictable 0 / ( 1024 * 1024 ) : syntax error in expression (error token is "1661095936
rss 3999059968
rss_huge 0
shmem 0
mapped_file 0
dirty 540672
writeback 0
swap 0
pgpgin 1407052779
pgpgout 1405670648
pgfault 255782109
pgmajfault 0
pgmajfault_s 0
pgmajfault_a 0
pgmajfault_f 0
inactive_anon 0
active_anon 3965030400
inactive_file 1695313920
active_file 806912
unevictable 0
hierarchical_memory_limit 8589934592
hierarchical_memsw_limit 8589934592
total_cache 1661095936
total_rss 3999059968
total_rss_huge 0
total_shmem 0
total_mapped_file 0
total_dirty 540672
total_writeback 0
total_swap 0
total_pgpgin 1407052779
total_pgpgout 1405670648
total_pgfault 255782109
total_pgmajfault 0
total_pgmajfault_s 0
total_pgmajfault_a 0
total_pgmajfault_f 0
total_inactive_anon 0
total_active_anon 3965030400
total_inactive_file 1695313920
total_active_file 806912
total_unevictable 0 / ( 1024 * 1024 ) ")
Keydb image - eqalpha/keydb:x86_64_v6.0.8
Non-deafult parameters:
--active-replica no \
--multi-master no \
--maxmemory 8gb \
--maxmemory-policy noeviction \
--appendonly no \
--proto-max-bulk-len 2gb \
--client-query-buffer-limit 2gb \
--bind 0.0.0.0 \
--port "$port" \
--protected-mode yes \
--save 600 10000 \
--tcp-backlog 1000 \
--replica-serve-stale-data yes \
--server-threads 2 \
Thanks in advance