Skip to content

RedisGears crashing KeyDB after 1-2hrs  #276

@mattaylor

Description

@mattaylor

Describe the bug

I am running KeyDB 6.06 with RediGears 1.0.3 in an Ubuntu docker container on OSX. I am writing test data every test 10seconds from a Node-Red app, and have some python scripts triggered by RedisGears to publish messages from redis streams back to an MQTT broker. Everything was works as expected for about an hour or two and then then keydb crashes - See crash report below.
I have also noticed that KeyDB blocks completely whenever I try to re-register a redis gear even after all previously registered gears have. been sucessfully unregistered, so I have to remove the previous records of all registrations from the AOF before the process the starts. I am not sure if this is a related problem or not.

** Log Files **

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
13:18:M 04 Jan 2021 09:23:31.958 # KeyDB 6.0.16 crashed by signal: 7
13:18:M 04 Jan 2021 09:23:31.958 # Crashed running the instruction at: 0x7f55d0121d5b
13:18:M 04 Jan 2021 09:23:31.958 # Accessing address: (nil)
13:18:M 04 Jan 2021 09:23:31.958 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
/usr/lib/keydb/modules/redisgears.so(Gears_dictRehash+0x7b) [0x7f55d0121d5b]

Backtrace:
keydb-server :6379(logStackTrace(ucontext_t)+0x6b) [0x55c7721d892b]
keydb-server :6379(sigsegvHandler(int, siginfo_t, void*)+0xc9) [0x55c7721d90c9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f55d7e8b8a0]
/usr/lib/keydb/modules/redisgears.so(Gears_dictRehash+0x7b) [0x7f55d0121d5b]
/usr/lib/keydb/modules/redisgears.so(Gears_dictFind+0xca) [0x7f55d01225da]
/usr/lib/keydb/modules/redisgears.so(Gears_dictFetchValue+0x9) [0x7f55d01225e9]
/usr/lib/keydb/modules/redisgears.so(ExecutionPlan_FindByStrId+0x82) [0x7f55d012ab42]
/usr/lib/keydb/modules/redisgears.so(+0xfb181) [0x7f55d0131181]
/usr/lib/keydb/modules/redisgears.so(EPStatus_DoneAction+0x73) [0x7f55d012c5f3]
/usr/lib/keydb/modules/redisgears.so(+0xf16b5) [0x7f55d01276b5]
/usr/lib/keydb/modules/redisgears.so(+0xf3ffb) [0x7f55d0129ffb]
/usr/lib/keydb/modules/redisgears.so(+0xfbd1e) [0x7f55d0131d1e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f55d7e806db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f55d7ba9a3f]

------ INFO OUTPUT ------

Server

redis_version:6.0.16
redis_git_sha1:63c47a09
redis_git_dirty:1
redis_build_id:f20f71e10920f6e0
redis_mode:standalone
os:Linux 4.19.121-linuxkit x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.5.0
process_id:13
run_id:ab281ceb6e2c6a003a01a8daf794df39533e7a66
tcp_port:6379
uptime_in_seconds:22014
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:15916691
executable:/data/keydb-server
config_file:/etc/keydb/keydb.conf

Clients

connected_clients:10
client_recent_max_input_buffer:4
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:0
thread_0_clients:10
thread_1_clients:0

Memory

used_memory:20075216
used_memory_human:19.15M
used_memory_rss:39182336
used_memory_rss_human:37.37M
used_memory_peak:20075216
used_memory_peak_human:19.15M
used_memory_peak_perc:101.78%
used_memory_overhead:14218110
used_memory_startup:14023624
used_memory_dataset:5857106
used_memory_dataset_perc:96.79%
allocator_allocated:21014248
allocator_active:23867392
allocator_resident:29298688
total_system_memory:2084605952
total_system_memory_human:1.94G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.14
allocator_frag_bytes:2853144
allocator_rss_ratio:1.23
allocator_rss_bytes:5431296
rss_overhead_ratio:1.34
rss_overhead_bytes:9883648
mem_fragmentation_ratio:1.99
mem_fragmentation_bytes:19516384
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:171878
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

Persistence

loading:0
rdb_changes_since_last_save:359
rdb_bgsave_in_progress:1
rdb_last_save_time:1609744732
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:0
rdb_last_cow_size:1101824
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:836246
aof_base_size:0
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

Stats

total_connections_received:23
total_commands_processed:10624
instantaneous_ops_per_sec:0
total_net_input_bytes:143782
total_net_output_bytes:23709
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:684
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:7938
keyspace_misses:4298
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:1281
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0

Replication

role:master
connected_slaves:0
master_replid:e47f9aa372ff09053d8d315656f261b63f409322
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:59.054420
used_cpu_user:57.687634
used_cpu_sys_children:0.844160
used_cpu_user_children:2.318396
server_threads:2
long_lock_waits:2

Modules

module:name=rg,ver=10003,api=1,filters=0,usedby=[],using=[],options=[]

Commandstats

cmdstat_xadd:calls=1078,usec=119653,usec_per_call=111.00
cmdstat_xack:calls=1077,usec=14752,usec_per_call=13.70
cmdstat_scan:calls=3,usec=45,usec_per_call=15.00
cmdstat_xtrim:calls=1078,usec=14176,usec_per_call=13.15
cmdstat_xreadgroup:calls=1995,usec=66209,usec_per_call=33.19
cmdstat_xgroup:calls=306,usec=4224,usec_per_call=13.80
cmdstat_rg.pyexecute:calls=1,usec=4153,usec_per_call=4153.00
cmdstat_xinfo:calls=306,usec=5481,usec_per_call=17.91
cmdstat_set:calls=1079,usec=54158,usec_per_call=50.19
cmdstat_config:calls=1,usec=22,usec_per_call=22.00
cmdstat_info:calls=1,usec=17,usec_per_call=17.00
cmdstat_get:calls=734,usec=6751,usec_per_call=9.20
cmdstat_xlen:calls=1077,usec=8925,usec_per_call=8.29
cmdstat_xread:calls=1889,usec=22227,usec_per_call=11.77

Cluster

cluster_enabled:0

Keyspace

db0:keys=309,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=3657 addr=172.17.0.1:55716 fd=96 name= age=16424 idle=16424 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=3658 addr=172.17.0.1:55718 fd=97 name= age=16424 idle=16424 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=3663 addr=172.17.0.1:55738 fd=103 name= age=16424 idle=7196 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=xadd user=default
id=3664 addr=172.17.0.1:55740 fd=104 name= age=16424 idle=16424 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=3665 addr=172.17.0.1:55736 fd=102 name= age=16424 idle=16424 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=3659 addr=172.17.0.1:55722 fd=98 name= age=16424 idle=16424 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=3660 addr=172.17.0.1:55724 fd=99 name= age=16424 idle=16424 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=3661 addr=172.17.0.1:55720 fd=100 name= age=16424 idle=16424 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=3656 addr=172.17.0.1:55708 fd=95 name= age=16424 idle=16424 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=3662 addr=172.17.0.1:55732 fd=101 name= age=16424 idle=16424 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default

------ REGISTERS ------
13:18:M 04 Jan 2021 09:23:31.960 #
RAX:00007f55d061d220 RBX:00007f55d2613760
RCX:0000000000000000 RDX:fffffffffffffffe
RDI:00007f55c42a1000 RSI:0000000000000001
RBP:0d352a02016a00b7 RSP:00007f55cb5f8eb0
R8 :0000000000000001 R9 :0000000000000000
R10:00007f55d7c26f20 R11:00007f55d0347060
R12:0000000000000000 R13:0000000000000000
R14:0000000000000009 R15:00007f55c4333300
RIP:00007f55d0121d5b EFL:0000000000010206
CSGSFS:002b000000000033
13:18:M 04 Jan 2021 09:23:31.960 # (00007f55cb5f8ebf) -> 00007f55d01225e9
13:18:M 04 Jan 2021 09:23:31.960 # (00007f55cb5f8ebe) -> 00007f55c4333300
13:18:M 04 Jan 2021 09:23:31.960 # (00007f55cb5f8ebd) -> 00007f55d03378a8
13:18:M 04 Jan 2021 09:23:31.960 # (00007f55cb5f8ebc) -> 00007f55c4202940
13:18:M 04 Jan 2021 09:23:31.960 # (00007f55cb5f8ebb) -> 00007f55c436c740
13:18:M 04 Jan 2021 09:23:31.960 # (00007f55cb5f8eba) -> 00007f55cb5f8f40
13:18:M 04 Jan 2021 09:23:31.961 # (00007f55cb5f8eb9) -> 00007f55c4202940
13:18:M 04 Jan 2021 09:23:31.961 # (00007f55cb5f8eb8) -> 00007f55a7c0b000
13:18:M 04 Jan 2021 09:23:31.961 # (00007f55cb5f8eb7) -> 00007f55d01225da
13:18:M 04 Jan 2021 09:23:31.961 # (00007f55cb5f8eb6) -> 00007f55c4333300
13:18:M 04 Jan 2021 09:23:31.961 # (00007f55cb5f8eb5) -> 00007f55d03378a8
13:18:M 04 Jan 2021 09:23:31.961 # (00007f55cb5f8eb4) -> 00007f55c4202940
13:18:M 04 Jan 2021 09:23:31.961 # (00007f55cb5f8eb3) -> 00007f55cb5f8f40
13:18:M 04 Jan 2021 09:23:31.961 # (00007f55cb5f8eb2) -> 00007f55d2613760
13:18:M 04 Jan 2021 09:23:31.961 # (00007f55cb5f8eb1) -> 00007f55c4202940
13:18:M 04 Jan 2021 09:23:31.961 # (00007f55cb5f8eb0) -> 0000000000000030

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
13:18:M 04 Jan 2021 09:23:31.961 # Bio thread for job type #0 terminated
13:18:M 04 Jan 2021 09:23:31.961 # Bio thread for job type #1 terminated
13:18:M 04 Jan 2021 09:23:31.961 # Bio thread for job type #2 terminated
*** Preparing to test memory region 55c772545000 (2281472 bytes)
*** Preparing to test memory region 55c774721000 (593920 bytes)
*** Preparing to test memory region 7f5590000000 (135168 bytes)
*** Preparing to test memory region 7f5594000000 (135168 bytes)
*** Preparing to test memory region 7f5598000000 (135168 bytes)
*** Preparing to test memory region 7f559c000000 (135168 bytes)
*** Preparing to test memory region 7f55a0000000 (135168 bytes)
*** Preparing to test memory region 7f55a4780000 (8912896 bytes)
*** Preparing to test memory region 7f55a5200000 (8388608 bytes)
*** Preparing to test memory region 7f55a5a00000 (2097152 bytes)
*** Preparing to test memory region 7f55a5e00000 (8388608 bytes)
*** Preparing to test memory region 7f55a6600000 (4194304 bytes)
*** Preparing to test memory region 7f55a6bff000 (8388608 bytes)
*** Preparing to test memory region 7f55a7400000 (8388608 bytes)
*** Preparing to test memory region 7f55a7c00000 (4329472 bytes)
*** Preparing to test memory region 7f55ac000000 (4194304 bytes)
*** Preparing to test memory region 7f55ac8ff000 (5242880 bytes)
*** Preparing to test memory region 7f55ace00000 (8388608 bytes)
*** Preparing to test memory region 7f55ad600000 (2097152 bytes)
*** Preparing to test memory region 7f55ad8be000 (8388608 bytes)
*** Preparing to test memory region 7f55af32d000 (2621440 bytes)
*** Preparing to test memory region 7f55af7cc000 (4096 bytes)
*** Preparing to test memory region 7f55b0000000 (135168 bytes)
*** Preparing to test memory region 7f55b4000000 (135168 bytes)
*** Preparing to test memory region 7f55b8000000 (135168 bytes)
*** Preparing to test memory region 7f55bcc00000 (8388608 bytes)
*** Preparing to test memory region 7f55bd400000 (2097152 bytes)
*** Preparing to test memory region 7f55bd800000 (8388608 bytes)
*** Preparing to test memory region 7f55be000000 (4194304 bytes)
*** Preparing to test memory region 7f55be600000 (8388608 bytes)
*** Preparing to test memory region 7f55bee00000 (6291456 bytes)
*** Preparing to test memory region 7f55bf5ff000 (2097152 bytes)
*** Preparing to test memory region 7f55bf800000 (8388608 bytes)
*** Preparing to test memory region 7f55c0000000 (135168 bytes)
*** Preparing to test memory region 7f55c4000000 (6291456 bytes)
*** Preparing to test memory region 7f55c47ff000 (2097152 bytes)
*** Preparing to test memory region 7f55c4a00000 (8388608 bytes)
*** Preparing to test memory region 7f55c5200000 (2097152 bytes)
*** Preparing to test memory region 7f55c54c6000 (8388608 bytes)
*** Preparing to test memory region 7f55c5cc7000 (8388608 bytes)
*** Preparing to test memory region 7f55c64c8000 (8388608 bytes)
*** Preparing to test memory region 7f55c6cc9000 (8388608 bytes)
*** Preparing to test memory region 7f55c74ca000 (8388608 bytes)
*** Preparing to test memory region 7f55c7ccb000 (8388608 bytes)
*** Preparing to test memory region 7f55c84cc000 (8388608 bytes)
*** Preparing to test memory region 7f55c90f3000 (5242880 bytes)
*** Preparing to test memory region 7f55c99f9000 (4194304 bytes)
*** Preparing to test memory region 7f55c9dfa000 (8388608 bytes)
*** Preparing to test memory region 7f55ca5fb000 (8388608 bytes)
*** Preparing to test memory region 7f55cadfc000 (8388608 bytes)
*** Preparing to test memory region 7f55cb5fd000 (8388608 bytes)
*** Preparing to test memory region 7f55cc000000 (135168 bytes)
*** Preparing to test memory region 7f55d0693000 (143360 bytes)
*** Preparing to test memory region 7f55d08be000 (8388608 bytes)
*** Preparing to test memory region 7f55d12ca000 (24576 bytes)
*** Preparing to test memory region 7f55d14e8000 (8192 bytes)
*** Preparing to test memory region 7f55d1900000 (17825792 bytes)
*** Preparing to test memory region 7f55d2d6c000 (188416 bytes)
*** Preparing to test memory region 7f55d30a2000 (4096 bytes)
*** Preparing to test memory region 7f55d32ec000 (4096 bytes)
*** Preparing to test memory region 7f55d3d78000 (4096 bytes)
*** Preparing to test memory region 7f55d42a7000 (4096 bytes)
*** Preparing to test memory region 7f55d4bf6000 (4096 bytes)
*** Preparing to test memory region 7f55d526c000 (8192 bytes)
*** Preparing to test memory region 7f55d58ae000 (4096 bytes)
*** Preparing to test memory region 7f55d65d5000 (4096 bytes)
*** Preparing to test memory region 7f55d6fcf000 (8192 bytes)
*** Preparing to test memory region 7f55d7e75000 (16384 bytes)
*** Preparing to test memory region 7f55d8094000 (16384 bytes)
*** Preparing to test memory region 7f55d89d3000 (16384 bytes)
*** Preparing to test memory region 7f55d8e9f000 (12288 bytes)
*** Preparing to test memory region 7f55d9bc9000 (114688 bytes)
*** Preparing to test memory region 7f55d9bea000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.13:18:M 04 Jan 2021 09:23:32.641 # KeyDB 6.0.16 crashed by signal: 11
13:18:M 04 Jan 2021 09:23:32.641 # Crashed running the instruction at: 0x7f55cb4f771b
13:18:M 04 Jan 2021 09:23:32.641 # Accessing address: (nil)
13:18:M 04 Jan 2021 09:23:32.641 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
[0x7f55cb4f771b]

Backtrace:
keydb-server :6379(logStackTrace(ucontext_t)+0x6b) [0x55c7721d892b]
keydb-server :6379(sigsegvHandler(int, siginfo_t, void*)+0xc9) [0x55c7721d90c9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f55d7e8b8a0]
[0x7f55cb4f771b]

------ INFO OUTPUT ------
Please copy everything after this line.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions