Skip to content

[CRASH] Redis cluster 7.2.0 crashed by signal: 11, si_code: 1 when joining existing cluster (ubuntu 20.04) #12507

@erihu78

Description

@erihu78

Trying to upgrade our Redis Lab Cluster from 7.0 to 7.2 but nodes keep crashing on startup
Have also tried to set up a completely new replica, there redis 7.2 works fine until i try to add the node to the cluster, then it also crashes.

After downgrading to 7.0.12 the nodes works fine again - so seems to something specific to 7.2

Additional information
OS: Ubuntu 20.04
HW: Virtual (VmWare)

Bug report:

=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis 7.2.0 crashed by signal: 11, si_code: 1
Accessing address: (nil)
Crashed running the instruction at: 0x562d0d6623fb

------ REGISTERS ------
RAX:3966386237303330 RBX:00007f241e76fa00
RCX:0000000000000001 RDX:3235306133326465
RDI:00007f241e76fa00 RSI:0000000000000000
RBP:0000000000000000 RSP:00007ffcb3f46590
R8 :0000000064dec2ad R9 :000000000ff7e624
R10:00007ffcb3fdf080 R11:00007f241e770400
R12:00007f241e76fa30 R13:0000018a06287589
R14:00000000000000b9 R15:00007f241e76fa00
RIP:0000562d0d6623fb EFL:0000000000010206
CSGSFS:002b000000000033
(00007ffcb3f4659f) -> 0000000000000400
(00007ffcb3f4659e) -> 00007ffcb3f466b0
(00007ffcb3f4659d) -> ddd7f1f3918dd200
(00007ffcb3f4659c) -> 0000562d0dfd90e0
(00007ffcb3f4659b) -> ddd7f1f3918dd200
(00007ffcb3f4659a) -> 00007ffcb3f466b0
(00007ffcb3f46599) -> 00007f241f05caad
(00007ffcb3f46598) -> 00007f241ea1bb60
(00007ffcb3f46597) -> 0000562d0d8660e0
(00007ffcb3f46596) -> 0000562d0d8660e0
(00007ffcb3f46595) -> 0000000000000001
(00007ffcb3f46594) -> 0000562d0d8660e0
(00007ffcb3f46593) -> 0000562d0d65cbd6
(00007ffcb3f46592) -> 00007f24060cd000
(00007ffcb3f46591) -> 00007f240b41c7d0
(00007ffcb3f46590) -> 00007f24060cd050

------ INFO OUTPUT ------
# Server
redis_version:7.2.0
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:3a1b7000f8371834
redis_mode:cluster
os:Linux 5.4.0-155-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:9.4.0
process_id:3540560
process_supervised:no
run_id:66c7f64c1fd1a8e265b162e3a810759c6cea4385
tcp_port:6379
server_time_usec:1692320429449515
uptime_in_seconds:1
uptime_in_days:0
hz:100
configured_hz:100
lru_clock:14598829
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0
listener2:name=tls,bind=127.0.0.1,bind=10.3.48.124,port=6379
# Clients
connected_clients:0
cluster_connections:16
maxclients:1000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
# Memory
used_memory:305765264
used_memory_human:291.60M
used_memory_rss:318922752
used_memory_rss_human:304.15M
used_memory_peak:305765264
used_memory_peak_human:291.60M
used_memory_peak_perc:100.00%
used_memory_overhead:34522872
used_memory_startup:1268264
used_memory_dataset:271242392
used_memory_dataset_perc:89.08%
allocator_allocated:302066304
allocator_active:302301184
allocator_resident:311668736
total_system_memory:16742162432
total_system_memory_human:15.59G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:234880
allocator_rss_ratio:1.03
allocator_rss_bytes:9367552
rss_overhead_ratio:1.02
rss_overhead_bytes:7254016
mem_fragmentation_ratio:1.06
mem_fragmentation_bytes:17047752
mem_not_counted_for_evict:8
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:0
mem_cluster_links:16008
mem_aof_buffer:8
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1692320428
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:518644
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:197591285
aof_base_size:197591285
aof_pending_rewrite:0
aof_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:0
total_commands_processed:2
instantaneous_ops_per_sec:0
total_net_input_bytes:0
total_net_output_bytes:0
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:518777
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:0
total_writes_processed:0
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:0
reply_buffer_expands:0
eventloop_cycles:2
eventloop_duration_sum:58115
eventloop_duration_cmd_sum:0
instantaneous_eventloop_cycles_per_sec:0
instantaneous_eventloop_duration_usec:0
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:d52983f364f9364b19f2c10cd846de059fd39830
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:10485760
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.153794
used_cpu_user:1.183030
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.153781
used_cpu_user_main_thread:1.182937
# Modules
module:name=search,ver=20804,api=1,filters=0,usedby=[],using=[],options=[]
# Commandstats
cmdstat_info:calls=1,usec=21,usec_per_call=21.00,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=1,usec=8,usec_per_call=8.00,rejected_calls=0,failed_calls=0
# Errorstats
# Latencystats
latency_percentiles_usec_info:p50=21.119,p99=21.119,p99.9=21.119
latency_percentiles_usec_config|get:p50=8.031,p99=8.031,p99.9=8.031
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=518644,expires=0,avg_ttl=0
# Cluster info
cluster_state:fail
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:9
cluster_size:1
cluster_current_epoch:187
cluster_my_epoch:185
cluster_stats_messages_pong_sent:1
cluster_stats_messages_sent:1
cluster_stats_messages_ping_received:1
cluster_stats_messages_received:1
total_cluster_links_buffer_limit_exceeded:0

------ CLUSTER NODES OUTPUT ------
3b4c34db7a673629d5acafa3fd2f2d6de0212ade 10.3.48.128:0@16379,,tls-port=6379,shard-id=6d5d1e470c2147de434f691196176fc95e50ca75 slave 
de9123b19a3250036e53e65da8cd03e2e8ac7932 0 1692320428064 186 disconnected
e2163f560a2e89955dd222d940f32d2917ed1a30 10.3.48.134:0@16379,,tls-port=6379,shard-id=1351b2c3cee680e1cdb199d4ab7364211b7a32c2 slave 
5c7101d30e77b341d923539492998fcadaca6392 0 1692320428064 187 disconnected
8544cf7802fb5fc751c02029f553af7737c12f39 10.3.48.124:0@16379,,tls-port=6379,shard-id=031b66292ffa165413bed5d2ed51459ca2cf0ad7 myself,slave 
12ec951aeceeb2803feece639410231615b3dac3 0 1692320428064 185 connected
5c7101d30e77b341d923539492998fcadaca6392 10.3.48.112:0@16379,,tls-port=6379,shard-id=a69d91dcac934e26e4562902e67cf43312964d2a master - 0 1692320428064 1
10923-16383
9c25abd4846d829426b1c439e28148654a5fd5bd 10.3.48.63:0@16379,,tls-port=6379,shard-id=0307b8f9ed23a052f0b87fba06270e9ab23fc8f2 slave 12ec951aeceeb2803feec
0 1692320428064 185 disconnected
5e93950692ef5dcc1f492e661f16880eeb127471 10.3.48.127:0@16379,,tls-port=6379,shard-id=44f1d80663b0b248978219969da897543080a7c5 slave 
de9123b19a3250036e53e65da8cd03e2e8ac7932 0 1692320428064 186 disconnected
4f3ebae03f901cfb4a48f2fee01be2b99ee277d4 10.3.48.69:0@16379,,tls-port=6379,shard-id=427ae9b57255bd7e2b067887e73eac362d12345f slave 5c7101d30e77b341d9235
0 1692320428064 187 disconnected
12ec951aeceeb2803feece639410231615b3dac3 10.3.48.125:0@16379,,tls-port=6379,shard-id=f8fda916f49e56e8b866b8b874e4b66a38aeccf6 master - 0 1692320428064 1
0-5460
de9123b19a3250036e53e65da8cd03e2e8ac7932 10.3.48.66:0@16379,,tls-port=6379,shard-id=673c6db8a7d82eba1374b8d0bcab881605df5cdc master - 0 1692320428064 18
5461-10922

------ CLIENT LIST OUTPUT ------

------ MODULES INFO OUTPUT ------
# search_version
search_version:2.8.4
search_redis_version:7.2.0 - oss
# search_index
search_number_of_indexes:2
# search_fields_statistics
search_fields_numeric:Numeric=10,Sortable=2
search_fields_tag:Tag=2
# search_dialect_statistics
search_dialect_1:0
search_dialect_2:0
search_dialect_3:0
search_dialect_4:0
# search_runtime_configurations
search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:1000000
search_max_aggregate_results:-1
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3

------ CONFIG DEBUG OUTPUT ------
lazyfree-lazy-user-flush no
repl-diskless-load disabled
slave-read-only yes
list-compress-depth 0
sanitize-dump-payload no
activedefrag no
replica-read-only yes
lazyfree-lazy-user-del no
repl-diskless-sync no
proto-max-bulk-len 512mb
io-threads 1
client-query-buffer-limit 1gb
lazyfree-lazy-eviction no
io-threads-do-reads no
lazyfree-lazy-server-del no
lazyfree-lazy-expire no
------ FAST MEMORY TEST ------
Bio worker thread #0 terminated
Bio worker thread #1 terminated
Bio worker thread #2 terminated
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /usr/bin/redis-server 127.0.0.1:6379 [cluster] (base 0x562d0d568000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------
=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions