Skip to content

[CRASH] Redis cluster node crashes with assertion failure 'myself->numslots == 0' is not true #12268

@Congyu-Liu

Description

@Congyu-Liu

When testing redis cluster, we triggered an assertion failure in a cluster node. This happens in both the latest release (7.0.11) and the unstable branch.

Log

4013349:C 05 Jun 2023 19:17:28.867 # WARNING: Changing databases number from 16 to 1 since we are in cluster mode
4013349:C 05 Jun 2023 19:17:28.867 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
4013349:C 05 Jun 2023 19:17:28.867 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4013349:C 05 Jun 2023 19:17:28.867 * Redis version=255.255.255, bits=64, commit=0bd1a3a4, modified=0, pid=4013349, just started
4013349:C 05 Jun 2023 19:17:28.867 * Configuration loaded
4013349:M 05 Jun 2023 19:17:28.868 * Increased maximum number of open files to 10032 (it was originally set to 1024).
4013349:M 05 Jun 2023 19:17:28.868 * monotonic clock: POSIX clock_gettime
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 255.255.255 (0bd1a3a4/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in cluster mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 4013349
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           https://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

4013349:M 05 Jun 2023 19:17:28.868 * Node configuration loaded, I'm 98e8b3aab4861ee22fdf92972c206a8420287b69
4013349:M 05 Jun 2023 19:17:28.868 * Server initialized
4013349:M 05 Jun 2023 19:17:28.869 * Loading RDB produced by version 255.255.255
4013349:M 05 Jun 2023 19:17:28.869 * RDB age 3 seconds
4013349:M 05 Jun 2023 19:17:28.869 * RDB memory usage when created 1.56 Mb
4013349:M 05 Jun 2023 19:17:28.869 * Done loading RDB, keys loaded: 0, keys expired: 0.
4013349:M 05 Jun 2023 19:17:28.869 * DB loaded from disk: 0.000 seconds
4013349:M 05 Jun 2023 19:17:28.869 * Ready to accept connections tcp
4013349:M 05 Jun 2023 19:18:31.186 - Accepted 127.0.0.1:49084
4013349:M 05 Jun 2023 19:18:31.192 * configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH
4013349:M 05 Jun 2023 19:18:31.216 - Accepting cluster node connection from 127.0.0.1:34398
4013349:M 05 Jun 2023 19:18:31.216 * IP address for this node updated to 127.0.0.1
4013349:M 05 Jun 2023 19:18:31.252 - Accepting cluster node connection from 127.0.0.1:34404
4013349:M 05 Jun 2023 19:18:35.195 - Client closed connection id=3 addr=127.0.0.1:49084 laddr=127.0.0.1:6379 fd=11 name= age=4 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=1024 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=cluster|nodes user=default redir=-1 resp=2 lib-name= lib-ver=
4013349:M 05 Jun 2023 19:18:36.170 * Cluster state changed: ok
4013349:M 05 Jun 2023 19:18:43.847 - Accepted 127.0.0.1:52222
4013349:M 05 Jun 2023 19:18:43.847 # Cluster state changed: fail
4013349:M 05 Jun 2023 19:18:43.848 - Client closed connection id=4 addr=127.0.0.1:52222 laddr=127.0.0.1:6379 fd=11 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=cluster|flushslots user=default redir=-1 resp=2 lib-name= lib-ver=
4013349:M 05 Jun 2023 19:18:58.606 - Accepted 127.0.0.1:47218
4013349:S 05 Jun 2023 19:18:58.606 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
4013349:S 05 Jun 2023 19:18:58.606 * Connecting to MASTER 127.0.0.1:6381
4013349:S 05 Jun 2023 19:18:58.606 * MASTER <-> REPLICA sync started
4013349:S 05 Jun 2023 19:18:58.606 * Non blocking connect for SYNC fired the event.
4013349:S 05 Jun 2023 19:18:58.606 * Master replied to PING, replication can continue...
4013349:S 05 Jun 2023 19:18:58.606 * Trying a partial resynchronization (request 87fa56f55113dec207d1dbf0caf7b5ec93da853d:1).
4013349:S 05 Jun 2023 19:18:58.606 - Client closed connection id=5 addr=127.0.0.1:47218 laddr=127.0.0.1:6379 fd=11 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=cluster|replicate user=default redir=-1 resp=2 lib-name= lib-ver=
4013349:S 05 Jun 2023 19:19:03.878 * Full resync from master: b0d2ec9ec5e1a58ce258843c59f0f8ed63720f0a:14
4013349:S 05 Jun 2023 19:19:03.879 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
4013349:S 05 Jun 2023 19:19:03.879 * Discarding previously cached master state.
4013349:S 05 Jun 2023 19:19:03.879 * MASTER <-> REPLICA sync: Flushing old data
4013349:S 05 Jun 2023 19:19:03.879 * MASTER <-> REPLICA sync: Loading DB in memory
4013349:S 05 Jun 2023 19:19:03.880 * Loading RDB produced by version 255.255.255
4013349:S 05 Jun 2023 19:19:03.880 * RDB age 0 seconds
4013349:S 05 Jun 2023 19:19:03.880 * RDB memory usage when created 1.78 Mb
4013349:S 05 Jun 2023 19:19:03.880 * Done loading RDB, keys loaded: 0, keys expired: 0.
4013349:S 05 Jun 2023 19:19:03.880 * MASTER <-> REPLICA sync: Finished with success
4013349:S 05 Jun 2023 19:19:23.106 - Accepted 127.0.0.1:39290
4013349:S 05 Jun 2023 19:19:23.107 - Client closed connection id=8 addr=127.0.0.1:39290 laddr=127.0.0.1:6379 fd=17 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=cluster|addslots user=default redir=-1 resp=2 lib-name= lib-ver=
4013349:S 05 Jun 2023 19:19:37.890 - Accepted 127.0.0.1:55736


=== REDIS BUG REPORT START: Cut & paste starting from here ===
4013349:S 05 Jun 2023 19:19:37.890 # === ASSERTION FAILED ===
4013349:S 05 Jun 2023 19:19:37.890 # ==> cluster.c:4931 'myself->numslots == 0' is not true

------ STACK TRACE ------

Backtrace:
../../redis-server *:6379 [cluster](+0x13755b)[0x555b25be055b]
../../redis-server *:6379 [cluster](clusterCommand+0xf67)[0x555b25be5ee7]
../../redis-server *:6379 [cluster](call+0x186)[0x555b25b40bb6]
../../redis-server *:6379 [cluster](processCommand+0xba9)[0x555b25b42149]
../../redis-server *:6379 [cluster](processInputBuffer+0x107)[0x555b25b67f27]
../../redis-server *:6379 [cluster](readQueryFromClient+0x368)[0x555b25b684a8]
../../redis-server *:6379 [cluster](+0x1c11ac)[0x555b25c6a1ac]
../../redis-server *:6379 [cluster](aeMain+0xf9)[0x555b25b368d9]
../../redis-server *:6379 [cluster](main+0x3df)[0x555b25b2afbf]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f843b1d7d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f843b1d7e40]
../../redis-server *:6379 [cluster](_start+0x25)[0x555b25b2b785]

------ INFO OUTPUT ------
# Server
redis_version:255.255.255
redis_git_sha1:0bd1a3a4
redis_git_dirty:0
redis_build_id:b5019077ff9fccd6
redis_mode:cluster
os:Linux 5.15.0-47-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.3.0
process_id:4013349
process_supervised:no
run_id:1a181acb00bab7e09911f945efb86254322d9dfa
tcp_port:6379
server_time_usec:1685992777890620
uptime_in_seconds:129
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:8271177
executable:/home/congyu/redis/src/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=*,bind=-::*,port=6379

# Clients
connected_clients:2
cluster_connections:4
maxclients:10000
client_recent_max_input_buffer:24
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:1854360
used_memory_human:1.77M
used_memory_rss:8544256
used_memory_rss_human:8.15M
used_memory_peak:1900440
used_memory_peak_human:1.81M
used_memory_peak_perc:97.58%
used_memory_overhead:1609404
used_memory_startup:1582480
used_memory_dataset:244956
used_memory_dataset_perc:90.10%
allocator_allocated:2138096
allocator_active:2600960
allocator_resident:12455936
total_system_memory:134750920704
total_system_memory_human:125.50G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.22
allocator_frag_bytes:462864
allocator_rss_ratio:4.79
allocator_rss_bytes:9854976
rss_overhead_ratio:0.69
rss_overhead_bytes:-3911680
mem_fragmentation_ratio:4.71
mem_fragmentation_bytes:6730152
mem_not_counted_for_evict:0
mem_replication_backlog:20508
mem_total_replication_buffers:20504
mem_clients_slaves:0
mem_clients_normal:1944
mem_cluster_links:4288
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1685992648
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:5
total_commands_processed:19
instantaneous_ops_per_sec:0
total_net_input_bytes:54382
total_net_output_bytes:19564
total_net_repl_input_bytes:306
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.02
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:26
total_writes_processed:51
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:2
reply_buffer_expands:0
eventloop_cycles:1601
eventloop_duration_sum:59284
eventloop_duration_cmd_sum:1109
instantaneous_eventloop_cycles_per_sec:12
instantaneous_eventloop_duration_usec:33
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:slave
master_host:127.0.0.1
master_port:6381
master_link_status:up
master_last_io_seconds_ago:5
master_sync_in_progress:0
slave_read_repl_offset:56
slave_repl_offset:56
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:b0d2ec9ec5e1a58ce258843c59f0f8ed63720f0a
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:56
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:15
repl_backlog_histlen:42

# CPU
used_cpu_sys:0.032695
used_cpu_user:0.033974
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.040645
used_cpu_user_main_thread:0.025403

# Modules

# Commandstats
cmdstat_ping:calls=3,usec=0,usec_per_call=0.00,rejected_calls=0,failed_calls=0
cmdstat_info:calls=3,usec=134,usec_per_call=44.67,rejected_calls=0,failed_calls=0
cmdstat_cluster|addslots:calls=2,usec=160,usec_per_call=80.00,rejected_calls=0,failed_calls=0
cmdstat_cluster|info:calls=1,usec=8,usec_per_call=8.00,rejected_calls=0,failed_calls=0
cmdstat_cluster|flushslots:calls=1,usec=492,usec_per_call=492.00,rejected_calls=0,failed_calls=0
cmdstat_cluster|nodes:calls=7,usec=211,usec_per_call=30.14,rejected_calls=0,failed_calls=0
cmdstat_cluster|replicate:calls=1,usec=94,usec_per_call=94.00,rejected_calls=0,failed_calls=0
cmdstat_cluster|set-config-epoch:calls=1,usec=10,usec_per_call=10.00,rejected_calls=0,failed_calls=0

# Errorstats

# Latencystats
latency_percentiles_usec_ping:p50=0.001,p99=0.001,p99.9=0.001
latency_percentiles_usec_info:p50=38.143,p99=72.191,p99.9=72.191
latency_percentiles_usec_cluster|addslots:p50=15.039,p99=145.407,p99.9=145.407
latency_percentiles_usec_cluster|info:p50=8.031,p99=8.031,p99.9=8.031
latency_percentiles_usec_cluster|flushslots:p50=493.567,p99=493.567,p99.9=493.567
latency_percentiles_usec_cluster|nodes:p50=28.031,p99=38.143,p99.9=38.143
latency_percentiles_usec_cluster|replicate:p50=94.207,p99=94.207,p99.9=94.207
latency_percentiles_usec_cluster|set-config-epoch:p50=10.047,p99=10.047,p99.9=10.047

# Cluster
cluster_enabled:1

# Keyspace

# Cluster info
cluster_state:fail
cluster_slots_assigned:10924
cluster_slots_ok:10924
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:2
cluster_current_epoch:3
cluster_my_epoch:3
cluster_stats_messages_ping_sent:64
cluster_stats_messages_pong_sent:70
cluster_stats_messages_sent:134
cluster_stats_messages_ping_received:68
cluster_stats_messages_pong_received:64
cluster_stats_messages_meet_received:2
cluster_stats_messages_received:134
total_cluster_links_buffer_limit_exceeded:0

------ CLUSTER NODES OUTPUT ------
a4d687936a05fabea0798f35c494390af1fd6a24 127.0.0.1:6380@16380,,shard-id=9a86af15129537b02350ee8bfe72ae2c040fd2d1 master - 0 1685992775976 2 connected 5461-10922
cab2ad6ff2a6f611b7e9873c3cafa455bd951222 127.0.0.1:6381@16381,,shard-id=63204e65945e016afa82fc37d12a465affe6b5b5 master - 0 1685992776978 3 connected 10923-16383
98e8b3aab4861ee22fdf92972c206a8420287b69 127.0.0.1:6379@16379,,shard-id=63204e65945e016afa82fc37d12a465affe6b5b5 myself,slave cab2ad6ff2a6f611b7e9873c3cafa455bd951222 0 1685992776000 3 connected 1

------ CLIENT LIST OUTPUT ------
id=7 addr=127.0.0.1:6381 laddr=127.0.0.1:42290 fd=16 name= age=34 idle=5 flags=M db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=14 argv-mem=0 multi-mem=0 rbs=1024 rbp=35 obl=0 oll=0 omem=0 tot-mem=1944 events=r cmd=ping user=(superuser) redir=-1 resp=2 lib-name= lib-ver=
id=9 addr=127.0.0.1:55736 laddr=127.0.0.1:6379 fd=17 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=79 qbuf-free=20395 argv-mem=56 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37840 events=r cmd=cluster|replicate user=default redir=-1 resp=2 lib-name= lib-ver=

------ CURRENT CLIENT INFO ------
id=9 addr=127.0.0.1:55736 laddr=127.0.0.1:6379 fd=17 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=79 qbuf-free=20395 argv-mem=56 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37840 events=r cmd=cluster|replicate user=default redir=-1 resp=2 lib-name= lib-ver=
argc: '3'
argv[0]: '"CLUSTER"'
argv[1]: '"REPLICATE"'
argv[2]: '"a4d687936a05fabea0798f35c494390af1fd6a24"'

------ EXECUTING CLIENT INFO ------
id=9 addr=127.0.0.1:55736 laddr=127.0.0.1:6379 fd=17 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=79 qbuf-free=20395 argv-mem=56 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37840 events=r cmd=cluster|replicate user=default redir=-1 resp=2 lib-name= lib-ver=
argc: '3'
argv[0]: '"CLUSTER"'
argv[1]: '"REPLICATE"'
argv[2]: '"a4d687936a05fabea0798f35c494390af1fd6a24"'

------ MODULES INFO OUTPUT ------

------ CONFIG DEBUG OUTPUT ------
repl-diskless-load disabled
lazyfree-lazy-user-flush no
io-threads-do-reads no
proto-max-bulk-len 512mb
replica-read-only yes
lazyfree-lazy-server-del no
sanitize-dump-payload no
lazyfree-lazy-user-del no
client-query-buffer-limit 1gb
lazyfree-lazy-expire no
io-threads 1
repl-diskless-sync yes
lazyfree-lazy-eviction no
activedefrag no
slave-read-only yes
list-compress-depth 0

------ FAST MEMORY TEST ------
4013349:S 05 Jun 2023 19:19:37.891 # Bio worker thread #0 terminated
4013349:S 05 Jun 2023 19:19:37.891 # Bio worker thread #1 terminated
4013349:S 05 Jun 2023 19:19:37.891 # Bio worker thread #2 terminated
*** Preparing to test memory region 555b25e26000 (2269184 bytes)
*** Preparing to test memory region 555b276d4000 (135168 bytes)
*** Preparing to test memory region 7f8430000000 (135168 bytes)
*** Preparing to test memory region 7f8437000000 (8388608 bytes)
*** Preparing to test memory region 7f8437800000 (2097152 bytes)
*** Preparing to test memory region 7f8437c00000 (8388608 bytes)
*** Preparing to test memory region 7f8438400000 (6291456 bytes)
*** Preparing to test memory region 7f8438a15000 (8388608 bytes)
*** Preparing to test memory region 7f8439216000 (8388608 bytes)
*** Preparing to test memory region 7f8439a17000 (8388608 bytes)
*** Preparing to test memory region 7f843a217000 (3145728 bytes)
*** Preparing to test memory region 7f843a800000 (8388608 bytes)
*** Preparing to test memory region 7f843b1ab000 (12288 bytes)
*** Preparing to test memory region 7f843b3c9000 (53248 bytes)
*** Preparing to test memory region 7f843b4cc000 (8192 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           http://github.com/redis/redis/issues

  If a Redis module was involved, please open in the module's repo instead.

  Suspect RAM error? Use redis-server --test-memory to verify it.

  Some other issues could be detected by redis-server --check-system
Aborted (core dumped)

Reproduce

First start a redis cluster with 3 nodes with options --protected-mode no --cluster-enabled yes --loglevel verbose --port .... Then run following commands. Here I started three nodes locally with port 6379, 6380 and 6381. Node 6379 crashes eventually.

$./redis-cli --cluster-yes --cluster create 127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: 98e8b3aab4861ee22fdf92972c206a8420287b69 127.0.0.1:6379
   slots:[0-5460] (5461 slots) master
M: a4d687936a05fabea0798f35c494390af1fd6a24 127.0.0.1:6380
   slots:[5461-10922] (5462 slots) master
M: cab2ad6ff2a6f611b7e9873c3cafa455bd951222 127.0.0.1:6381
   slots:[10923-16383] (5461 slots) master
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 127.0.0.1:6379)
M: 98e8b3aab4861ee22fdf92972c206a8420287b69 127.0.0.1:6379
   slots:[0-5460] (5461 slots) master
M: a4d687936a05fabea0798f35c494390af1fd6a24 127.0.0.1:6380
   slots:[5461-10922] (5462 slots) master
M: cab2ad6ff2a6f611b7e9873c3cafa455bd951222 127.0.0.1:6381
   slots:[10923-16383] (5461 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
$ ./redis-cli -p 6379 -c CLUSTER FLUSHSLOTS
OK
$ ./redis-cli -p 6381 -c CLUSTER MYID
"cab2ad6ff2a6f611b7e9873c3cafa455bd951222"
$ ./redis-cli -p 6379 -c CLUSTER REPLICATE cab2ad6ff2a6f611b7e9873c3cafa455bd951222
OK
$ ./redis-cli -p 6379 -c CLUSTER ADDSLOTS 1
OK
$ ./redis-cli -p 6380 -c CLUSTER MYID
"a4d687936a05fabea0798f35c494390af1fd6a24"
$ ./redis-cli -p 6379 -c CLUSTER REPLICATE a4d687936a05fabea0798f35c494390af1fd6a24
Error: Server closed the connection

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions