-
Notifications
You must be signed in to change notification settings - Fork 573
Closed
Labels
Description
Describe the bug
Am tried to use the redisearch module it works normally on my functional test but crash in load test, tested on multiple machines RED HAT and Linux
the whole node and its replica is down and then the cluster is down
=== REDIS BUG REPORT START: Cut & paste starting from here ===
1491639:M 15 Jan 2025 13:20:24.219 # Redis 7.2.0 crashed by signal: 11, si_code: 128
1491639:M 15 Jan 2025 13:20:24.219 # Accessing address: (nil)
1491639:M 15 Jan 2025 13:20:24.219 # Crashed running the instruction at: 0x7ff8453517a8
here is the crash report:
1491639:M 15 Jan 2025 13:20:24.083 . --- Processing packet of type ping, 2616 bytes
1491639:M 15 Jan 2025 13:20:24.083 . ping packet received: 9a16bdd59195961a231c528735b5202750da84f0
1491639:M 15 Jan 2025 13:20:24.083 . GOSSIP 031053b9e2e5e730788080a732624d26c3ce97d5 10.0.0.130:6379@16379 master
1491639:M 15 Jan 2025 13:20:24.083 . GOSSIP f8166b0f549bb584dfacdb0383a8fcf2873e04be 10.0.0.106:6379@16379 master
1491639:M 15 Jan 2025 13:20:24.083 . GOSSIP 4740f438b06a4734437782f940d07b3bfb877472 10.0.0.102:6379@16379 master
1491639:M 15 Jan 2025 13:20:24.087 . --- Processing packet of type ping, 2616 bytes
1491639:M 15 Jan 2025 13:20:24.087 . ping packet received: c44d5386b2c0eaa0f7c0d2dba4b04fd26ef5fe9b
1491639:M 15 Jan 2025 13:20:24.087 . GOSSIP f8166b0f549bb584dfacdb0383a8fcf2873e04be 10.0.0.106:6379@16379 master
1491639:M 15 Jan 2025 13:20:24.087 . GOSSIP 4740f438b06a4734437782f940d07b3bfb877472 10.0.0.102:6379@16379 master
1491639:M 15 Jan 2025 13:20:24.087 . GOSSIP 8e9d44176cb5157c49e7aa10ba89b0524f55315e 10.0.0.130:6380@16380 slave
1491639:M 15 Jan 2025 13:20:24.100 . --- Processing packet of type ping, 2616 bytes
1491639:M 15 Jan 2025 13:20:24.100 . ping packet received: 4740f438b06a4734437782f940d07b3bfb877472
1491639:M 15 Jan 2025 13:20:24.100 . GOSSIP 8e9d44176cb5157c49e7aa10ba89b0524f55315e 10.0.0.130:6380@16380 slave
1491639:M 15 Jan 2025 13:20:24.100 . GOSSIP 9a16bdd59195961a231c528735b5202750da84f0 10.0.0.102:6380@16380 slave
1491639:M 15 Jan 2025 13:20:24.100 . GOSSIP c44d5386b2c0eaa0f7c0d2dba4b04fd26ef5fe9b 10.0.0.106:6380@16380 slave
1491639:M 15 Jan 2025 13:20:24.101 - Accepted 10.0.0.84:58329
1491639:M 15 Jan 2025 13:20:24.107 - Reading from client: error:00000000:lib(0):func(0):reason(0)
1491639:M 15 Jan 2025 13:20:24.203 - Accepted 10.0.0.84:58340
1491639:M 15 Jan 2025 13:20:24.209 - Reading from client: error:00000000:lib(0):func(0):reason(0)
=== REDIS BUG REPORT START: Cut & paste starting from here ===
1491639:M 15 Jan 2025 13:20:24.219 # Redis 7.2.0 crashed by signal: 11, si_code: 128
1491639:M 15 Jan 2025 13:20:24.219 # Accessing address: (nil)
1491639:M 15 Jan 2025 13:20:24.219 # Crashed running the instruction at: 0x7ff8453517a8
------ STACK TRACE ------
EIP:
./../../settings/modules/redisearch.so(+0x1a87a8)[0x7ff8453517a8]
Backtrace:
/lib64/libpthread.so.0(+0x12cf0)[0x7ff853848cf0]
./../../settings/modules/redisearch.so(+0x1a87a8)[0x7ff8453517a8]
./../../settings/modules/redisearch.so(IR_Read+0x48)[0x7ff845352218]
./../../settings/modules/redisearch.so(IR_SkipTo+0x1d3)[0x7ff8453524b3]
./../../settings/modules/redisearch.so(+0x19f091)[0x7ff845348091]
./../../settings/modules/redisearch.so(+0x1cd8b7)[0x7ff8453768b7]
./../../settings/modules/redisearch.so(+0x1ce061)[0x7ff845377061]
./../../settings/modules/redisearch.so(+0x1ce12e)[0x7ff84537712e]
./../../settings/modules/redisearch.so(+0x1cd7e4)[0x7ff8453767e4]
./../../settings/modules/redisearch.so(startPipeline+0x27)[0x7ff845310457]
./../../settings/modules/redisearch.so(sendChunk+0x12d)[0x7ff8453108fd]
./../../settings/modules/redisearch.so(AREQ_Execute+0x26)[0x7ff845311476]
./../../settings/modules/redisearch.so(+0x168c03)[0x7ff845311c03]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](RedisModuleCommandDispatcher+0x59)[0x5596702f16f9]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](call+0xfe)[0x55967025e04e]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](processCommand+0x92a)[0x55967025ef8a]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](processInputBuffer+0xeb)[0x559670276d6b]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](readQueryFromClient+0x3d1)[0x559670278121]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](+0xc8376)[0x559670212376]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](+0xcee57)[0x559670218e57]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](aeProcessEvents+0x321)[0x5596701d5761]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](aeMain+0x1d)[0x5596701d5b4d]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](main+0x364)[0x5596701cb164]
/lib64/libc.so.6(__libc_start_main+0xe5)[0x7ff8534abd85]
/root/Redis-Cluster/src/redis-server *:6379 [cluster](_start+0x2a)[0x5596701cb82a]
------ REGISTERS ------
1491639:M 15 Jan 2025 13:20:24.221 #
RAX:00007ff852e3a648 RBX:00007ff852e3a640
RCX:0000000000000000 RDX:33315f3730335f31
RDI:00007ff852e3a680 RSI:00007ff8309f9780
RBP:00007ffc95037778 RSP:00007ffc950376d0
R8 :0000000000000001 R9 :00007ff83c4a38d0
R10:00007ff83c4a3910 R11:00007ff82d5c1130
R12:00007ff8453514f0 R13:00007ff852e3a648
R14:00007ff852e3a680 R15:0000000000000000
RIP:00007ff8453517a8 EFL:0000000000010202
CSGSFS:002b000000000033
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376df) -> 00007ffc950377d8
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376de) -> 0000000000000000
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376dd) -> 0000000000000000
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376dc) -> 00007ff844142d00
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376db) -> 00007ff8453524b3
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376da) -> 00007ffc95037778
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376d9) -> 0000000000000066
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376d8) -> 00007ff852e3a640
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376d7) -> 00007ff845352218
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376d6) -> 00007ff852e3a680
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376d5) -> 00007ff8453514f0
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376d4) -> 00007ffc95037778
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376d3) -> 00007ff852e3a640
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376d2) -> 00007ff81906f638
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376d1) -> 000000000000eefc
1491639:M 15 Jan 2025 13:20:24.221 # (00007ffc950376d0) -> 0000000000000000
------ INFO OUTPUT ------
# Server
redis_version:7.2.0
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:3fda8312849c92db
redis_mode:cluster
os:Linux 4.18.0-513.18.1.el8_9.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:7.5.0
process_id:1491639
process_supervised:no
run_id:c890a3e200fd5ceb239df444881998a74157ed69
tcp_port:6379
server_time_usec:1736936424219400
uptime_in_seconds:76964
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:8883176
executable:/root/Redis-Cluster/src/redis-server
config_file:/root/Redis-Cluster/settings/config/master-TLS.conf
io_threads_active:0
listener2:name=tls,bind=*,bind=-::*,port=6379
# Clients
connected_clients:15
cluster_connections:10
maxclients:10000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:327704
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
# Memory
used_memory:614168808
used_memory_human:585.72M
used_memory_rss:645353472
used_memory_rss_human:615.46M
used_memory_peak:635051536
used_memory_peak_human:605.63M
used_memory_peak_perc:96.71%
used_memory_overhead:6516908
used_memory_startup:1786696
used_memory_dataset:607651900
used_memory_dataset_perc:99.23%
allocator_allocated:615257808
allocator_active:618606592
allocator_resident:630448128
total_system_memory:9921908736
total_system_memory_human:9.24G
used_memory_lua:36864
used_memory_vm_eval:36864
used_memory_lua_human:36.00K
used_memory_scripts_eval:912
number_of_cached_scripts:2
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:69632
used_memory_vm_total_human:68.00K
used_memory_functions:184
used_memory_scripts:1096
used_memory_scripts_human:1.07K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.01
allocator_frag_bytes:3348784
allocator_rss_ratio:1.02
allocator_rss_bytes:11841536
rss_overhead_ratio:1.02
rss_overhead_bytes:14905344
mem_fragmentation_ratio:1.05
mem_fragmentation_bytes:31135664
mem_not_counted_for_evict:16544
mem_replication_backlog:1048580
mem_total_replication_buffers:1066208
mem_clients_slaves:17632
mem_clients_normal:136416
mem_cluster_links:10720
mem_aof_buffer:3072
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:4683
rdb_bgsave_in_progress:0
rdb_last_save_time:1736936411
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:1
rdb_current_bgsave_time_sec:-1
rdb_saves:69
rdb_last_cow_size:28803072
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:53869
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:35
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:22237184
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:147069366
aof_base_size:80165363
aof_pending_rewrite:0
aof_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:458608
total_commands_processed:9312251
instantaneous_ops_per_sec:1073
total_net_input_bytes:4313035623
total_net_output_bytes:94773947347
total_net_repl_input_bytes:0
total_net_repl_output_bytes:2290293363
instantaneous_input_kbps:823.33
instantaneous_output_kbps:13673.72
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:521.86
rejected_connections:0
sync_full:1
sync_partial_ok:1
sync_partial_err:1
expired_keys:2
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:504
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:4819921
keyspace_misses:377406
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:9831
total_forks:244
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:3470240
dump_payload_sanitizations:0
total_reads_processed:9600478
total_writes_processed:10865468
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:2371
reply_buffer_expands:4284
eventloop_cycles:12487895
eventloop_duration_sum:1278826202
eventloop_duration_cmd_sum:330342464
instantaneous_eventloop_cycles_per_sec:1078
instantaneous_eventloop_duration_usec:186
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.102,port=6380,state=online,offset=2203991751,lag=1
master_failover_state:no-failover
master_replid:1de3ccb0cd2fd96e613fa362adbe6db1a587bdad
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:2204117915
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2203064083
repl_backlog_histlen:1053833
# CPU
used_cpu_sys:322.315323
used_cpu_user:989.508562
used_cpu_sys_children:6.197829
used_cpu_user_children:42.640211
used_cpu_sys_main_thread:317.973881
used_cpu_user_main_thread:988.610891
# Modules
module:name=ReJSON,ver=999999,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=search,ver=20821,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
# Commandstats
cmdstat_flushall:calls=7,usec=725232,usec_per_call=103604.57,rejected_calls=0,failed_calls=0
cmdstat_evalsha:calls=742010,usec=16651355,usec_per_call=22.44,rejected_calls=0,failed_calls=2
cmdstat_json.objlen:calls=12,usec=104,usec_per_call=8.67,rejected_calls=0,failed_calls=0
cmdstat_eval:calls=2,usec=169,usec_per_call=84.50,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=458608,usec=1458827,usec_per_call=3.18,rejected_calls=0,failed_calls=0
cmdstat_json.debug:calls=7,usec=138,usec_per_call=19.71,rejected_calls=0,failed_calls=0
cmdstat_type:calls=106,usec=248,usec_per_call=2.34,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=0,usec=0,usec_per_call=0.00,rejected_calls=2,failed_calls=0
cmdstat_memory|usage:calls=88,usec=444,usec_per_call=5.05,rejected_calls=0,failed_calls=0
cmdstat_FT._LIST:calls=42,usec=329,usec_per_call=7.83,rejected_calls=0,failed_calls=0
cmdstat_json.objkeys:calls=5,usec=141,usec_per_call=28.20,rejected_calls=0,failed_calls=0
cmdstat_info:calls=742,usec=47481,usec_per_call=63.99,rejected_calls=0,failed_calls=0
cmdstat_json.arrlen:calls=15,usec=166,usec_per_call=11.07,rejected_calls=0,failed_calls=0
cmdstat_client|setname:calls=458552,usec=739938,usec_per_call=1.61,rejected_calls=0,failed_calls=0
cmdstat_client|list:calls=6,usec=185,usec_per_call=30.83,rejected_calls=0,failed_calls=0
cmdstat_module|list:calls=9,usec=32,usec_per_call=3.56,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=30,usec=117,usec_per_call=3.90,rejected_calls=0,failed_calls=0
cmdstat_json.type:calls=348,usec=3301,usec_per_call=9.49,rejected_calls=0,failed_calls=0
cmdstat_exists:calls=371063,usec=498560,usec_per_call=1.34,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=62,usec=115653,usec_per_call=1865.37,rejected_calls=0,failed_calls=0
cmdstat_get:calls=370947,usec=601328,usec_per_call=1.62,rejected_calls=0,failed_calls=0
cmdstat_FT.SEARCH:calls=192536,usec=13488495,usec_per_call=70.06,rejected_calls=0,failed_calls=1
cmdstat_psync:calls=2,usec=6274,usec_per_call=3137.00,rejected_calls=0,failed_calls=0
cmdstat_json.set:calls=829153,usec=53325626,usec_per_call=64.31,rejected_calls=520459,failed_calls=0
cmdstat_FT.CREATE:calls=9,usec=1708,usec_per_call=189.78,rejected_calls=0,failed_calls=0
cmdstat_set:calls=370907,usec=2737255,usec_per_call=7.38,rejected_calls=0,failed_calls=0
cmdstat_ttl:calls=88,usec=125,usec_per_call=1.42,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=696,usec=796,usec_per_call=1.14,rejected_calls=0,failed_calls=0
cmdstat_json.del:calls=8993,usec=187109,usec_per_call=20.81,rejected_calls=24,failed_calls=0
cmdstat_unwatch:calls=2728,usec=2198,usec_per_call=0.81,rejected_calls=0,failed_calls=0
cmdstat_replconf:calls=76457,usec=90886,usec_per_call=1.19,rejected_calls=0,failed_calls=0
cmdstat_del:calls=370903,usec=1624170,usec_per_call=4.38,rejected_calls=0,failed_calls=0
cmdstat_FT.AGGREGATE:calls=197676,usec=22328138,usec_per_call=112.95,rejected_calls=0,failed_calls=0
cmdstat_json.get:calls=4400867,usec=212268481,usec_per_call=48.23,rejected_calls=2929234,failed_calls=20518
cmdstat_cluster|shards:calls=4,usec=175,usec_per_call=43.75,rejected_calls=0,failed_calls=0
cmdstat_cluster|slots:calls=458517,usec=8896724,usec_per_call=19.40,rejected_calls=0,failed_calls=0
cmdstat_cluster|info:calls=39,usec=990,usec_per_call=25.38,rejected_calls=0,failed_calls=0
cmdstat_cluster|nodes:calls=15,usec=889,usec_per_call=59.27,rejected_calls=0,failed_calls=0
# Errorstats
errorstat_ERR:count=20518
errorstat_MOVED:count=3449717
errorstat_NOAUTH:count=2
errorstat_NOSCRIPT:count=2
errorstat_TransactionIndex_:count=1
# Latencystats
latency_percentiles_usec_flushall:p50=5865.471,p99=373293.055,p99.9=373293.055
latency_percentiles_usec_evalsha:p50=21.119,p99=72.191,p99.9=116.223
latency_percentiles_usec_json.objlen:p50=8.031,p99=12.031,p99.9=12.031
latency_percentiles_usec_eval:p50=77.311,p99=92.159,p99.9=92.159
latency_percentiles_usec_auth:p50=3.007,p99=10.047,p99.9=27.007
latency_percentiles_usec_json.debug:p50=21.119,p99=30.079,p99.9=30.079
latency_percentiles_usec_type:p50=2.007,p99=10.047,p99.9=18.047
latency_percentiles_usec_memory|usage:p50=3.007,p99=14.015,p99.9=14.015
latency_percentiles_usec_FT._LIST:p50=7.007,p99=19.071,p99.9=19.071
latency_percentiles_usec_json.objkeys:p50=25.087,p99=54.015,p99.9=54.015
latency_percentiles_usec_info:p50=60.159,p99=149.503,p99.9=187.391
latency_percentiles_usec_json.arrlen:p50=11.007,p99=20.095,p99.9=20.095
latency_percentiles_usec_client|setname:p50=2.007,p99=4.015,p99.9=17.023
latency_percentiles_usec_client|list:p50=23.039,p99=64.255,p99.9=64.255
latency_percentiles_usec_module|list:p50=4.015,p99=6.015,p99.9=6.015
latency_percentiles_usec_config|get:p50=3.007,p99=19.071,p99.9=19.071
latency_percentiles_usec_json.type:p50=8.031,p99=28.031,p99.9=37.119
latency_percentiles_usec_exists:p50=1.003,p99=4.015,p99.9=20.095
latency_percentiles_usec_scan:p50=2441.215,p99=3784.703,p99.9=3964.927
latency_percentiles_usec_get:p50=1.003,p99=5.023,p99.9=24.063
latency_percentiles_usec_FT.SEARCH:p50=57.087,p99=286.719,p99.9=464.895
latency_percentiles_usec_psync:p50=63.231,p99=6225.919,p99.9=6225.919
latency_percentiles_usec_json.set:p50=52.223,p99=224.255,p99.9=319.487
latency_percentiles_usec_FT.CREATE:p50=186.367,p99=234.495,p99.9=234.495
latency_percentiles_usec_set:p50=7.007,p99=27.007,p99.9=53.247
latency_percentiles_usec_ttl:p50=1.003,p99=4.015,p99.9=4.015
latency_percentiles_usec_dbsize:p50=1.003,p99=9.023,p99.9=31.103
latency_percentiles_usec_json.del:p50=19.071,p99=61.183,p99.9=97.279
latency_percentiles_usec_unwatch:p50=1.003,p99=2.007,p99.9=10.047
latency_percentiles_usec_replconf:p50=1.003,p99=3.007,p99.9=17.023
latency_percentiles_usec_del:p50=4.015,p99=17.023,p99.9=41.215
latency_percentiles_usec_FT.AGGREGATE:p50=52.223,p99=798.719,p99.9=1327.103
latency_percentiles_usec_json.get:p50=15.039,p99=790.527,p99.9=1376.255
latency_percentiles_usec_cluster|shards:p50=28.031,p99=91.135,p99.9=91.135
latency_percentiles_usec_cluster|slots:p50=16.063,p99=54.015,p99.9=81.407
latency_percentiles_usec_cluster|info:p50=22.015,p99=54.015,p99.9=54.015
latency_percentiles_usec_cluster|nodes:p50=63.231,p99=94.207,p99.9=94.207
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=53362,expires=4,avg_ttl=9950
# Cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:25
cluster_my_epoch:24
cluster_stats_messages_ping_sent:139312
cluster_stats_messages_pong_sent:139297
cluster_stats_messages_fail_sent:4
cluster_stats_messages_auth-req_sent:5
cluster_stats_messages_update_sent:1
cluster_stats_messages_sent:278619
cluster_stats_messages_ping_received:139293
cluster_stats_messages_pong_received:139307
cluster_stats_messages_auth-req_received:1
cluster_stats_messages_auth-ack_received:2
cluster_stats_messages_update_received:4
cluster_stats_messages_received:278607
total_cluster_links_buffer_limit_exceeded:0
------ CLUSTER NODES OUTPUT ------
9a16bdd59195961a231c528735b5202750da84f0 10.0.0.102:0@16380,,tls-port=6380,shard-id=daf1289f347bcf539257a321996c485f5eedd424 slave 031053b9e2e5e730788080a732624d26c3ce97d5 0 1736936422696 24 connected
8e9d44176cb5157c49e7aa10ba89b0524f55315e 10.0.0.130:0@16380,,tls-port=6380,shard-id=525e61fb4b41f51dacf8b95965133303c11f93a4 slave f8166b0f549bb584dfacdb0383a8fcf2873e04be 0 1736936423501 13 connected
f8166b0f549bb584dfacdb0383a8fcf2873e04be 10.0.0.106:0@16379,,tls-port=6379,shard-id=525e61fb4b41f51dacf8b95965133303c11f93a4 master - 0 1736936422598 13 connected 0-5460
031053b9e2e5e730788080a732624d26c3ce97d5 10.0.0.130:0@16379,,tls-port=6379,shard-id=daf1289f347bcf539257a321996c485f5eedd424 myself,master - 0 1736936421000 24 connected 5461-10922
c44d5386b2c0eaa0f7c0d2dba4b04fd26ef5fe9b 10.0.0.106:0@16380,,tls-port=6380,shard-id=76622b5d4b5c3e22af79b3e8291eacce07e6697d slave 4740f438b06a4734437782f940d07b3bfb877472 0 1736936423000 19 connected
4740f438b06a4734437782f940d07b3bfb877472 10.0.0.102:0@16379,,tls-port=6379,shard-id=76622b5d4b5c3e22af79b3e8291eacce07e6697d master - 0 1736936423703 19 connected 10923-16383
------ CLIENT LIST OUTPUT ------
id=257138 addr=10.0.0.174:64699 laddr=10.0.0.130:6379 fd=36 name= age=5198 idle=5193 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=json.get user=default redir=-1 resp=2 lib-name= lib-ver=
id=254984 addr=10.0.0.84:58896 laddr=10.0.0.130:6379 fd=27 name= age=5276 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=65530 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=82816 events=r cmd=evalsha user=default redir=-1 resp=2 lib-name= lib-ver=
id=242074 addr=10.0.0.131:55642 laddr=10.0.0.130:6379 fd=34 name= age=5829 idle=5827 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=json.get user=default redir=-1 resp=2 lib-name= lib-ver=
id=253877 addr=10.0.0.84:55797 laddr=10.0.0.130:6379 fd=28 name= age=5307 idle=5307 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=cluster|info user=default redir=-1 resp=2 lib-name= lib-ver=
id=170361 addr=10.0.0.102:41314 laddr=10.0.0.130:6379 fd=12 name= age=43457 idle=0 flags=S db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=1 omem=20504 tot-mem=42904 events=r cmd=replconf user=default redir=-1 resp=2 lib-name= lib-ver=
id=255133 addr=10.0.0.84:59655 laddr=10.0.0.130:6379 fd=29 name= age=5263 idle=5262 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=json.get user=default redir=-1 resp=2 lib-name= lib-ver=
id=257446 addr=10.0.0.174:65340 laddr=10.0.0.130:6379 fd=37 name= age=5193 idle=5192 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=json.get user=default redir=-1 resp=2 lib-name= lib-ver=
id=170395 addr=10.0.0.131:59658 laddr=10.0.0.130:6379 fd=32 name= age=9308 idle=9308 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=cluster|info user=default redir=-1 resp=2 lib-name= lib-ver=
id=403058 addr=10.0.0.174:56177 laddr=10.0.0.130:6379 fd=38 name= age=1600 idle=1600 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=cluster|info user=default redir=-1 resp=2 lib-name= lib-ver=
id=256930 addr=10.0.0.174:64662 laddr=10.0.0.130:6379 fd=35 name= age=5204 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=16384 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=json.get user=default redir=-1 resp=2 lib-name= lib-ver=
id=238396 addr=10.0.0.131:54417 laddr=10.0.0.130:6379 fd=26 name= age=6037 idle=6037 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=flushall user=default redir=-1 resp=2 lib-name= lib-ver=
id=238402 addr=10.0.0.131:54503 laddr=10.0.0.130:6379 fd=31 name= age=5952 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=245 qbuf-free=16529 argv-mem=147 multi-mem=0 rbs=16384 rbp=16384 obl=13720 oll=0 omem=0 tot-mem=37931 events=r cmd=FT.SEARCH user=default redir=-1 resp=2 lib-name= lib-ver=
id=170377 addr=10.0.0.205:54907 laddr=10.0.0.130:6379 fd=23 name=redisinsight-common-0dc00587-70f3-4e54-8a5a-1698bbbda702---1-1- age=9410 idle=2760 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=dbsize user=default redir=-1 resp=2 lib-name= lib-ver=
id=254987 addr=10.0.0.84:58939 laddr=10.0.0.130:6379 fd=25 name= age=5272 idle=5268 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=json.get user=default redir=-1 resp=2 lib-name= lib-ver=
id=170380 addr=10.0.0.205:54945 laddr=10.0.0.130:6379 fd=24 name=redisinsight-browser-0dc00587-70f3-4e54-8a5a-1698bbbda702---1-1- age=9394 idle=2777 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=type user=default redir=-1 resp=2 lib-name= lib-ver=
id=238405 addr=10.0.0.131:54520 laddr=10.0.0.130:6379 fd=33 name= age=5952 idle=5826 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=json.get user=default redir=-1 resp=2 lib-name= lib-ver=
------ CURRENT CLIENT INFO ------
id=238402 addr=10.0.0.131:54503 laddr=10.0.0.130:6379 fd=31 name= age=5952 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=245 qbuf-free=16529 argv-mem=147 multi-mem=0 rbs=16384 rbp=16384 obl=13720 oll=0 omem=0 tot-mem=37931 events=r cmd=FT.SEARCH user=default redir=-1 resp=2 lib-name= lib-ver=
argc: '3'
argv[0]: '"FT.SEARCH"'
argv[1]: '"NextTransactionsIndex"'
argv[2]: '"@queueBranch_ID:[61180 61180] @_priorityTime_Num:[1736920805000 1736920805000] @state:[0 1] @_servingCounters:{61197}"'
------ EXECUTING CLIENT INFO ------
id=238402 addr=10.0.0.131:54503 laddr=10.0.0.130:6379 fd=31 name= age=5952 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=245 qbuf-free=16529 argv-mem=147 multi-mem=0 rbs=16384 rbp=16384 obl=13720 oll=0 omem=0 tot-mem=37931 events=r cmd=FT.SEARCH user=default redir=-1 resp=2 lib-name= lib-ver=
argc: '3'
argv[0]: '"FT.SEARCH"'
argv[1]: '"NextTransactionsIndex"'
argv[2]: '"@queueBranch_ID:[61180 61180] @_priorityTime_Num:[1736920805000 1736920805000] @state:[0 1] @_servingCounters:{61197}"'
------ MODULES INFO OUTPUT ------
# ReJSON_trace
ReJSON_backtrace: 0: redis_module::basic_info_command_handler
1: rejson::__info_func
2: modulesCollectInfo
at /home/redis1/redis-7.2.0/src/module.c:10293
3: logModulesInfo
at /home/redis1/redis-7.2.0/src/debug.c:1900
printCrashReport
at /home/redis1/redis-7.2.0/src/debug.c:2183
4: sigsegvHandler
at /home/redis1/redis-7.2.0/src/debug.c:2164
5: <unknown>
6: Buffer_Read
at /home/omar/redisearch-src/RediSearch/src/buffer.h:66:3
readNumeric
at /home/omar/redisearch-src/RediSearch/src/inverted_index.c:626:3
7: IR_Read
at /home/omar/redisearch-src/RediSearch/src/inverted_index.c:998:14
8: IR_SkipTo
at /home/omar/redisearch-src/RediSearch/src/inverted_index.c:1130:35
9: II_ReadSorted
at /home/omar/redisearch-src/RediSearch/src/index.c:1085:16
10: rpidxNext
at /home/omar/redisearch-src/RediSearch/src/result_processor.c:110:10
11: rpscoreNext
at /home/omar/redisearch-src/RediSearch/src/result_processor.c:221:10
12: rpsortNext_innerLoop
at /home/omar/redisearch-src/RediSearch/src/result_processor.c:395:12
rpsortNext_Accum
at /home/omar/redisearch-src/RediSearch/src/result_processor.c:445:16
13: rploaderNext
at /home/omar/redisearch-src/RediSearch/src/result_processor.c:649:12
14: startPipeline
at /home/omar/redisearch-src/RediSearch/src/aggregate/aggregate_exec.c:361:13
15: sendChunk_Resp2
at /home/omar/redisearch-src/RediSearch/src/aggregate/aggregate_exec.c:428:5
sendChunk
at /home/omar/redisearch-src/RediSearch/src/aggregate/aggregate_exec.c:678:5
16: AREQ_Execute
at /home/omar/redisearch-src/RediSearch/src/aggregate/aggregate_exec.c:688:3
17: execCommandCommon
at /home/omar/redisearch-src/RediSearch/src/aggregate/aggregate_exec.c:973:7
18: RedisModuleCommandDispatcher
at /home/redis1/redis-7.2.0/src/module.c:917
19: call
at /home/redis1/redis-7.2.0/src/server.c:3519
20: processCommand
at /home/redis1/redis-7.2.0/src/server.c:4160
21: processCommandAndResetClient
at /home/redis1/redis-7.2.0/src/networking.c:2466
processInputBuffer
at /home/redis1/redis-7.2.0/src/networking.c:2574
22: readQueryFromClient
at /home/redis1/redis-7.2.0/src/networking.c:2713
23: callHandler
at /home/redis1/redis-7.2.0/src/connhelpers.h:79
24: tlsHandleEvent
at /home/redis1/redis-7.2.0/src/tls.c:720
25: aeProcessEvents
at /home/redis1/redis-7.2.0/src/ae.c:436
26: aeMain
at /home/redis1/redis-7.2.0/src/ae.c:496
27: main
at /home/redis1/redis-7.2.0/src/server.c:7360
28: __libc_start_main
29: _start
# search_version
search_version:2.8.21
search_redis_version:7.2.0 - oss
# search_index
search_number_of_indexes:2
search_number_of_active_indexes:1
search_number_of_active_indexes_running_queries:1
search_number_of_active_indexes_indexing:0
search_total_active_write_threads:0
# search_fields_statistics
search_fields_numeric:Numeric=10,Sortable=2,IndexErrors=0
search_fields_tag:Tag=4,IndexErrors=0
# search_memory
search_used_memory_indexes:24338864
search_used_memory_indexes_human:23.211349487304688
search_smallest_memory_index:8878443
search_smallest_memory_index_human:8.4671430587768555
search_largest_memory_index:15460421
search_largest_memory_index_human:14.744206428527832
search_total_indexing_time:4587.466796875
search_used_memory_vector_index:0
# search_cursors
search_global_idle:0
search_global_total:0
# search_gc
search_bytes_collected:1502787
search_total_cycles:21
search_total_ms_run:1251
search_total_docs_not_collected_by_gc:720
search_marked_deleted_vectors:0
# search_queries
search_total_queries_processed:390211
search_total_query_commands:390211
search_total_query_execution_time_ms:33553
search_total_active_queries:1
# search_warnings_and_errors
search_errors_indexing_failures:0
search_errors_for_index_with_max_failures:0
# search_dialect_statistics
search_dialect_1:1
search_dialect_2:0
search_dialect_3:0
search_dialect_4:0
# search_runtime_configurations
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:1000000
search_max_aggregate_results:2147483648
search_gc_scan_size:100
search_min_phonetic_term_length:3
------ CONFIG DEBUG OUTPUT ------
activedefrag no
repl-diskless-load disabled
lazyfree-lazy-server-del no
io-threads-do-reads no
lazyfree-lazy-eviction no
sanitize-dump-payload no
lazyfree-lazy-user-del no
proto-max-bulk-len 512mb
slave-read-only yes
lazyfree-lazy-user-flush no
io-threads 1
list-compress-depth 0
repl-diskless-sync no
client-query-buffer-limit 1gb
lazyfree-lazy-expire no
replica-read-only yes
------ FAST MEMORY TEST ------
1491639:M 15 Jan 2025 13:20:24.236 # Bio worker thread #0 terminated
1491639:M 15 Jan 2025 13:20:24.236 # Bio worker thread #1 terminated
1491639:M 15 Jan 2025 13:20:24.236 # Bio worker thread #2 terminated
*** Preparing to test memory region 559670689000 (2273280 bytes)
*** Preparing to test memory region 559671bae000 (2179072 bytes)
*** Preparing to test memory region 7ff806eff000 (9437184 bytes)
*** Preparing to test memory region 7ff807800000 (8388608 bytes)
*** Preparing to test memory region 7ff808000000 (135168 bytes)
*** Preparing to test memory region 7ff80c000000 (6291456 bytes)
*** Preparing to test memory region 7ff80c800000 (8388608 bytes)
*** Preparing to test memory region 7ff80d000000 (2097152 bytes)
*** Preparing to test memory region 7ff80d400000 (8388608 bytes)
*** Preparing to test memory region 7ff80dc00000 (2097152 bytes)
*** Preparing to test memory region 7ff80df00000 (705826816 bytes)
*** Preparing to test memory region 7ff83c180000 (19398656 bytes)
*** Preparing to test memory region 7ff83d4fb000 (3145728 bytes)
*** Preparing to test memory region 7ff83d7fc000 (8388608 bytes)
*** Preparing to test memory region 7ff83dffd000 (8388608 bytes)
*** Preparing to test memory region 7ff83e7fe000 (8388608 bytes)
*** Preparing to test memory region 7ff83efff000 (8388608 bytes)
*** Preparing to test memory region 7ff83f800000 (8388608 bytes)
*** Preparing to test memory region 7ff840000000 (135168 bytes)
*** Preparing to test memory region 7ff844089000 (2097152 bytes)
*** Preparing to test memory region 7ff844791000 (8388608 bytes)
*** Preparing to test memory region 7ff8457c5000 (20480 bytes)
*** Preparing to test memory region 7ff8457ca000 (2621440 bytes)
*** Preparing to test memory region 7ff852a00000 (8388608 bytes)
*** Preparing to test memory region 7ff853470000 (4096 bytes)
*** Preparing to test memory region 7ff853832000 (16384 bytes)
*** Preparing to test memory region 7ff853a52000 (16384 bytes)
*** Preparing to test memory region 7ff853f3c000 (20480 bytes)
*** Preparing to test memory region 7ff8541d4000 (4096 bytes)
*** Preparing to test memory region 7ff854b7e000 (32768 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: ./../../settings/modules/redisearch.so (base 0x7ff8451a9000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/redis/redis/issues
If a Redis module was involved, please open in the module's repo instead.
Suspect RAM error? Use redis-server --test-memory to verify it.
Some other issues could be detected by redis-server --check-system
Reactions are currently unavailable