Skip to content

[CRASH] ZINTER betwen SET and ZSET crashes #11578

@hbina

Description

@hbina

Crash report

Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.

=== REDIS BUG REPORT START: Cut & paste starting from here ===
51320:M 05 Dec 2022 08:10:35.109 # ------------------------------------------------
51320:M 05 Dec 2022 08:10:35.109 # !!! Software Failure. Press left mouse button to continue
51320:M 05 Dec 2022 08:10:35.109 # Guru Meditation: Unknown set encoding #t_zset.c:2279

------ STACK TRACE ------

Backtrace:
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(+0xee79f)[0x55555564279f]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(zunionInterDiffGenericCommand+0xbda)[0x55555564347a]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(call+0xe2)[0x5555555dcae2]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(processCommand+0xb10)[0x5555555de020]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(processInputBuffer+0x107)[0x5555555febd7]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(readQueryFromClient+0x348)[0x5555555ff138]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(+0x1ade9c)[0x555555701e9c]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(aeMain+0xf1)[0x5555555d2ab1]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(main+0x3d4)[0x5555555c8254]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7ffff7c29d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7ffff7c29e40]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(_start+0x25)[0x5555555c89b5]

------ INFO OUTPUT ------
# Server
redis_version:255.255.255
redis_git_sha1:61c85a2b
redis_git_dirty:0
redis_build_id:ce6e622df897e510
redis_mode:standalone
os:Linux 6.0.9-060009-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.3.0
process_id:51320
process_supervised:no
run_id:bfb2eb674e0616154f7e79c17e39fbfc7c5b340f
tcp_port:6379
server_time_usec:1670199035109087
uptime_in_seconds:117
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:9254651
executable:/home/hbina/git/redis/src/redis-server
config_file:/home/hbina/git/redis/redis.conf
io_threads_active:0
listener0:name=tcp,bind=127.0.0.1,bind=-::1,port=6379

# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:990144
used_memory_human:966.94K
used_memory_rss:7516160
used_memory_rss_human:7.17M
used_memory_peak:1112776
used_memory_peak_human:1.06M
used_memory_peak_perc:88.98%
used_memory_overhead:867368
used_memory_startup:865272
used_memory_dataset:122776
used_memory_dataset_perc:98.32%
allocator_allocated:1526896
allocator_active:1904640
allocator_resident:6438912
total_system_memory:33047990272
total_system_memory_human:30.78G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.25
allocator_frag_bytes:377744
allocator_rss_ratio:3.38
allocator_rss_bytes:4534272
rss_overhead_ratio:1.17
rss_overhead_bytes:1077248
mem_fragmentation_ratio:7.61
mem_fragmentation_bytes:6528912
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:1800
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:7
rdb_bgsave_in_progress:0
rdb_last_save_time:1670199029
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:1
total_commands_processed:4
instantaneous_ops_per_sec:0
total_net_input_bytes:212
total_net_output_bytes:200352
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:3
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:2
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:5
total_writes_processed:6
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:1
reply_buffer_expands:0
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:acfdf4ca028008dfe4493fd8694c70987c2b735d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.132816
used_cpu_user:0.070558
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.132267
used_cpu_user_main_thread:0.070267

# Modules

# Commandstats
cmdstat_sadd:calls=1,usec=15,usec_per_call=15.00,rejected_calls=0,failed_calls=0
cmdstat_command|docs:calls=1,usec=1588,usec_per_call=1588.00,rejected_calls=0,failed_calls=0
cmdstat_zadd:calls=1,usec=41,usec_per_call=41.00,rejected_calls=0,failed_calls=0
cmdstat_flushall:calls=1,usec=19022,usec_per_call=19022.00,rejected_calls=0,failed_calls=0

# Errorstats

# Latencystats
latency_percentiles_usec_sadd:p50=15.039,p99=15.039,p99.9=15.039
latency_percentiles_usec_command|docs:p50=1589.247,p99=1589.247,p99.9=1589.247
latency_percentiles_usec_zadd:p50=41.215,p99=41.215,p99.9=41.215
latency_percentiles_usec_flushall:p50=19136.511,p99=19136.511,p99.9=19136.511

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=2,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=3 addr=127.0.0.1:56792 laddr=127.0.0.1:6379 fd=8 name= age=7 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=41 qbuf-free=20433 argv-mem=13 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22317 events=r cmd=zdiff user=default redir=-1 resp=2

------ CURRENT CLIENT INFO ------
id=3 addr=127.0.0.1:56792 laddr=127.0.0.1:6379 fd=8 name= age=7 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=41 qbuf-free=20433 argv-mem=13 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22317 events=r cmd=zdiff user=default redir=-1 resp=2
argv[0]: '"zdiff"'
argv[1]: '"2"'
argv[2]: '"key"'
argv[3]: '"key2"'

------ MODULES INFO OUTPUT ------

------ CONFIG DEBUG OUTPUT ------
client-query-buffer-limit 1gb
lazyfree-lazy-eviction no
io-threads-do-reads no
replica-read-only yes
lazyfree-lazy-server-del no
activedefrag no
list-compress-depth 0
sanitize-dump-payload no
repl-diskless-sync yes
lazyfree-lazy-expire no
proto-max-bulk-len 512mb
lazyfree-lazy-user-del no
slave-read-only yes
lazyfree-lazy-user-flush no
repl-diskless-load disabled
io-threads 1

------ FAST MEMORY TEST ------
51320:M 05 Dec 2022 08:10:35.116 # Bio thread for job type #0 terminated
51320:M 05 Dec 2022 08:10:35.117 # Bio thread for job type #1 terminated
51320:M 05 Dec 2022 08:10:35.117 # Bio thread for job type #2 terminated
*** Preparing to test memory region 5555558c2000 (2441216 bytes)
*** Preparing to test memory region 7ffff4b7c000 (2621440 bytes)
*** Preparing to test memory region 7ffff4dfd000 (8388608 bytes)
*** Preparing to test memory region 7ffff55fe000 (8388608 bytes)
*** Preparing to test memory region 7ffff5dff000 (8388608 bytes)
*** Preparing to test memory region 7ffff6600000 (8388608 bytes)
*** Preparing to test memory region 7ffff7400000 (8388608 bytes)
*** Preparing to test memory region 7ffff7e1b000 (53248 bytes)
*** Preparing to test memory region 7ffff7eb9000 (16384 bytes)
*** Preparing to test memory region 7ffff7fbb000 (8192 bytes)
.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           http://github.com/redis/redis/issues

  If a Redis module was involved, please open in the module's repo instead.

  Suspect RAM error? Use redis-server --test-memory to verify it.

  Some other issues could be detected by redis-server --check-system

Additional information

  1. Screenfetch
hbina@akarin ~> screenfetch -n
 hbina@akarin
 OS: Ubuntu 22.04 jammy
 Kernel: x86_64 Linux 6.0.9-060009-generic
 Uptime: 1h 24m
 Packages: 2535
 Shell: fish 3.3.1
 Resolution: 1920x1080
 DE: GNOME 41.7
 WM: Mutter
 WM Theme: Adwaita
 GTK Theme: Yaru-red-dark [GTK2/3]
 Icon Theme: Yaru-red
 Font: Ubuntu 11
 Disk: 290G / 436G (71%)
 CPU: AMD Ryzen 7 4800U with Radeon Graphics @ 16x 1.8GHz
 GPU: AMD/ATI
 RAM: 6676MiB / 31517MiB

Reproduction Steps (works every time!)

  1. Build from source (61c85a2)
  2. Run the following commands:
hbina@akarin ~/g/redis (unstable)> redis-cli -p 6379
127.0.0.1:6379> zadd key 2 a 7 b 9 d 12 c
(integer) 4
127.0.0.1:6379> sadd key2 a b c
(integer) 3
127.0.0.1:6379> zdiff 2 key key2
Error: Server closed the connection
(231.17s)
not connected>
  1. Boom

I tried to look into the source code and the only thing I can worked out is that key2 is being stored as a OBJ_ENCODING_LISTPACK but the diff algorithm doesn't consider this possibility and just panics. Adding the following simple change

else if (op->encoding == OBJ_ENCODING_LISTPACK) {
            zuiSdsFromValue(val);
            if (zzlFind(op->subject->ptr,val->ele,score) != NULL) {
                /* Score is already set by zzlFind. */
                return 1;
            } else {
                return 0;
            }
        }

does not fix the issue becase zzlFind will also crash. I tried to debug zzFind but debugger doesn't really work (lots of things get optimized out) and its honestly such a mess of code it'll take me a very long time to decipher it. I'd be very interested to know how you fix it!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions