Avoid performing IO on coverage when child exits due to signal handler by kei-nan · Pull Request #14072 · redis/redis

kei-nan · 2025-05-25T12:29:43Z

Compiled Redis with COVERAGE_TEST, while using the fork API encountered the following issue:

Forked process calls RedisModule_ExitFromChild - child process starts to report its COW while performing IO operations
Parent process terminates child process with RedisModule_KillForkChild
Child process signal handler gets called while an IO operation is called
exit() is called because COVERAGE_TEST was on during compilation.
exit() tries to perform more IO operations in its exit handlers.
process gets deadlocked

Backtrace snippet:

#0  futex_wait (private=0, expected=2, futex_word=0x7e1220000c50) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait_private (futex=0x7e1220000c50) at ./nptl/lowlevellock.c:34
#2  0x00007e1234696429 in __GI__IO_flush_all () at ./libio/genops.c:698
#3  0x00007e123469680d in _IO_cleanup () at ./libio/genops.c:843
#4  0x00007e1234647b74 in __run_exit_handlers (status=status@entry=255, listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:129
#5  0x00007e1234647bbe in __GI_exit (status=status@entry=255) at ./stdlib/exit.c:138
#6  0x00005ef753264e13 in exitFromChild (retcode=255) at /home/jonathan/CLionProjects/redis/src/server.c:263
#7  sigKillChildHandler (sig=<optimized out>) at /home/jonathan/CLionProjects/redis/src/server.c:6794
#8  <signal handler called>
#9  0x00007e1234685b94 in _IO_fgets (buf=buf@entry=0x7e122dafdd90 "KSM:", ' ' <repeats 19 times>, "0 kB\n", n=n@entry=1024, fp=fp@entry=0x7e1220000b70) at ./libio/iofgets.c:47
#10 0x00005ef75326c5e0 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>, __s=<optimized out>, __n=<optimized out>, __stream=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:200
#11 zmalloc_get_smap_bytes_by_field (field=0x5ef7534c42fd "Private_Dirty:", pid=<optimized out>) at /home/jonathan/CLionProjects/redis/src/zmalloc.c:928
#12 0x00005ef75338ab1f in zmalloc_get_private_dirty (pid=-1) at /home/jonathan/CLionProjects/redis/src/zmalloc.c:978
#13 sendChildInfoGeneric (info_type=CHILD_INFO_TYPE_MODULE_COW_SIZE, keys=0, progress=-1, pname=0x5ef7534c95b2 "Module fork") at /home/jonathan/CLionProjects/redis/src/childinfo.c:71
#14 0x00005ef75337962c in sendChildCowInfo (pname=0x5ef7534c95b2 "Module fork", info_type=CHILD_INFO_TYPE_MODULE_COW_SIZE) at /home/jonathan/CLionProjects/redis/src/server.c:6895
#15 RM_ExitFromChild (retcode=0) at /home/jonathan/CLionProjects/redis/src/module.c:11468

Change is to make the exit() _exit() calls conditional based on a parameter to exitFromChild function.
The signal handler should exit without io operations since it doesn't know its history.(If we were in the middle of IO operations before it was called)

snyk-io · 2025-05-25T12:29:56Z

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

src/childinfo.c

src/server.c

Co-authored-by: Yuan Wang <[email protected]>

src/server.c

ShooterIT · 2025-05-28T08:21:32Z

Since this bug happens only when compiling with COVERAGE_TEST, that is, make gcov. i think it is not mandatory to back port to 8.0 and older versions

oshadmi · 2025-07-03T12:54:47Z

Since this bug happens only when compiling with COVERAGE_TEST, that is, make gcov. i think it is not mandatory to back port to 8.0 and older versions

@ShooterIT In order to have code coverage for the child fork process, we also need this PR backported to 8.0.

redis#14072) Compiled Redis with COVERAGE_TEST, while using the fork API encountered the following issue: - Forked process calls `RedisModule_ExitFromChild` - child process starts to report its COW while performing IO operations - Parent process terminates child process with `RedisModule_KillForkChild` - Child process signal handler gets called while an IO operation is called - exit() is called because COVERAGE_TEST was on during compilation. - exit() tries to perform more IO operations in its exit handlers. - process gets deadlocked Backtrace snippet: ``` #0 futex_wait (private=0, expected=2, futex_word=0x7e1220000c50) at ../sysdeps/nptl/futex-internal.h:146 #1 __GI___lll_lock_wait_private (futex=0x7e1220000c50) at ./nptl/lowlevellock.c:34 #2 0x00007e1234696429 in __GI__IO_flush_all () at ./libio/genops.c:698 #3 0x00007e123469680d in _IO_cleanup () at ./libio/genops.c:843 #4 0x00007e1234647b74 in __run_exit_handlers (status=status@entry=255, listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:129 #5 0x00007e1234647bbe in __GI_exit (status=status@entry=255) at ./stdlib/exit.c:138 #6 0x00005ef753264e13 in exitFromChild (retcode=255) at /home/jonathan/CLionProjects/redis/src/server.c:263 #7 sigKillChildHandler (sig=<optimized out>) at /home/jonathan/CLionProjects/redis/src/server.c:6794 #8 <signal handler called> #9 0x00007e1234685b94 in _IO_fgets (buf=buf@entry=0x7e122dafdd90 "KSM:", ' ' <repeats 19 times>, "0 kB\n", n=n@entry=1024, fp=fp@entry=0x7e1220000b70) at ./libio/iofgets.c:47 #10 0x00005ef75326c5e0 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>, __s=<optimized out>, __n=<optimized out>, __stream=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:200 #11 zmalloc_get_smap_bytes_by_field (field=0x5ef7534c42fd "Private_Dirty:", pid=<optimized out>) at /home/jonathan/CLionProjects/redis/src/zmalloc.c:928 #12 0x00005ef75338ab1f in zmalloc_get_private_dirty (pid=-1) at /home/jonathan/CLionProjects/redis/src/zmalloc.c:978 #13 sendChildInfoGeneric (info_type=CHILD_INFO_TYPE_MODULE_COW_SIZE, keys=0, progress=-1, pname=0x5ef7534c95b2 "Module fork") at /home/jonathan/CLionProjects/redis/src/childinfo.c:71 #14 0x00005ef75337962c in sendChildCowInfo (pname=0x5ef7534c95b2 "Module fork", info_type=CHILD_INFO_TYPE_MODULE_COW_SIZE) at /home/jonathan/CLionProjects/redis/src/server.c:6895 #15 RM_ExitFromChild (retcode=0) at /home/jonathan/CLionProjects/redis/src/module.c:11468 ``` Change is to make the exit() _exit() calls conditional based on a parameter to exitFromChild function. The signal handler should exit without io operations since it doesn't know its history.(If we were in the middle of IO operations before it was called) --------- Co-authored-by: Yuan Wang <[email protected]>

#14072) Compiled Redis with COVERAGE_TEST, while using the fork API encountered the following issue: - Forked process calls `RedisModule_ExitFromChild` - child process starts to report its COW while performing IO operations - Parent process terminates child process with `RedisModule_KillForkChild` - Child process signal handler gets called while an IO operation is called - exit() is called because COVERAGE_TEST was on during compilation. - exit() tries to perform more IO operations in its exit handlers. - process gets deadlocked Backtrace snippet: ``` #0 futex_wait (private=0, expected=2, futex_word=0x7e1220000c50) at ../sysdeps/nptl/futex-internal.h:146 #1 __GI___lll_lock_wait_private (futex=0x7e1220000c50) at ./nptl/lowlevellock.c:34 #2 0x00007e1234696429 in __GI__IO_flush_all () at ./libio/genops.c:698 #3 0x00007e123469680d in _IO_cleanup () at ./libio/genops.c:843 #4 0x00007e1234647b74 in __run_exit_handlers (status=status@entry=255, listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:129 #5 0x00007e1234647bbe in __GI_exit (status=status@entry=255) at ./stdlib/exit.c:138 #6 0x00005ef753264e13 in exitFromChild (retcode=255) at /home/jonathan/CLionProjects/redis/src/server.c:263 #7 sigKillChildHandler (sig=<optimized out>) at /home/jonathan/CLionProjects/redis/src/server.c:6794 #8 <signal handler called> #9 0x00007e1234685b94 in _IO_fgets (buf=buf@entry=0x7e122dafdd90 "KSM:", ' ' <repeats 19 times>, "0 kB\n", n=n@entry=1024, fp=fp@entry=0x7e1220000b70) at ./libio/iofgets.c:47 #10 0x00005ef75326c5e0 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>, __s=<optimized out>, __n=<optimized out>, __stream=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:200 #11 zmalloc_get_smap_bytes_by_field (field=0x5ef7534c42fd "Private_Dirty:", pid=<optimized out>) at /home/jonathan/CLionProjects/redis/src/zmalloc.c:928 #12 0x00005ef75338ab1f in zmalloc_get_private_dirty (pid=-1) at /home/jonathan/CLionProjects/redis/src/zmalloc.c:978 #13 sendChildInfoGeneric (info_type=CHILD_INFO_TYPE_MODULE_COW_SIZE, keys=0, progress=-1, pname=0x5ef7534c95b2 "Module fork") at /home/jonathan/CLionProjects/redis/src/childinfo.c:71 #14 0x00005ef75337962c in sendChildCowInfo (pname=0x5ef7534c95b2 "Module fork", info_type=CHILD_INFO_TYPE_MODULE_COW_SIZE) at /home/jonathan/CLionProjects/redis/src/server.c:6895 #15 RM_ExitFromChild (retcode=0) at /home/jonathan/CLionProjects/redis/src/module.c:11468 ``` Change is to make the exit() _exit() calls conditional based on a parameter to exitFromChild function. The signal handler should exit without io operations since it doesn't know its history.(If we were in the middle of IO operations before it was called) --------- Co-authored-by: Yuan Wang <[email protected]>

redis#14072) Compiled Redis with COVERAGE_TEST, while using the fork API encountered the following issue: - Forked process calls `RedisModule_ExitFromChild` - child process starts to report its COW while performing IO operations - Parent process terminates child process with `RedisModule_KillForkChild` - Child process signal handler gets called while an IO operation is called - exit() is called because COVERAGE_TEST was on during compilation. - exit() tries to perform more IO operations in its exit handlers. - process gets deadlocked Backtrace snippet: ``` #0 futex_wait (private=0, expected=2, futex_word=0x7e1220000c50) at ../sysdeps/nptl/futex-internal.h:146 redis#1 __GI___lll_lock_wait_private (futex=0x7e1220000c50) at ./nptl/lowlevellock.c:34 redis#2 0x00007e1234696429 in __GI__IO_flush_all () at ./libio/genops.c:698 redis#3 0x00007e123469680d in _IO_cleanup () at ./libio/genops.c:843 redis#4 0x00007e1234647b74 in __run_exit_handlers (status=status@entry=255, listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:129 redis#5 0x00007e1234647bbe in __GI_exit (status=status@entry=255) at ./stdlib/exit.c:138 redis#6 0x00005ef753264e13 in exitFromChild (retcode=255) at /home/jonathan/CLionProjects/redis/src/server.c:263 redis#7 sigKillChildHandler (sig=<optimized out>) at /home/jonathan/CLionProjects/redis/src/server.c:6794 redis#8 <signal handler called> redis#9 0x00007e1234685b94 in _IO_fgets (buf=buf@entry=0x7e122dafdd90 "KSM:", ' ' <repeats 19 times>, "0 kB\n", n=n@entry=1024, fp=fp@entry=0x7e1220000b70) at ./libio/iofgets.c:47 redis#10 0x00005ef75326c5e0 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>, __s=<optimized out>, __n=<optimized out>, __stream=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:200 redis#11 zmalloc_get_smap_bytes_by_field (field=0x5ef7534c42fd "Private_Dirty:", pid=<optimized out>) at /home/jonathan/CLionProjects/redis/src/zmalloc.c:928 redis#12 0x00005ef75338ab1f in zmalloc_get_private_dirty (pid=-1) at /home/jonathan/CLionProjects/redis/src/zmalloc.c:978 redis#13 sendChildInfoGeneric (info_type=CHILD_INFO_TYPE_MODULE_COW_SIZE, keys=0, progress=-1, pname=0x5ef7534c95b2 "Module fork") at /home/jonathan/CLionProjects/redis/src/childinfo.c:71 redis#14 0x00005ef75337962c in sendChildCowInfo (pname=0x5ef7534c95b2 "Module fork", info_type=CHILD_INFO_TYPE_MODULE_COW_SIZE) at /home/jonathan/CLionProjects/redis/src/server.c:6895 redis#15 RM_ExitFromChild (retcode=0) at /home/jonathan/CLionProjects/redis/src/module.c:11468 ``` Change is to make the exit() _exit() calls conditional based on a parameter to exitFromChild function. The signal handler should exit without io operations since it doesn't know its history.(If we were in the middle of IO operations before it was called) --------- Co-authored-by: Yuan Wang <[email protected]>

initial commit

22b9a25

kei-nan added 2 commits May 25, 2025 15:32

solve unused argument error

993b6e9

fix for spellcheck and coverage flows

aa1095b

oranagra requested a review from ShooterIT May 25, 2025 12:42

kei-nan mentioned this pull request May 25, 2025

MOD-9775: Fix Coverage Deadlock In GC During Tests RediSearch/RediSearch#6202

Merged

2 tasks

ShooterIT reviewed May 26, 2025

View reviewed changes

src/childinfo.c Show resolved Hide resolved

src/server.c Outdated Show resolved Hide resolved

src/server.c Outdated Show resolved Hide resolved

code review fixes

fc2be00

ShooterIT reviewed May 26, 2025

View reviewed changes

src/server.c Outdated Show resolved Hide resolved

kei-nan and others added 3 commits May 26, 2025 13:47

Update src/server.c

f4051e8

Co-authored-by: Yuan Wang <[email protected]>

code review comment

a5e3ced

remove redundant comment

9e4f8de

ShooterIT approved these changes May 27, 2025

View reviewed changes

src/server.c Outdated Show resolved Hide resolved

Update src/server.c

ea46b49

ShooterIT added the state:to-be-merged The PR should be merged soon, even if not yet ready, this is used so that it won't be forgotten label May 27, 2025

ShooterIT changed the title ~~Avoid Performing IO On Coverage When Child Exits Due To Signal Handler~~ Avoid performing IO on coverage when child exits due to signal handler May 28, 2025

ShooterIT merged commit 161326d into redis:unstable May 28, 2025
18 checks passed

ShooterIT added this to Redis 8.2 Jul 3, 2025

github-project-automation bot moved this to Todo in Redis 8.2 Jul 3, 2025

ShooterIT removed this from Redis 8.2 Jul 3, 2025

ShooterIT added this to Redis 8.0 Backport Jul 3, 2025

github-project-automation bot moved this to Todo in Redis 8.0 Backport Jul 3, 2025

YaacovHazan moved this from Todo to Done in Redis 8.0 Backport Sep 29, 2025

kei-nan mentioned this pull request Nov 11, 2025

MOD-9609: Enhance Redis Coverage In CI RediSearch/RediSearch#7311

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid performing IO on coverage when child exits due to signal handler#14072

Avoid performing IO on coverage when child exits due to signal handler#14072
ShooterIT merged 8 commits intoredis:unstablefrom
kei-nan:child_signal_handler_should_not_call_regular_exit

kei-nan commented May 25, 2025

Uh oh!

snyk-io bot commented May 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ShooterIT commented May 28, 2025

Uh oh!

Uh oh!

oshadmi commented Jul 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kei-nan commented May 25, 2025

Uh oh!

snyk-io bot commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎉 Snyk checks have passed. No issues have been found so far.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ShooterIT commented May 28, 2025

Uh oh!

Uh oh!

oshadmi commented Jul 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

snyk-io bot commented May 25, 2025 •

edited

Loading