Skip to content

[Profiler] workaround for stack overflow during nested signal handling (#177)#8162

Closed
korniltsev-grafanista wants to merge 1 commit into
DataDog:masterfrom
grafana-cold-storage:kk/mask-sigprof
Closed

[Profiler] workaround for stack overflow during nested signal handling (#177)#8162
korniltsev-grafanista wants to merge 1 commit into
DataDog:masterfrom
grafana-cold-storage:kk/mask-sigprof

Conversation

@korniltsev-grafanista

@korniltsev-grafanista korniltsev-grafanista commented Feb 5, 2026 •
edited
Loading

Copy link
Copy Markdown

There is a stack overflow on the altstack when coreclr's SIGSEGV handler is interrupted by SIGPROF signal.

Here is the trace from kernel (obtained with this vibecoded ebpf tracer https://github.com/korniltsev-grafanista/signalsnoop ):

rsp: 0x00007ee895e36b30
get_sigframe for tgid=974490 tid=974548 (.NET TP Worker), ret=0x7ee895e35f38
x64_setup_rt_frame failed for tgid=974490 tid=974548 (.NET TP Worker), sig=27, ret=-14
7ee895e36000-7ee895e39000 rw-p 00000000 00:00 0 // altstack
Details 
get_sigframe for tgid=974490 tid=974548 (.NET TP Worker), ret=0x7ee895e35f38
        x64_setup_rt_frame
        arch_do_signal_or_restart
        irqentry_exit_to_user_mode
        asm_sysvec_apic_timer_interrupt
    Userspace registers:
        rip: 0x00007f2a70c16874  rsp: 0x00007ee895e36b30  flags: 0x0000000000000202
        rax: 0x00007ee895e37800  rbx: 0x00007ee87fbfdd68  rcx:   0x00007f2a70c16972
        rdx: 0x00007ee895e36b70  rsi: 0x0000000000000000  rdi:   0x00007ee895e36b90
        rbp: 0x00007ee895e377e0  r8:  0x00007ee895e37800  r9:    0x00007ee87418b629
        r10: 0x00000000000000f5  r11: 0x00007f2a714c1000  r12:   0x00007ee87fbfdd70
        r13: 0x00007ee895e36b90  r14: 0x00007ee895e38480  r15:   0x00007ee87fbfde00

x64_setup_rt_frame failed for tgid=974490 tid=974548 (.NET TP Worker), sig=27, ret=-14
    sa_flags: 0x14000004 (SA_RESTORER=true)
        arch_do_signal_or_restart
        irqentry_exit_to_user_mode
        asm_sysvec_apic_timer_interrupt
    Userspace registers:
        rip: 0x00007f2a70c16874  rsp: 0x00007ee895e36b30  flags: 0x0000000000000202
        rax: 0x00007ee895e37800  rbx: 0x00007ee87fbfdd68  rcx:   0x00007f2a70c16972
        rdx: 0x00007ee895e36b70  rsi: 0x0000000000000000  rdi:   0x00007ee895e36b90
        rbp: 0x00007ee895e377e0  r8:  0x00007ee895e37800  r9:    0x00007ee87418b629
        r10: 0x00000000000000f5  r11: 0x00007f2a714c1000  r12:   0x00007ee87fbfdd70
        r13: 0x00007ee895e36b90  r14: 0x00007ee895e38480  r15:   0x00007ee87fbfde00
    Stack probe:
        [sp+0] 0x00007ee895e36b30: 0x00007f2a70be4ad0
        [sp-128] 0x00007ee895e36ab0: 0x0000000000000000
        [sp-568] 0x00007ee895e368f8: 0x0000000000000000
        [sp-700] 0x00007ee895e36874: 0x0000000000000000

signal_setup_done failed for tgid=974490 tid=974548 (.NET TP Worker), sig=27, ret=1
        signal_setup_done
        arch_do_signal_or_restart
        irqentry_exit_to_user_mode
        asm_sysvec_apic_timer_interrupt
    Userspace registers:
        rip: 0x00007f2a70c16874  rsp: 0x00007ee895e36b30  flags: 0x0000000000000202
        rax: 0x00007ee895e37800  rbx: 0x00007ee87fbfdd68  rcx:   0x00007f2a70c16972
        rdx: 0x00007ee895e36b70  rsi: 0x0000000000000000  rdi:   0x00007ee895e36b90
        rbp: 0x00007ee895e377e0  r8:  0x00007ee895e37800  r9:    0x00007ee87418b629
        r10: 0x00000000000000f5  r11: 0x00007f2a714c1000  r12:   0x00007ee87fbfdd70
        r13: 0x00007ee895e36b90  r14: 0x00007ee895e38480  r15:   0x00007ee87fbfde00
    Stack probe:
        [sp+0] 0x00007ee895e36b30: 0x00007f2a70be4ad0
        [sp-128] 0x00007ee895e36ab0: 0x0000000000000000
        [sp-568] 0x00007ee895e368f8: 0x0000000000000000
        [sp-700] 0x00007ee895e36874: 0x0000000000000000

vfs_coredump for tgid=974490 tid=974548 (.NET TP Worker)
        vfs_coredump
        get_signal
        arch_do_signal_or_restart
        irqentry_exit_to_user_mode
        asm_sysvec_apic_timer_interrupt
    Userspace registers:
        rip: 0x00007f2a70c16874  rsp: 0x00007ee895e36b30  flags: 0x0000000000000202
        rax: 0x00007ee895e37800  rbx: 0x00007ee87fbfdd68  rcx:   0x00007f2a70c16972
        rdx: 0x00007ee895e36b70  rsi: 0x0000000000000000  rdi:   0x00007ee895e36b90
        rbp: 0x00007ee895e377e0  r8:  0x00007ee895e37800  r9:    0x00007ee87418b629
        r10: 0x00000000000000f5  r11: 0x00007f2a714c1000  r12:   0x00007ee87fbfdd70
        r13: 0x00007ee895e36b90  r14: 0x00007ee895e38480  r15:   0x00007ee87fbfde00
    Stack probe:
        [sp+0] 0x00007ee895e36b30: 0x00007f2a70be4ad0
        [sp-128] 0x00007ee895e36ab0: 0x0000000000000000
        [sp-568] 0x00007ee895e368f8: 0x0000000000000000
        [sp-700] 0x00007ee895e36874: 0x0000000000000000
    Process maps:
    
    
7ee895e36000-7ee895e39000 rw-p 00000000 00:00 0   <-- rsp=0x7ee895e36b30, rbp=0x7ee895e377e0, rax=0x7ee895e37800, rdx=0x7ee895e36b70, rdi=0x7ee895e36b90, r8=0x7ee895e37800, r13=0x7ee895e36b90, r14=0x7ee895e38480
7ee895e35f38

This PR adds wrappers to sigaction and pthread_sigmask.

sigaction masks SIGPROF if the signal is SIGSEGV - to prevent SIGSEGV stack overflow during signal delivering.
pthread_sigmask unblocks SIGPROF if we are unblocking SIGSEGV - the runtime unblocks SIGSEGV, and we should unblock SIGPROF as well to allow the profiler continue working and delivering SIGPROF signals.

This is the code I used to simplify reproducing - triggering null deref in managed code in a busy loop. https://github.com/grafana/pyroscope-dotnet/pull/177/files#diff-75c68efd0775648bc723a9557e3cc83a294c79852dc5dc2c733e8bb80e15a9fcR18

Disclaimer: the problem was initially reported by grafana user(using the fork https://github.com/grafana/pyroscope-dotnet) we then confirmed the same problem with the genuine (not fork) dd-trace-dotnet v0.34.0.

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

@korniltsev-grafanista
korniltsev-grafanista requested a review from a team as a code owner February 5, 2026 11:04
@gleocadie

gleocadie commented Feb 5, 2026

Copy link
Copy Markdown
Collaborator

👋 @korniltsev-grafanista , thanks for your contribution.
Do you want to add a test in our integration tests or I should do it ?

@gleocadie gleocadie changed the title workaround for stack overflow during nested signal handling (#177) [Profiler] workaround for stack overflow during nested signal handling (#177) Feb 5, 2026
@gleocadie gleocadie added the area:profiler Issues related to the continous-profiler label Feb 5, 2026
@korniltsev-grafanista

korniltsev-grafanista commented Feb 6, 2026

Copy link
Copy Markdown
Author

please do it, I don't really have capacity at the moment.

@gleocadie

gleocadie commented Feb 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

👋
looking around in the .NET runtime code source, and I do not see calls to pthread_sigmask to unblock SIGSEGV.
Only sigaction wrapper is needed today, no ?
did you see/witness something for the pthread_sigmask case ?

@korniltsev-grafanista

korniltsev-grafanista commented Feb 7, 2026

Copy link
Copy Markdown
Author

I believe it's the kernel who blocks the delivered signal (SIGSEGV) together with sigaction.sa_mask so that it is not delivered second time while the program is already handling same signal.

Usually when the signal handler returns , libc performs sigreturn syscall and the kernel then unblock the delivered signal together with signals from sigaction.sa_mask .

However coreclr never returns from the sigsegv signal handler, instead it jumps back from altstack to the real thread stack and manually unblocks the signal here https://github.com/dotnet/runtime/blob/2d638dc1179164a08d9387cbe6354fe2b7e4d823/src/coreclr/pal/src/exception/signal.cpp#L1119 .

The new wrapped sigaction is needed to make sure SIGPROF is not delivered while we handle SIGSEGV to prevent stack overflow crash - it automatically blocks SIGPROF when SIGSEGV is happening.
The new wrapped pthread_sigmask is needed to unblock SIGPROF together with SIGSEGV. If we don't do this, then the kernel queues SIGPROF signals to the thread and never deliver them because they remain blocked.

Hope this makes sense.

@gleocadie

gleocadie commented Feb 7, 2026

Copy link
Copy Markdown
Collaborator

I believe it's the kernel who blocks the delivered signal (SIGSEGV) together with sigaction.sa_mask so that it is not delivered second time while the program is already handling same signal.

Usually when the signal handler returns , libc performs sigreturn syscall and the kernel then unblock the delivered signal together with signals from sigaction.sa_mask .

However coreclr never returns from the sigsegv signal handler, instead it jumps back from altstack to the real thread stack and manually unblocks the signal here https://github.com/dotnet/runtime/blob/2d638dc1179164a08d9387cbe6354fe2b7e4d823/src/coreclr/pal/src/exception/signal.cpp#L1119 .

The new wrapped sigaction is needed to make sure SIGPROF is not delivered while we handle SIGSEGV to prevent stack overflow crash - it automatically blocks SIGPROF when SIGSEGV is happening. The new wrapped pthread_sigmask is needed to unblock SIGPROF together with SIGSEGV. If we don't do this, then the kernel queues SIGPROF signals to the thread and never deliver them because they remain blocked.

Hope this makes sense.

Thanks for the explanation. I missed the part where the runtime was "jumping" and that calls to pthread_sigmask.

@gleocadie

gleocadie commented Feb 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Interesting, we already have a test for that https://github.com/DataDog/dd-trace-dotnet/blob/master/profiler/src/Demos/Samples.Computer01/SigSegvHandlerExecution.cs#L22
And if fell through 🤔
I faced this kind of crash in the past and I tried to fix it by detecting in our signal handlers if we were in a sigsegv handler before. But that's was no ideal because there was still a race (that could explain that we do not see it in CI).
Anyway, I prefer your approach :) I'll see if I can 100% reproduce it with you reproducer. If it's the case, I'll change my test code.
Again, thanks for that.

@korniltsev-grafanista
workaround for stack overflow during nested signal handling (DataDog#177
c2786ee 
)

There is a stack buffer overflow on the altstack when coreclr's SIGSEGV handler is interrupted by SIGPROF signal.

Here is the trace from kernel (obtained with this tracer https://github.com/korniltsev-grafanista/signalsnoop ):

rsp: 0x00007ee895e36b30
get_sigframe for tgid=974490 tid=974548 (.NET TP Worker), ret=0x7ee895e35f38
x64_setup_rt_frame failed for tgid=974490 tid=974548 (.NET TP Worker), sig=27, ret=-14
7ee895e36000-7ee895e39000 rw-p 00000000 00:00 0 // altstack

This PR adds wrappers to sigaction and pthread_sigmask.

sigaction masks SIGPROF if the signal is SIGSEGV - to prevent SIGSEGV stack overflow during signal delivering.
pthread_sigmask unblocks SIGPROF if we are unblocking SIGSEGV - the runtime unblocks SIGSEGV, and we should unblock SIGPROF as well to allow the profiler continue working and delivering SIGPROF signals.

Added an NPE class to Rideshare app for easier problem reproducing. // Copilot ignore this code, review the wrapper
@korniltsev-grafanista

korniltsev-grafanista commented Apr 14, 2026

Copy link
Copy Markdown
Author

signed here just in case as well

@korniltsev-grafanista korniltsev-grafanista mentioned this pull request Apr 14, 2026
Closed
@gleocadie gleocadie mentioned this pull request Apr 17, 2026
Merged
@korniltsev-grafanista

korniltsev-grafanista commented Apr 21, 2026

Copy link
Copy Markdown
Author

seems to have landed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area:profiler Issues related to the continous-profiler

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@korniltsev-grafanista @gleocadie