Skip to content

branch-4.1: [feature](be) Add BE stack trace HTTP API #64454 - #64625

Merged
yiguolei merged 1 commit into
branch-4.1from
auto-pick-64454-branch-4.1
Jul 3, 2026
Merged

yiguolei merged 1 commit into
branch-4.1from
auto-pick-64454-branch-4.1

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Cherry-picked from #64454

@github-actions
github-actions Bot requested a review from yiguolei as a code owner June 18, 2026 02:12
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

Copy link
Copy Markdown
Contributor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 62.99% (599/951) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.85% (27499/37234)
Line Coverage 57.44% (298006/518829)
Region Coverage 54.96% (249579/454092)
Branch Coverage 56.46% (108357/191919)

This PR adds a BE-side diagnostic HTTP API for collecting thread stack
traces from a live BE process:

```text
GET /api/stack_trace
GET /api/stack_trace?thread_id=<tid>[,<tid>...]
GET /api/stack_trace?mode=<DISABLED|FAST|FULL|FULL_WITH_INLINE>
GET /api/stack_trace?dwarf_location_info_mode=<DISABLED|FAST|FULL|FULL_WITH_INLINE>
GET /api/stack_trace?timeout_ms=<ms>
GET /api/stack_trace?skip_blocking_syscalls=<true|false>
```

`thread_id` is the preferred explicit selector. `tid` is kept only as a
legacy alias.

The default full-process collection samples all BE threads without a
signal-attempt cap and does not skip blocking syscalls. This is
intentional: during real incidents, blocked worker stacks are often the
useful part of the dump. `skip_blocking_syscalls=true` remains available
as an explicit conservative mode for isolating EINTR-sensitive paths.

The output is plain text and includes:

- BE pid, diagnostic signal number, thread count, timeout, DWARF mode,
and syscall-skip mode.
- One section per thread, with TID, thread name, capture status, capture
method, frame count, frame-pointer status, and stack bounds.
- A summary line with captured, skipped, timed-out, signal-error, and
exited counts.

Example header:

```text
BE thread stack traces
pid: 96543
service_signal: 40
thread_count: 1168
timeout_ms_per_thread: 3000
dwarf_location_info_mode: fast
skip_blocking_syscalls: false
signal_handler_unwinder: frame_pointer_with_coordinator_signal_context_libunwind_fallback
```

Example execution stack captured under load:

```text
----- thread 12438 (p_normal_simple) status=ok capture_method=frame_pointer frames=32 fp_status=end_of_chain stack_bounds=... -----
    doris::CastToStringFunction::execute_impl(...)
    doris::VectorizedFnCall::execute_column_impl(...)
    doris::AggSinkLocalState::_execute_without_key(...)
    doris::AggSinkOperatorX::sink_impl(...)
    doris::DataSinkOperatorXBase::sink(...)
    doris::PipelineTask::execute(bool*)
    doris::TaskScheduler::_do_work(int)
```

Implementation notes:

- The remote-thread capture path uses a signal handler to copy only
lightweight frame-pointer/context data from the target thread.
Symbolization and fallback unwinding run outside the handler.
- The handler publishes through one process-wide slot, so HTTP
collection is serialized. The coordinator waits until the previous
handler has fully released its latch before signaling the next TID;
otherwise back-to-back captures can drop the next signal.
- Stack bounds are collected before sending any signal, because the
handler must not open `/proc`, allocate memory, or take locks while
interrupting an arbitrary BE thread.
- Frame-pointer walking validates stack-range bounds and pointer
alignment before reading frame records.
- The same PCs can be rendered with different DWARF modes, so the
StackTrace cache key includes `dwarf_location_info_mode`. This prevents
a `DISABLED` rendering from poisoning a later `FULL` request and
prevents full file/line detail from leaking into a disabled request.
- Instruction addresses are compared via `uintptr_t` instead of pointer
arithmetic on non-object instruction addresses.
- Threads that block the diagnostic signal are reported as
`status=signal_blocked` instead of being treated as ambiguous timeouts.
- This is not a global stop-the-world snapshot. Each thread is sampled
independently as the coordinator walks the TID list, so stacks are close
in time but not guaranteed to be from the exact same instant.

EINTR handling:

- Sending a diagnostic signal can interrupt blocking syscalls even when
`SA_RESTART` is used, especially for poll/select/accept-style waits.
- Doris Thrift servers previously could treat `Interrupted system call`
from `TServerSocket::acceptImpl()` as a fatal server-loop exception.
- This PR wraps BE Thrift server sockets with `ImprovedServerSocket`,
which retries only the narrow EINTR/`Interrupted system call` case for
accept/read/peek. Other transport exceptions keep the original behavior.
@yiguolei
yiguolei force-pushed the auto-pick-64454-branch-4.1 branch from c37134b to 5a24c88 Compare July 2, 2026 00:31
@yiguolei

yiguolei commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 62.99% (599/951) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.82% (27585/37366)
Line Coverage 57.46% (299195/520715)
Region Coverage 55.16% (251367/455687)
Branch Coverage 56.48% (108781/192594)

@yiguolei
yiguolei merged commit d2b3ab8 into branch-4.1 Jul 3, 2026
28 of 32 checks passed
@morningman
morningman deleted the auto-pick-64454-branch-4.1 branch July 8, 2026 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants