Skip to content

branch-4.1.4: [fix](thirdparty) Upgrade libunwind to 1.8.3 to fix aarch64 BE startup crash on 64 KiB-page kernels - #68507

Merged
morningman merged 1 commit into
apache:branch-4.1.4from
morningman:pick-68339-branch-4.1.4
Sep 25, 2026
Merged

morningman merged 1 commit into
apache:branch-4.1.4from
morningman:pick-68339-branch-4.1.4

Conversation

@morningman

Copy link
Copy Markdown
Contributor

Cherry-picked from #67877 (master commit e797e99fcbe) via its branch-4.1 pick #68339 (commit 5e1ee245e23) to branch-4.1.4.

Conflict resolution. This branch is a 2026-09-04 cut of branch-4.1, so two of the seven files differed:

  • thirdparty/CHANGELOG.md: branch-4.1's ## 20260901 thrift 0.16.0 → 0.24.0 entry is not on this branch (it still pins thrift 0.16.0), and neither is master's ## 20260909 snappy 1.2.1 entry. Only the ## 20260911 libunwind entry is inserted, above ## 20260429.
  • thirdparty/patches/libunwind-1.8.3-doris-phdr-cache.patch: this branch still carries the pre-branch-4.1: [fix](thirdparty) Repair the libunwind phdr-cache patch mangled in the #64093 backport #67564 1.6.2 patch, and a patch file has no mergeable content, so the new file is taken wholesale from the branch-4.1 pick. It is byte-identical to the branch-4.1 and master versions.

The other five files applied without conflict. thirdparty/vars.sh, thirdparty/build-thirdparty.sh, thirdparty/download-thirdparty.sh and .github/workflows/build-thirdparty.yml still differ from branch-4.1, but only by thirdparty drift this branch has not taken (thrift 0.24.0, aws-sdk-cpp 1.11.221, simdutf, google-cloud-cpp, lance-c 0.1.9, jemalloc); no hunk of this change is affected. thirdparty/test/libunwind-page-size-test.sh and the rebased patch are byte-identical to master's.

What problem does this PR solve?

Related PRs: #64093, #67877, #68339

Problem Summary:

Every aarch64 BE built from branch-4.1.4 since 4.1.4-rc01 crashes before main() on a 64 KiB-page kernel (Kunpeng 920 + Kylin V10 / openEuler / CentOS aarch64, the common ARM server stack). The core from the 4.1.4 arm64 deployment shows:

access_mem (addr=0)            src/aarch64/Ginit.c
is_plt_entry                   src/aarch64/Gstep.c
_ULaarch64_step
_ULaarch64_tdep_trace
unw_backtrace
<bthread/mutex.cpp static initializer: backtrace(dummy_buf, 4)>   ← bRPC "Warm up backtrace before main()"
__libc_csu_init

SIGSEGV with si_addr=0, AUXV PAGESZ=65536, and inside the libunwind cursor ip=0, validate=0, mem_validate_func=msync_validate, last_good_addr[]={0,0,0,0}.

How it happens:

  1. [Enhancement](BE) Make print stack API more stable #64093 links GNU libunwind statically into the aarch64 BE (it forces USE_UNWIND=ON on Linux, references unw_* from phdr_cache.cpp without the __x86_64__ guard, and builds jemalloc with --enable-prof-libunwind). libunwind's weak backtrace alias then replaces glibc's backtrace() for every caller in the process: bRPC's pre-main warm-up, StackTrace::tryCapture() on aarch64, jemalloc heap profiling. The 4.1.3 arm64 package was immune only because it was built with USE_UNWIND=OFF (which the ARM compilation doc still prescribes and [Enhancement](BE) Make print stack API more stable #64093 removed): it has backtrace@GLIBC_2.17 and zero _ULaarch64_* symbols.
  2. libunwind 1.6.2 hard-codes PAGE_SIZE 4096 in src/aarch64/Ginit.c. On a 64 KiB-page kernel mincore()/msync() reject its 4 KiB-aligned probes with EINVAL, so validate_mem() fails for 15/16 of all addresses (the core's msync_validate + empty last_good_addr are exactly this).
  3. unw_step() therefore gets unw_is_signal_frame() < 0 for a valid IP, takes the "IP points to non-mapped memory, use LR" recovery path and loads uc.regs[30] — a slot _Uaarch64_getcontext_trace never writes (it saves only FP/SP/PC). On the reporting host that stack garbage is 0.
  4. dwarf_step(ip=0) finds nothing, and the fallback calls is_plt_entry() with c->validate restored to 0, so access_mem executes ldr x8, [x1] with x1=0.

The PHDR cache being empty before main() is not what breaks this (step 3 has already poisoned ip before any FDE lookup, and the same crash reproduces without the Doris hook); changing the cache initialisation order would not fix it, and every later backtrace() on such a host rolls the same dice.

Changes

Fix: upgrade libunwind 1.6.2 → 1.8.3, which was never bumped since it was introduced in 2023 (#21938):

  • 1.7.0+: unw_page_size = sysconf(_SC_PAGESIZE) instead of the hard-coded 4096.
  • 1.8.0+: src/mi/Gaddress_validator.c validates through the pipe-write probe only, so mincore/msync alignment no longer matters at all; unw_step() sets c->validate = 1 before the DWARF-failure fallback, so a garbage IP is rejected instead of dereferenced.

Specifically:

  • thirdparty/vars.sh: libunwind 1.6.2 → 1.8.3 (md5 13bc7b41462ac6ea157d350eaf6c1503).
  • thirdparty/patches/libunwind-1.8.3-doris-phdr-cache.patch: the [Enhancement](BE) Make print stack API more stable #64093 hook rebased onto 1.8.3's as->iterate_phdr_function call site; semantics unchanged (doris_unwind_iterate_phdr still takes precedence whenever it is linked in). 1.8 also exposes unw_set_iterate_phdr_function(), which could replace the source patch in a follow-up.
  • thirdparty/build-thirdparty.sh: --disable-tests --disable-documentation; only libunwind.a is consumed.
  • thirdparty/test/libunwind-page-size-test.sh (+ run at the end of the Linux thirdparty CI build): reproduces the failure on any Linux host. An LD_PRELOAD shim gives libunwind a 64 KiB-page kernel's behaviour (EINVAL for unaligned mincore/msync, sysconf(_SC_PAGESIZE)=65536), and a program shaped like the bRPC warm-up calls backtrace() from a constructor with a zeroed stack. Against 1.6.2 it segfaults with the customer's exact stack and cursor state; against 1.8.3 it passes.

No BE source change: every unw_* API the BE, jemalloc and the x86_64 stack-trace code use is unchanged in 1.8.3, cxx_exceptions stays off on x86_64/aarch64, and the weak backtrace alias is still provided.

Release note

Fix the aarch64 BE crashing before main() (SIGSEGV in access_mem/is_plt_entry under unw_backtrace) on 64 KiB-page kernels such as Kunpeng 920 with Kylin, openEuler or CentOS, by upgrading the bundled libunwind from 1.6.2 to 1.8.3.

Check List (For Author)

  • Test

  • Behavior changed:

    • Yes. Bundled libunwind 1.6.2 → 1.8.3; libunwind's own test programs and man pages are no longer built. Unwinding semantics for the BE otherwise unchanged.
  • Does this need documentation?

    • No.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…ch64 BE startup crash on 64 KiB-page kernels

Cherry-picked from apache#67877 (master e797e99), via its branch-4.1 pick apache#68339
(commit 5e1ee24).

Conflict resolution on this branch: two files differ from branch-4.1.

- `thirdparty/CHANGELOG.md`: this branch has not taken the `## 20260901` thrift
  0.16.0 -> 0.24.0 entry (nor master's `## 20260909` snappy one), so only the
  `## 20260911` libunwind entry is inserted, above `## 20260429`.
- `thirdparty/patches/libunwind-1.8.3-doris-phdr-cache.patch`: this branch still
  carries the pre-apache#67564 1.6.2 patch, and a patch file has no mergeable content,
  so the new file is taken wholesale from the branch-4.1 pick. It is now
  byte-identical to the branch-4.1 and master versions.

The remaining five files applied without conflict. `thirdparty/vars.sh`,
`thirdparty/build-thirdparty.sh`, `thirdparty/download-thirdparty.sh` and
`.github/workflows/build-thirdparty.yml` still differ from branch-4.1, but only by
thirdparty drift this branch has not taken (thrift 0.24.0, aws-sdk-cpp 1.11.221,
simdutf, google-cloud-cpp, lance-c 0.1.9, jemalloc); no hunk of this change is
affected, and `thirdparty/test/libunwind-page-size-test.sh` is byte-identical.

Release note: Fix the aarch64 BE crashing before `main()` (SIGSEGV in
`access_mem`/`is_plt_entry` under `unw_backtrace`) on 64 KiB-page kernels such as
Kunpeng 920 with Kylin, openEuler or CentOS, by upgrading the bundled libunwind
from 1.6.2 to 1.8.3.

(cherry picked from commit 5e1ee24)
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
morningman merged commit 714cbe9 into apache:branch-4.1.4 Sep 25, 2026
20 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants