Fix debug symbols by azat · Pull Request #40873 · ClickHouse/ClickHouse

azat · 2022-08-31T18:25:52Z

Changelog category (leave one):

Build/Testing/Packaging Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fix debug symbols

Patchset

Debug symbols had been broken in many places and should not work for a long time, the fix includes the following patches:

Fix debug symbols (Add a quirk to force clang emit .debug_aranges with ThinLTO)
Also note, that to support it without a quirk, patch for llvm is required, the discussion can be found here and the patch here
Fix loading external symbols (previously it does not look at /usr/lib/debug/usr/bin/clickhouse.debug)
Do not compress debug symbols since dwarf parser cannot handle it, this increases the debug file from 510MB to 1.8GB, but deb packages compressed anyway, but the size of the deb also slightly increased from 834M to 962M
Do not try to load empty debug files (this is required due to build creates empty files for sanitizers and debug builds)
And also it disables the 02161_addressToLineWithInlines test, since they do not work always for now (will take a look at this)

Fixes: #36797
Fixes: #31639

cmake/ld.lld.in

azat · 2022-08-31T21:20:08Z

Everything works as expected, and just for the info:

binary without LTO

azat@s2:.../tmp/debug-symbols$ objdump -j .debug_aranges -h clickhouse-binary 

clickhouse-binary:     file format elf64-x86-64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn                 
 35 .debug_aranges 005e92e0  0000000000000000  0000000000000000  622f814e  2**0                
                  CONTENTS, READONLY, DEBUGGING, OCTETS

binary with LTO

azat@s4:~/ch/tmp/40873$ objdump -j .debug_aranges -h clickhouse

clickhouse:     file format elf64-x86-64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
 34 .debug_aranges 0055eb30  0000000000000000  0000000000000000  4b52c259  2**0
                  CONTENTS, READONLY, DEBUGGING, OCTETS

So size more or less the same.

…h ThinLTO) Wrap a linker into a script that will add some settings (`-mllvm -generate-arange-section`) in case of ThinLTO to emit `.debug_aranges` symbols. Dicussion in the LLVM can be found here [1]. [1]: https://discourse.llvm.org/t/clang-does-not-produce-full-debug-aranges-section-with-thinlto/64898 Signed-off-by: Azat Khuzhin <[email protected]>

Signed-off-by: Azat Khuzhin <[email protected]>

…uch) Although this increase debug symbol size from 510MB to 1.8GB, but it is not a problem for packages, since they are compressed anyway. Checked deb package, and size slightly increased though, 834M -> 962M. Signed-off-by: Azat Khuzhin <[email protected]>

Signed-off-by: Azat Khuzhin <[email protected]>

This will avoid CANNOT_PARSE_ELF error for builds that has empty debug file in clickhouse-common-static-dbg package, i.e. debug build. Signed-off-by: Azat Khuzhin <[email protected]>

addressToLineWithInlines() may lead to the following error: Code: 465. DB::Exception: Received from localhost:9000. DB::Exception: could not find abbreviation code: while executing 'FUNCTION addressToLineWithInlines(arrayJoin(trace) :: 1) -> addressToLineWithInlines(arrayJoin(trace)) Array(String) : 0'. (CANNOT_PARSE_DWARF) CI: https://s3.amazonaws.com/clickhouse-test-reports/40873/45fd2bcb218ace3231a026eb91d688f0093c6407/stateless_tests__release_.html Signed-off-by: Azat Khuzhin <[email protected]>

azat · 2022-09-03T07:28:03Z

Now it works (and it was broken in lots of places, for details see pull request description), but inlines does not, I did not managed to fix it quickly so for now 02161_addressToLineWithInlines had been disabled and in the meantime I will take a look.

Test failures does not looks related and I will take a look in details a little bit later.

novikd

In general, LGTM, but I think it might be useful to create issues for inlines and --compress-debug-sections because it's easy to forget that we are missing it.

tests/queries/0_stateless/02161_addressToLineWithInlines.sql

azat · 2022-09-04T10:09:17Z

I think it might be useful to create issues for inlines and --compress-debug-sections because it's easy to forget that we are missing it.

Actually I was going to take a look at the llvm parser (folly does not support it still), and I guess it will support them automatically.

However I'm not sure that this worth it, because symbolizing stacktraces takes lots of time here (and hence have a cache), and increasing it even more does not looks good to me, anyway deb packages are compressed (so debug file will be compressed anyway), and regular binaries compressed with zstd now.

But robust DWARF parser is still required, and maybe decompression will not take that much...

I will create needable issues today (need to take a brief look at some stuff).

azat · 2022-09-05T12:54:00Z

Ok, #41000, #35405 and maybe #40772 should be enough to improve the situation

azat · 2022-09-07T10:17:06Z

I would prefer to merge this before switching to clang/llvm-15 #41046 (it may help with the 00974_query_profiler failures there and also maybe clang-15 will make something different and this will stop working again).

novikd · 2022-09-07T15:31:22Z

Stress test (msan) — Killed by signal (in clickhouse-server.log)

Must be fixed in #41050

Stateless tests (release, s3 storage) — fail: 1, passed: 4364, skipped: 61
Stateless tests (ubsan) — Tests are not finished, fail: 1, passed: 90, skipped: 3

Seems unrelated

robot-ch-test-poll1 added the pr-build Pull request with build/testing/packaging improvement label Aug 31, 2022

azat changed the title ~~Fix debug symbols (Add a quirk to force clang emit .debug_aranges with ThinLTO)~~ RFC: Fix debug symbols (Add a quirk to force clang emit .debug_aranges with ThinLTO) Aug 31, 2022

azat marked this pull request as draft August 31, 2022 20:47

azat commented Aug 31, 2022

View reviewed changes

cmake/ld.lld.in Outdated Show resolved Hide resolved

azat changed the title ~~RFC: Fix debug symbols (Add a quirk to force clang emit .debug_aranges with ThinLTO)~~ RFC: Fix debug symbols Sep 2, 2022

azat added 4 commits September 2, 2022 23:22

Fix loading external symbols

45afaa6

Signed-off-by: Azat Khuzhin <[email protected]>

Improve error messages in Elf loader

597197c

Signed-off-by: Azat Khuzhin <[email protected]>

azat force-pushed the build/fix-debug-symbols-quirk branch from 45fd2bc to 2fd8098 Compare September 2, 2022 21:22

azat added 2 commits September 2, 2022 23:23

Do not try to load empty debug files

9479e21

This will avoid CANNOT_PARSE_ELF error for builds that has empty debug file in clickhouse-common-static-dbg package, i.e. debug build. Signed-off-by: Azat Khuzhin <[email protected]>

azat force-pushed the build/fix-debug-symbols-quirk branch from 2fd8098 to c6cbd98 Compare September 2, 2022 21:23

azat marked this pull request as ready for review September 3, 2022 07:21

novikd self-assigned this Sep 4, 2022

novikd approved these changes Sep 4, 2022

View reviewed changes

tests/queries/0_stateless/02161_addressToLineWithInlines.sql Show resolved Hide resolved

azat changed the title ~~RFC: Fix debug symbols~~ Fix debug symbols Sep 5, 2022

azat mentioned this pull request Sep 5, 2022

Try llvm DWARF parser #41000

Open

azat mentioned this pull request Sep 7, 2022

Switch to llvm/clang 15 #41046

Merged

7 tasks

novikd merged commit 499e479 into ClickHouse:master Sep 7, 2022

azat deleted the build/fix-debug-symbols-quirk branch September 7, 2022 15:33

azat mentioned this pull request Sep 11, 2022

addressToLine misbehaves in master #31639

Closed

azat mentioned this pull request Oct 18, 2022

Better workaround for emitting .debug_aranges section #42457

Merged

azat mentioned this pull request May 8, 2023

Switch to LLVM/clang 16 (16.0.3) #49678

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix debug symbols #40873

Fix debug symbols #40873
novikd merged 6 commits intoClickHouse:masterfrom
azat:build/fix-debug-symbols-quirk

azat commented Aug 31, 2022 •

edited

Loading

Uh oh!

Uh oh!

azat commented Aug 31, 2022 •

edited

Loading

Uh oh!

azat commented Sep 3, 2022 •

edited

Loading

Uh oh!

novikd left a comment

Uh oh!

Uh oh!

azat commented Sep 4, 2022 •

edited

Loading

Uh oh!

azat commented Sep 5, 2022

Uh oh!

azat commented Sep 7, 2022 •

edited

Loading

Uh oh!

novikd commented Sep 7, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

azat commented Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Patchset

Uh oh!

Uh oh!

azat commented Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

binary without LTO

binary with LTO

Uh oh!

azat commented Sep 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

novikd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

azat commented Sep 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azat commented Sep 5, 2022

Uh oh!

azat commented Sep 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

novikd commented Sep 7, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

azat commented Aug 31, 2022 •

edited

Loading

azat commented Aug 31, 2022 •

edited

Loading

azat commented Sep 3, 2022 •

edited

Loading

azat commented Sep 4, 2022 •

edited

Loading

azat commented Sep 7, 2022 •

edited

Loading