Skip to content

Get rid of malloc symbols in libcommon#7065

Merged
abyss7 merged 1 commit intoClickHouse:masterfrom
amosbird:clang-shared-build-fix
Sep 27, 2019
Merged

Get rid of malloc symbols in libcommon#7065
abyss7 merged 1 commit intoClickHouse:masterfrom
amosbird:clang-shared-build-fix

Conversation

@amosbird
Copy link
Copy Markdown
Collaborator

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

For changelog. Remove if this is non-significant change.

Category (leave one):

  • Build/Testing/Packaging Improvement

@alexey-milovidov
Copy link
Copy Markdown
Member

One build has failed.

@amosbird amosbird force-pushed the clang-shared-build-fix branch 2 times, most recently from 591130c to 47963f6 Compare September 24, 2019 02:45
@abyss7
Copy link
Copy Markdown
Contributor

abyss7 commented Sep 24, 2019

@amosbird Can you explain in details what is this fix about and why the issue can't be resolved with typical CMake dependencies and interfaces? Overriding default macro looks very hacky, generally it may lead to new problems, when someone decides to do any refactoring months later.

@amosbird
Copy link
Copy Markdown
Collaborator Author

amosbird commented Sep 24, 2019

@abyss7 When doing shared builds, TUs which try to override std symbols (such as malloc or operator new) should be statically linked into the final executable and only be linked once. It includes all unit testing executables. I cannot find easy way of collecting executables in one place other than overriding default macro.

@abyss7
Copy link
Copy Markdown
Contributor

abyss7 commented Sep 24, 2019

@abyss7 When doing shared builds, TUs which try to override std symbols (such as malloc or operator new) should be directly linked into the final executable and only be linked once. It includes all unit testing executables. I cannot find easy way of collecting executables in one place other than overriding default macro.

Thank you for the explanation. The logic is understandable from your changes - but I don't understand the underlying problem.

Can you explain why can't we link the malloc() override into DSO? What is the problem with linker or dynamic loader, if we do otherwise?

@amosbird
Copy link
Copy Markdown
Collaborator Author

Can you explain why can't we link the malloc() override into DSO? What is the problem with linker or dynamic loader, if we do otherwise?

Most likely because it might fail to load in time upon starting.

@alexey-milovidov
Copy link
Copy Markdown
Member

Maybe it is dependency issue. If all .so that do malloc/free depend on .so that provide overrides, this .so must be loaded beforehand.

@amosbird
Copy link
Copy Markdown
Collaborator Author

Tbh, I still don't have a clear mind of all other broken cases, but the way in current PR should work. I've asked for a minimal broken example in jemalloc/jemalloc#1636 . Hopefully we can get a better idea of what might go wrong when linking to a DSO containing jemalloc symbols.

@abyss7
Copy link
Copy Markdown
Contributor

abyss7 commented Sep 25, 2019

Tbh, I still don't have a clear mind of all other broken cases, but the way in current PR should work. I've asked for a minimal broken example in jemalloc/jemalloc#1636 . Hopefully we can get a better idea of what might go wrong when linking to a DSO containing jemalloc symbols.

I will make a little investigation today, and will post if I find anything.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is special about this file so that we need to move it in a separate lib?

CMakeLists.txt Outdated
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just link clickhouse_new_delete everywhere?

Copy link
Copy Markdown
Collaborator Author

@amosbird amosbird Sep 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because clickhouse_new_delete provide operator::new/delete symbol interposition in libcxx/stdlibc++, which is similar to malloc. I'd like to make this strategy consistent.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not everywhere

because I assume only executables under dbms require memory tracking.

@amosbird amosbird force-pushed the clang-shared-build-fix branch from 47963f6 to 08bff17 Compare September 27, 2019 00:49
@abyss7 abyss7 merged commit 4cb5309 into ClickHouse:master Sep 27, 2019
abyss7 added a commit that referenced this pull request Sep 27, 2019
abyss7 added a commit that referenced this pull request Sep 27, 2019
@stavrolia stavrolia added the pr-improvement Pull request with some product improvements label Oct 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-improvement Pull request with some product improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants