-
Notifications
You must be signed in to change notification settings - Fork 173
fix(ci): Skip flaky ASAN failures in Meson #2604
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pending CI success, i like it 😄
|
Seems we need to set more (maybe both CXX and CC have to be set?) |
|
I asked the Meson team about this on their IRC channel, and they mentioned that while gcc uses shared linkage to the ASAN runtime, clang requires static linkage to ASAN. It seems like something is awry with the latter, and it might have to do with how gtest is being used. I will have to investigate that further, or maybe see if there's a newer version of gcc that can be used instead of clang that still fixes the issue |
|
After some more research, the patch that was applied to clang had already been in place with gcc since 2013. So I think I erred in assuming a newer version would change this. Need to go back to the drawing board... |
|
For now I've given up on trying to solve this, instead skipping the flightsql and bigquery tests in ASAN where I think the problem arises |
|
I do wonder if this affects snowflake as well, since bigquery, flightsql and snowflake are all built using custom go targets... |
lidavidm
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did find the sanitizer was flaky here without this config (albeit in a different way)
arrow-adbc/.github/workflows/nightly-verify.yml
Lines 209 to 215 in fda0b2b
| - name: python-debug | |
| run: | | |
| # Need to set this or ASAN inside the container gets stuck | |
| # printing a loop of DEADLYSIGNAL | |
| sudo sysctl vm.mmap_rnd_bits=28 | |
| pushd arrow-adbc | |
| docker compose run -e PYTHON=3.13 --rm python-debug |
Could this be related?
|
From personal experience I've found ASAN challenging to use when launched via the Python interpreter. ASAN itself will detect some leaks from most version of CPython, so the only way I have gotten it to reliably run is to turn off leak detection while also making sure to LD_PRELOAD the ASAN library (this is required unless the Python interpreter itself was built with ASAN support) ASAN_OPTIONS=detect_leaks=0 LD_PRELOAD="$(gcc -print-file-name=libasan.so)" python ...Without that, I have seen that issue where you end up with an endless loop of ASAN_ERROR:DeadlySignal (or something to the effect) which is the error you are seeing right? |
|
Yeah, we do turn off leak checking and preload ASAN, but I found that I also needed to change the ASLR randomization bits to get ASAN to work consistently |
Per comment #2546 (comment) this might help fix the flaky CI failures with meson + ASAN