doc: document workaround and fallback for macOS fuzzing #32084

l0rinc · 2025-03-17T12:18:38Z

After the recent adjustment to fuzzing docs in #31954, most of the tests started working on macOS again.

A few of them are still failing for weird reasons, let's document how to work around those:

Since libfuzzer-nosan builds to a different folder, I've added the full build steps after configuration.
I've also deleted the brew install llvm duplication, fixed a typo (non-systems clang), and adjusted the fuzzer link for mac in Quickstart guide

DrahtBot · 2025-03-17T12:18:41Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/32084.

Reviews

See the guideline for information on the review process.
A summary of reviews will appear here.

brunoerg · 2025-03-17T12:41:11Z

"Changing the preset to avoid the other sanitizers seems to solve the issue."

It would be good to understand why it doesn't work anymore first.

l0rinc · 2025-03-17T13:19:16Z

Thanks for the hint @brunoerg, I did a bisect to see if it's the commits or the update, which gave me contradictory results.
Clearing all build caches however fixed it for master as well:

ccache -C && git clean -fxd && git reset --hard

Guess we can't rely on cache invalidation here.

hebasto · 2025-03-17T13:33:25Z

Thanks for the hint @brunoerg, I did a bisect to see if it's the commits or the update, which gave me contradictory results. Clearing all build caches however fixed it for master as well:
ccache -C && git clean -fxd && git reset --hard
Guess we can't rely on cache invalidation here.

I'd appreciate it if you could provide steps to reproduce the scenario where Ccache cache causes a build issue.

l0rinc · 2025-03-17T13:39:00Z

I can't, I deleted everything as the above steps show.
But it was likely the local build folder and not ccache (based on how git bisect behaved, i.e. after the first build passed, everything did).

brunoerg · 2025-03-18T13:20:05Z

Thanks for the hint @brunoerg, I did a bisect to see if it's the commits or the update, which gave me contradictory results. Clearing all build caches however fixed it for master as well:
ccache -C && git clean -fxd && git reset --hard
Guess we can't rely on cache invalidation here.
I'd appreciate it if you could provide steps to reproduce the scenario where Ccache cache causes a build issue.

I'm getting the same issue and it works when I explicity set -DAPPEND_LDFLAGS="-fsanitize=..."

e.g.:

cmake --preset=libfuzzer-nosan \
   -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" \
   -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" \
   -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
   -DAPPEND_LDFLAGS="-fsanitize=fuzzer"

l0rinc · 2025-03-25T20:15:15Z

I'm getting the same issue and it works when

It's still not working for me in a few cases, no matter how many caches I delete or how much I go back in time.
And the failures don't seem to make any sense to me:

rm -rfd build_fuzz \
  && cmake --preset=libfuzzer \
  -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" \
  -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" \
  -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
  && cmake --build build_fuzz \
  && FUZZ=psbt_base64_decode build_fuzz/bin/fuzz

which fails immediately with

fuzz(70461,0x1ec764840) malloc: nano zone abandoned due to inability to reserve vm space.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1288419648
INFO: Loaded 1 modules   (1253405 inline 8-bit counters): 1253405 [0x107024000, 0x10715601d), 
INFO: Loaded 1 PC tables (1253405 PCs): 1253405 [0x107156020,0x1084761f0), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
libc++abi: terminating due to uncaught exception of type std::__1::ios_base::failure: DataStream::read(): end of data: unspecified iostream_category error
==70461== ERROR: libFuzzer: deadly signal
    #0 0x00010a3c9248 in __sanitizer_print_stack_trace+0x28 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x5d248)
    #1 0x00010661c544 in fuzzer::PrintStackTrace()+0x2c (fuzz:arm64+0x1037a4544)
    #2 0x00010660fef8 in fuzzer::Fuzzer::CrashCallback()+0x54 (fuzz:arm64+0x103797ef8)
    #3 0x000182e2ade0 in _sigtramp+0x34 (libsystem_platform.dylib:arm64+0x3de0)
    #4 0x9d18800182df3f6c  (<unknown module>)
    #5 0x7c15800182d00904  (<unknown module>)
    #6 0xd303800182daa448  (<unknown module>)
    #7 0x1925000182d98a20  (<unknown module>)
    #8 0x1e31800182a413f0  (<unknown module>)
    #9 0x242b000182da970c  (<unknown module>)
    #10 0x3e28000182daccd8  (<unknown module>)
    #11 0xc62c800182dacc80  (<unknown module>)
    #12 0x7e1e80010309ac20  (<unknown module>)
    #13 0x000103248c84 in void PartiallySignedTransaction::Unserialize<DataStream>(DataStream&)+0x224 (fuzz:arm64+0x1003d0c84)
    #14 0x000103e66ad0 in DecodeRawPSBT(PartiallySignedTransaction&, std::__1::span<std::byte const, 18446744073709551615ul>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)+0x200 (fuzz:arm64+0x100feead0)
    #15 0x000103e6645c in DecodeBase64PSBT(PartiallySignedTransaction&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)+0x1dc (fuzz:arm64+0x100fee45c)
    #16 0x000102ec7a68 in psbt_base64_decode_fuzz_target(std::__1::span<unsigned char const, 18446744073709551615ul>)+0x270 (fuzz:arm64+0x10004fa68)
    #17 0x000103b324d8 in LLVMFuzzerTestOneInput+0x198 (fuzz:arm64+0x100cba4d8)
    #18 0x0001066114a4 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long)+0x12c (fuzz:arm64+0x1037994a4)
    #19 0x000106612614 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, std::__1::allocator<fuzzer::SizedFile>>&)+0x220 (fuzz:arm64+0x10379a614)
    #20 0x000106612c9c in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, std::__1::allocator<fuzzer::SizedFile>>&)+0x98 (fuzz:arm64+0x10379ac9c)
    #21 0x000106609654 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))+0x1dd0 (fuzz:arm64+0x103791654)
    #22 0x00010661cf48 in main+0x24 (fuzz:arm64+0x1037a4f48)
    #23 0x000182a74270  (<unknown module>)
    #24 0x9d3afffffffffffc  (<unknown module>)

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 0 ; base unit: 0000000000000000000000000000000000000000


artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:

Which doesn't make sense since PartiallySignedTransaction::Unserialize in DecodeRawPSBT is called inside a try/catch (const std::exception& e).

Similarly to #32089, --preset=libfuzzer-nosan works with some warnings:

[100%] Built target fuzz
WARNING: Failed to find function "__sanitizer_acquire_crash_state". Reason dlsym(RTLD_DEFAULT, __sanitizer_acquire_crash_state): symbol not found.
WARNING: Failed to find function "__sanitizer_print_stack_trace". Reason dlsym(RTLD_DEFAULT, __sanitizer_print_stack_trace): symbol not found.
WARNING: Failed to find function "__sanitizer_set_death_callback". Reason dlsym(RTLD_DEFAULT, __sanitizer_set_death_callback): symbol not found.
INFO: Running with entropic power schedule (0xFF, 100).

l0rinc · 2025-03-25T20:18:49Z

I'll reopen since others seem to have a similar problem as well - once we figure out what's causing it exactly, we can go back to --preset=libfuzzer on mac as well.

fjahr · 2025-03-26T14:28:12Z

I am currently having similar problems (also on macos). I am not sure if I want to change the docs though, obviously it would be great if we could figure out the actual issue instead.

Maybe a bit more context that I can add, I am seeing the same error as above when running without providing a corpus but with a corpus I see this:

$ FUZZ=base32_encode_decode build_fuzz/bin/fuzz ../qa-assets/fuzz_corpora/base32_encode_decode/
fuzz(54938,0x1fedb8840) malloc: nano zone abandoned due to inability to reserve vm space.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2131471744
INFO: Loaded 1 modules   (1253368 inline 8-bit counters): 1253368 [0x10867c000, 0x1087adff8),
INFO: Loaded 1 PC tables (1253368 PCs): 1253368 [0x1087adff8,0x109acdf78),
=================================================================
==54938==ERROR: AddressSanitizer: container-overflow on address 0x608000000ae8 at pc 0x000104518ef4 bp 0x00016b92e6f0 sp 0x00016b92e6e8
WRITE of size 8 at 0x608000000ae8 thread T0
    #0 0x000104518ef0 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::__init_copy_ctor_external(char const*, unsigned long)+0x1c4 (fuzz:arm64+0x100048ef0)
    #1 0x000107c66314 in fuzzer::ListFilesInDirRecursive(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, long*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>*, bool)+0x26c (fuzz:arm64+0x103796314)
    #2 0x000107c655dc in fuzzer::GetSizedFilesFromDir(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<fuzzer::SizedFile, std::__1::allocator<fuzzer::SizedFile>>*)+0x2c (fuzz:arm64+0x1037955dc)
    #3 0x000107c61154 in fuzzer::ReadCorpora(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)+0x4c (fuzz:arm64+0x103791154)
    #4 0x000107c60fbc in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))+0x1dbc (fuzz:arm64+0x103790fbc)
    #5 0x000107c748c4 in main+0x24 (fuzz:arm64+0x1037a48c4)
    #6 0x0001950c8270  (<unknown module>)
    #7 0xf297ffffffffffc  (<unknown module>)

0x608000000ae8 is located 72 bytes inside of 96-byte region [0x608000000aa0,0x608000000b00)
allocated by thread T0 here:
    #0 0x00010b9952c4 in _Znwm+0x6c (libclang_rt.asan_osx_dynamic.dylib:arm64+0x612c4)
    #1 0x000104a95160 in std::__1::__split_buffer<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>&)+0xf0 (fuzz:arm64+0x1005c5160)
    #2 0x000104decb48 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>* std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>::__push_back_slow_path<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)+0x244 (fuzz:arm64+0x10091cb48)
    #3 0x000107c66284 in fuzzer::ListFilesInDirRecursive(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, long*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>*, bool)+0x1dc (fuzz:arm64+0x103796284)
    #4 0x000107c655dc in fuzzer::GetSizedFilesFromDir(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<fuzzer::SizedFile, std::__1::allocator<fuzzer::SizedFile>>*)+0x2c (fuzz:arm64+0x1037955dc)
    #5 0x000107c61154 in fuzzer::ReadCorpora(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)+0x4c (fuzz:arm64+0x103791154)
    #6 0x000107c60fbc in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))+0x1dbc (fuzz:arm64+0x103790fbc)
    #7 0x000107c748c4 in main+0x24 (fuzz:arm64+0x1037a48c4)
    #8 0x0001950c8270  (<unknown module>)
    #9 0xf297ffffffffffc  (<unknown module>)

HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
SUMMARY: AddressSanitizer: container-overflow (fuzz:arm64+0x100048ef0) in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::__init_copy_ctor_external(char const*, unsigned long)+0x1c4
Shadow bytes around the buggy address:
  0x608000000800: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x608000000880: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x608000000900: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x608000000980: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x608000000a00: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
=>0x608000000a80: fa fa fa fa 00 00 00 00 00 00 00 00 00[fc]fc fc
  0x608000000b00: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x608000000b80: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x608000000c00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x608000000c80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x608000000d00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==54938==ABORTING
Abort trap: 6

So there seems to be some problem with reading the corpus when trying to provide a corpus.

brunoerg · 2025-03-26T16:49:38Z

@fjahr Did you try with ASAN_OPTIONS=detect_container_overflow=0? If so, did you get any issue?

fjahr · 2025-03-27T20:59:57Z

@fjahr Did you try with ASAN_OPTIONS=detect_container_overflow=0? If so, did you get any issue?

I actually hadn't because for what I was working on I didn't need the sanitiziers so I went with nosan right away but I just tried it now and it does indeed fix it for me with this with or without corpus, e.g. this works:

$ cmake --preset=libfuzzer -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld"
$ ASAN_OPTIONS=detect_container_overflow=0 FUZZ=mini_miner_selection build_fuzz/bin/fuzz ../qa-assets/fuzz_corpora/mini_miner_selection/

If that works for you as well @l0rinc maybe that's a better docs change to suggest trying that.

l0rinc · 2025-03-27T21:37:50Z

ASAN_OPTIONS=detect_container_overflow=0 that's a weird syntax - unfortunately that still fails for me.

Details

ASAN_OPTIONS=detect_container_overflow=0 FUZZ=psbt_base64_decode build_fuzz/bin/fuzz

fuzz(61762,0x1fa5e4840) malloc: nano zone abandoned due to inability to reserve vm space.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3061578036
INFO: Loaded 1 modules   (1253375 inline 8-bit counters): 1253375 [0x106344000, 0x106475fff), 
INFO: Loaded 1 PC tables (1253375 PCs): 1253375 [0x106476000,0x107795ff0), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
libc++abi: terminating due to uncaught exception of type std::__1::ios_base::failure: DataStream::read(): end of data: unspecified iostream_category error
==61762== ERROR: libFuzzer: deadly signal
    #0 0x0001097fd248 in __sanitizer_print_stack_trace+0x28 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x5d248)
    #1 0x00010593be34 in fuzzer::PrintStackTrace()+0x2c (fuzz:arm64+0x1037a3e34)
    #2 0x00010592f7e8 in fuzzer::Fuzzer::CrashCallback()+0x54 (fuzz:arm64+0x1037977e8)
    #3 0x000190caade0 in _sigtramp+0x34 (libsystem_platform.dylib:arm64+0x3de0)
    #4 0xd05c000190c73f6c  (<unknown module>)
    #5 0x1644800190b80904  (<unknown module>)
    #6 0x9817000190c2a448  (<unknown module>)
    #7 0xd109000190c18a20  (<unknown module>)
    #8 0xe8378001908c13f0  (<unknown module>)
    #9 0x2d14800190c2970c  (<unknown module>)
    #10 0x2d7a800190c2ccd8  (<unknown module>)
    #11 0xe77800190c2cc80  (<unknown module>)
    #12 0xed628001023bac20  (<unknown module>)
    #13 0x000102568c84 in void PartiallySignedTransaction::Unserialize<DataStream>(DataStream&)+0x224 (fuzz:arm64+0x1003d0c84)
    #14 0x000103186ad0 in DecodeRawPSBT(PartiallySignedTransaction&, std::__1::span<std::byte const, 18446744073709551615ul>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)+0x200 (fuzz:arm64+0x100feead0)
    #15 0x00010318645c in DecodeBase64PSBT(PartiallySignedTransaction&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)+0x1dc (fuzz:arm64+0x100fee45c)
    #16 0x0001021e7a68 in psbt_base64_decode_fuzz_target(std::__1::span<unsigned char const, 18446744073709551615ul>)+0x270 (fuzz:arm64+0x10004fa68)
    #17 0x000102e524d8 in LLVMFuzzerTestOneInput+0x198 (fuzz:arm64+0x100cba4d8)
    #18 0x000105930d94 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long)+0x12c (fuzz:arm64+0x103798d94)
    #19 0x000105931f04 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, std::__1::allocator<fuzzer::SizedFile>>&)+0x220 (fuzz:arm64+0x103799f04)
    #20 0x00010593258c in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, std::__1::allocator<fuzzer::SizedFile>>&)+0x98 (fuzz:arm64+0x10379a58c)
    #21 0x000105928f44 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))+0x1dd0 (fuzz:arm64+0x103790f44)
    #22 0x00010593c838 in main+0x24 (fuzz:arm64+0x1037a4838)
    #23 0x0001908f4270  (<unknown module>)
    #24 0xaa107ffffffffffc  (<unknown module>)

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 0 ; base unit: 0000000000000000000000000000000000000000


artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:

But the libfuzzer-nosan fallback does work, I've documented both, thanks for the hints - coauthored @brunoerg and @fjahr.

brunoerg · 2025-03-27T21:57:00Z

I think we have two different issues here: @l0rinc is getting "libc++abi: terminating due to uncaught exception of type std::__1::ios_base::failure: DataStream::read(): end of data: unspecified iostream_category error". Me and @fjahr were getting the AddressSanitizer: container-overflow one. For both nosan should work, but I think that the container-overflow is a false positive so might be good to document the ASAN_OPTIONS=detect_container_overflow=0.

l0rinc · 2025-03-27T22:01:02Z

might be good to document the ASAN_OPTIONS=detect_container_overflow=0

Already pushed, please review the changes.

fjahr · 2025-03-27T22:01:57Z

Yeah, I do still get the error that @l0rinc sees when I test psbt_base64_decode. When I test mini_miner_selection which I am working on then it is fixed with the ASAN option with and without corpus.

fjahr

Left some comments but I am still ~0 on this change and would much rather prefer that someone tries to spend more time on investigating the issue. I tried for a bit but wasn't successful so far.

doc/fuzzing.md

brunoerg · 2025-03-27T22:15:17Z

If we're going to put ASAN_OPTIONS=detect_container_overflow=0 into the documentation, I think it would be good to mention https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow#false-positives

l0rinc · 2025-03-27T22:21:40Z

it would be good to mention Wiki: AddressSanitizerContainerOverflow (false positives) (google/sanitizers)

I've put this in the description already, but added it to the doc as well now

maflcko · 2025-03-28T07:18:34Z

Left some comments but I am still ~0 on this change and would much rather prefer that someone tries to spend more time on investigating the issue. I tried for a bit but wasn't successful so far.

I tend to agree. It would be good to check:

Does it happen with a self-compiled clang. If yes, then it is likely an upstream brew issues. If no, then it is likely an upstream clang/llvm issue.

Unrelated to that, it would be good to minimize the DataStream crash, so that there is a single minimal main.cpp file (or two files) that reproduce it. This makes it easier to spot if the error is in this codebase or somewhere else.

l0rinc · 2025-04-07T12:29:48Z

rm -rfd build_fuzz \
 && cmake --preset=libfuzzer \
   -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" \
   -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" \
   -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
 && cmake --build build_fuzz \
 && FUZZ=psbt_base64_decode build_fuzz/bin/fuzz

Retried it with with latest Clang 20.1.2, but now it doesn't even compile anymore.

Details

In file included from <...>/bitcoin/src/wallet/coincontrol.cpp:5:
...
In file included from /opt/homebrew/Cellar/llvm/20.1.2/bin/../include/c++/v1/cstdint:149:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdint.h:52:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/_types.h:33:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/machine/_types.h:34:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:16:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/_types/_ptrdiff_t.h:41:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stddef.h:64:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_types.h:46:9: error: unknown type name '__uint32_t'
   46 | typedef __uint32_t      __darwin_wctype_t;
      |         ^

... (similar error for '__darwin_wint_t' in _wint_t.h) ...

In file included from <...>/bitcoin/src/wallet/coincontrol.cpp:5:
...
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/machine/_types.h:34:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:75:9: error: unknown type name 'ptrdiff_t'
   75 | typedef ptrdiff_t               __darwin_ptrdiff_t;     /* ptr1 - ptr2 */
      |         ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/arm/_types.h:85:9: error: unknown type name 'size_t'
   85 | typedef size_t                  __darwin_size_t;        /* sizeof() */
      |         ^

... (many subsequent errors for 'size_t' in stdlib.h, malloc/_malloc_type.h, etc.) ...

fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.

ccache -C && git clean -fxd && git reset --hard doesn't help.

sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
xcodebuild -runFirstLaunch

didn't help either.

Is it working for anyone else?

brunoerg · 2025-04-07T13:39:42Z

rm -rfd build_fuzz \
 && cmake --preset=libfuzzer \
   -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" \
   -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" \
   -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
 && cmake --build build_fuzz \
 && FUZZ=psbt_base64_decode build_fuzz/bin/fuzz
Retried it with with latest Clang 20.1.2, but now it doesn't even compile anymore.

Details
ccache -C && git clean -fxd && git reset --hard doesn't help.
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
xcodebuild -runFirstLaunch
didn't help either.

Is it working for anyone else?

I've tried with this exact command and it compiled, but I got the following error:

fuzz(27696,0x1e41a5c40) malloc: nano zone abandoned due to inability to reserve vm space.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1403259666
INFO: Loaded 1 modules   (1308295 inline 8-bit counters): 1308295 [0x1049ac000, 0x104aeb687),
INFO: Loaded 1 PC tables (1308295 PCs): 1308295 [0x104aeb688,0x105ee1ef8),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
libc++abi: terminating due to uncaught exception of type std::__1::ios_base::failure: DataStream::read(): end of data: unspecified iostream_category error
==27696== ERROR: libFuzzer: deadly signal
    #0 0x0001079eacd4 in __sanitizer_print_stack_trace+0x28 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x5ecd4)
    #1 0x000103f695e4 in fuzzer::PrintStackTrace()+0x2c (fuzz:arm64+0x1039ad5e4)
    #2 0x000103f5c8d0 in fuzzer::Fuzzer::CrashCallback()+0x54 (fuzz:arm64+0x1039a08d0)
    #3 0x00018d909a20 in _sigtramp+0x34 (libsystem_platform.dylib:arm64+0x3a20)
    #4 0xec7e80018d8d9cbc  (<unknown module>)
    #5 0xd05e80018d7e5a3c  (<unknown module>)
    #6 0x7d4b80018d89106c  (<unknown module>)
    #7 0xe06080018d88110c  (<unknown module>)
    #8 0x5e4180018d527998  (<unknown module>)
    #9 0x210180018d890430  (<unknown module>)
    #10 0xe33180018d89351c  (<unknown module>)
    #11 0xfb1200018d893460  (<unknown module>)
    #12 0xc668001007dfa88  (<unknown module>)
    #13 0x0001009a2d08 in void PartiallySignedTransaction::Unserialize<DataStream>(DataStream&)+0x228 (fuzz:arm64+0x1003e6d08)
    #14 0x0001016603d0 in DecodeRawPSBT(PartiallySignedTransaction&, std::__1::span<std::byte const, 18446744073709551615ul>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)+0x20c (fuzz:arm64+0x1010a43d0)
    #15 0x00010165fd50 in DecodeBase64PSBT(PartiallySignedTransaction&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)+0x1dc (fuzz:arm64+0x1010a3d50)
    #16 0x00010060f190 in psbt_base64_decode_fuzz_target(std::__1::span<unsigned char const, 18446744073709551615ul>)+0x270 (fuzz:arm64+0x100053190)
    #17 0x000101311890 in LLVMFuzzerTestOneInput+0x1bc (fuzz:arm64+0x100d55890)
    #18 0x000103f5df64 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long)+0x134 (fuzz:arm64+0x1039a1f64)
    #19 0x000103f5f330 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, std::__1::allocator<fuzzer::SizedFile>>&)+0x220 (fuzz:arm64+0x1039a3330)
    #20 0x000103f5f9b4 in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, std::__1::allocator<fuzzer::SizedFile>>&)+0x98 (fuzz:arm64+0x1039a39b4)
    #21 0x000103f562b8 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))+0x1ee8 (fuzz:arm64+0x10399a2b8)
    #22 0x000103f69fe4 in main+0x24 (fuzz:arm64+0x1039adfe4)
    #23 0x00018d5590dc  (<unknown module>)
    #24 0xd1437ffffffffffc  (<unknown module>)

Homebrew clang version 20.1.2
Target: arm64-apple-darwin23.3.0
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/llvm/20.1.2/bin
Configuration file: /opt/homebrew/etc/clang/arm64-apple-darwin23.cfg

l0rinc · 2025-04-23T19:44:08Z

it would be good to mention Wiki: AddressSanitizerContainerOverflow (false positives) (google/sanitizers)

Added the links to the doc and added another workaround of setting the target architecture an Apple Silicon for errors like (see #32084 (comment) for details):

 usr/include/malloc/_malloc_type.h:66:126: error: unknown type name 'size_t'; did you mean 'std::size_t'?

DrahtBot · 2025-04-23T19:48:17Z

🚧 At least one of the CI tasks failed.
_{Debug: https://github.com/bitcoin/bitcoin/runs/41039901383}

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the
affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

doc/fuzzing.md

On macOS, running fuzz targets with the default `libfuzzer` preset can fail due to linker errors or AddressSanitizer or header incompatibility issues such as: * Linker did not accept requested flags, you are missing required libraries * ==54938==ERROR: AddressSanitizer: container-overflow on address 0x608000000ae8 at pc 0x000104518ef4 bp 0x00016b92e6f0 sp 0x00016b92e6e8 Documented a workaround using: * `ASAN_OPTIONS=detect_container_overflow=0` - see https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow#false-positives; * `target arm64-apple-macos11` - see https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary#Update-the-Architecture-List-of-Custom-Makefiles * `--preset=libfuzzer-nosan` - see https://github.com/bitcoin/bitcoin/blob/master/CMakePresets.json#L50-L60 Since `libfuzzer-nosan` builds to a different folder, I've added the full build steps after configuration. I've also deleted the `brew install llvm` duplication a few lines below, fixed a typo (`non-systems clang`), removed the trailing `$` in the Mac examples to make the code easily copyable (and since Macs don't usually have that in the console) and adjusted the fuzzer link for Mac in `Quickstart guide`. Co-authored-by: brunoerg <[email protected]> Co-authored-by: Fabian Jahr <[email protected]>

l0rinc · 2025-10-02T14:28:35Z

For future me, wondering how to make fuzzing work on a Mac with latest Clang + AppleClang - this is the version that mostly works for me (though it fails in a lot of places where Linux doesn't):

rm -rfd build_fuzz && cmake --preset=libfuzzer \
   -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" \
   -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" \
   -DCMAKE_OSX_SYSROOT="$(xcrun --show-sdk-path)" \
   -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld -L$(brew --prefix llvm)/lib/c++ -Wl,-rpath,$(brew --prefix llvm)/lib/c++" && \
  cmake --build build_fuzz -j$(nproc) && \
  FUZZ=coins_view_db build_fuzz/bin/fuzz

Example failure on master

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3345256079
INFO: Loaded 1 modules   (1169147 inline 8-bit counters): 1169147 [0x107f94000, 0x1080b16fb), 
INFO: Loaded 1 PC tables (1169147 PCs): 1169147 [0x1080b1700,0x1092886b0), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
libc++abi: terminating due to uncaught exception of type std::logic_error: CCoinsViewCache cursor iteration not supported.
==85075== ERROR: libFuzzer: deadly signal
    #0 0x00010aaab334 in __sanitizer_print_stack_trace+0x28 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x5f334)
    #1 0x0001075cf190 in fuzzer::PrintStackTrace()+0x2c (fuzz:arm64+0x103517190)
    #2 0x0001075c2ed0 in fuzzer::Fuzzer::CrashCallback()+0x54 (fuzz:arm64+0x10350aed0)
    #3 0x00019e99a740 in _sigtramp+0x34 (libsystem_platform.dylib:arm64+0x3740)
    #4 0x00019e990884 in pthread_kill+0x124 (libsystem_pthread.dylib:arm64+0x6884)
    #5 0x00019e896804 in abort+0x78 (libsystem_c.dylib:arm64+0x79804)
    #6 0x00010a4d03ec in __abort_message+0x58 (libc++abi.1.0.dylib:arm64+0x283ec)
    #7 0x00010a4a94e8 in demangling_terminate_handler()+0xf0 (libc++abi.1.0.dylib:arm64+0x14e8)
    #8 0x00010a4ca738 in std::__terminate(void (*)())+0x8 (libc++abi.1.0.dylib:arm64+0x22738)
    #9 0x00010a4cf004 in __cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*)+0x1c (libc++abi.1.0.dylib:arm64+0x27004)
    #10 0x00010a4cefe4 in __cxa_throw+0x54 (libc++abi.1.0.dylib:arm64+0x26fe4)
    #11 0x0001043b9c64 in TestCoinsView(FuzzedDataProvider&, CCoinsView&, bool)+0x3d34 (fuzz:arm64+0x100301c64)
    #12 0x0001043becc0 in coins_view_db_fuzz_target(std::__1::span<unsigned char const, 18446744073709551615ul>)+0x2f4 (fuzz:arm64+0x100306cc0)
    #13 0x000104e8ed1c in LLVMFuzzerTestOneInput+0x230 (fuzz:arm64+0x100dd6d1c)
    #14 0x0001075c4564 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long)+0x134 (fuzz:arm64+0x10350c564)
    #15 0x0001075c5928 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, std::__1::allocator<fuzzer::SizedFile>>&)+0x220 (fuzz:arm64+0x10350d928)
    #16 0x0001075c5fac in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, std::__1::allocator<fuzzer::SizedFile>>&)+0x98 (fuzz:arm64+0x10350dfac)
    #17 0x0001075bc8b0 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))+0x1ed8 (fuzz:arm64+0x1035048b0)
    #18 0x0001075cfb90 in main+0x24 (fuzz:arm64+0x103517b90)
    #19 0x00019e5d1d50 in start+0x1c0c (dyld:arm64+0x3d50)

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 0 ; base unit: 0000000000000000000000000000000000000000


artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:

and a nosan version:

 rm -rfd build_fuzz_nosan && cmake --preset=libfuzzer-nosan \
   -DCMAKE_C_COMPILER="$(brew --prefix llvm)/bin/clang" \
   -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++" \
   -DCMAKE_OSX_SYSROOT="$(xcrun --show-sdk-path)" \
   -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld -L$(brew --prefix llvm)/lib/c++ -Wl,-rpath,$(brew --prefix llvm)/lib/c++" && \
  cmake --build build_fuzz_nosan -j$(nproc) && \
  FUZZ=coins_view_db build_fuzz_nosan/bin/fuzz

DrahtBot added the Docs label Mar 17, 2025

l0rinc closed this Mar 17, 2025

l0rinc deleted the l0rinc/libfuzzer-nosan-mac branch March 17, 2025 13:39

maflcko mentioned this pull request Mar 18, 2025

[rfc] build: Reject unclean configure? #31942

Open

l0rinc restored the l0rinc/libfuzzer-nosan-mac branch March 25, 2025 20:16

l0rinc reopened this Mar 25, 2025

l0rinc force-pushed the l0rinc/libfuzzer-nosan-mac branch from 0bc12d1 to 3bef6c4 Compare March 25, 2025 20:39

l0rinc mentioned this pull request Mar 25, 2025

fuzz: psbt_base64_decode fails on Windows #32135

Closed

l0rinc force-pushed the l0rinc/libfuzzer-nosan-mac branch from 3bef6c4 to 8b7c8a2 Compare March 27, 2025 21:37

l0rinc changed the title ~~doc: use libfuzzer-nosan for macOS~~ doc: document workaround and fallback for macOS fuzzing Mar 27, 2025

fjahr reviewed Mar 27, 2025

View reviewed changes

doc/fuzzing.md Outdated Show resolved Hide resolved

doc/fuzzing.md Outdated Show resolved Hide resolved

doc/fuzzing.md Outdated Show resolved Hide resolved

l0rinc force-pushed the l0rinc/libfuzzer-nosan-mac branch from 8b7c8a2 to 89477d0 Compare March 27, 2025 22:21

l0rinc force-pushed the l0rinc/libfuzzer-nosan-mac branch from 89477d0 to 4a6a649 Compare April 23, 2025 19:43

l0rinc force-pushed the l0rinc/libfuzzer-nosan-mac branch from 4a6a649 to d01a1a9 Compare April 23, 2025 19:48

DrahtBot added the CI failed label Apr 23, 2025

l0rinc force-pushed the l0rinc/libfuzzer-nosan-mac branch from d01a1a9 to dc3a0d8 Compare April 23, 2025 19:58

maflcko reviewed Apr 24, 2025

View reviewed changes

doc/fuzzing.md Outdated Show resolved Hide resolved

l0rinc force-pushed the l0rinc/libfuzzer-nosan-mac branch from dc3a0d8 to b9f54e1 Compare April 25, 2025 16:55

DrahtBot removed the CI failed label Apr 25, 2025

l0rinc closed this May 13, 2025

l0rinc mentioned this pull request Oct 2, 2025

build: Drop support for EOL macOS 13 #33489

Merged

maflcko mentioned this pull request Oct 21, 2025

Unable to fuzz in local on MacOS 15.4.1 #33667

Closed

l0rinc mentioned this pull request Oct 29, 2025

RFC: Do we want to support fuzzing on MacOS? #33731

Closed

l0rinc mentioned this pull request Nov 21, 2025

doc: clarify and cleanup macOS fuzzing notes #33921

Merged

doc: document workaround and fallback for macOS fuzzing #32084

doc: document workaround and fallback for macOS fuzzing #32084

Uh oh!

Conversation

l0rinc commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage & Benchmarks

Reviews

Uh oh!

brunoerg commented Mar 17, 2025

Uh oh!

l0rinc commented Mar 17, 2025

Uh oh!

hebasto commented Mar 17, 2025

Uh oh!

l0rinc commented Mar 17, 2025

Uh oh!

brunoerg commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

l0rinc commented Mar 25, 2025

Uh oh!

l0rinc commented Mar 25, 2025

Uh oh!

fjahr commented Mar 26, 2025

Uh oh!

brunoerg commented Mar 26, 2025

Uh oh!

fjahr commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

l0rinc commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brunoerg commented Mar 27, 2025

Uh oh!

l0rinc commented Mar 27, 2025

Uh oh!

fjahr commented Mar 27, 2025

Uh oh!

fjahr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brunoerg commented Mar 27, 2025

Uh oh!

l0rinc commented Mar 27, 2025

Uh oh!

maflcko commented Mar 28, 2025

Uh oh!

l0rinc commented Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brunoerg commented Apr 7, 2025

Uh oh!

l0rinc commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Apr 23, 2025

Uh oh!

Uh oh!

l0rinc commented Oct 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

l0rinc commented Mar 17, 2025 •

edited

Loading

DrahtBot commented Mar 17, 2025 •

edited

Loading

brunoerg commented Mar 18, 2025 •

edited

Loading

fjahr commented Mar 27, 2025 •

edited

Loading

l0rinc commented Mar 27, 2025 •

edited

Loading

l0rinc commented Apr 7, 2025 •

edited

Loading

l0rinc commented Apr 23, 2025 •

edited

Loading