Skip to content

[fuzz] adding dictionary stream round trip fuzzer#2139

Closed
bimbashrestha wants to merge 3 commits intofacebook:devfrom
bimbashrestha:ldm-fuzz
Closed

[fuzz] adding dictionary stream round trip fuzzer#2139
bimbashrestha wants to merge 3 commits intofacebook:devfrom
bimbashrestha:ldm-fuzz

Conversation

@bimbashrestha
Copy link
Contributor

@bimbashrestha bimbashrestha commented May 14, 2020

This exposes the bug mentioned in:

#2138

$ ./dictionary_stream_round_trip corpora/dictionary_round_trip
...
../../lib/compress/zstdmt_compress.c: ZSTDMT_setBufferSize: bSize = 16236
../../lib/compress/zstd_compress_internal.h: ZSTD_window_update
../../lib/compress/zstd_compress_internal.h: Non contiguous blocks, new segment starts at 1
=================================================================
==25047==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61900000a480 at pc 0x00010a752f24 bp 0x7ffee569e640 sp 0x7ffee569e638
READ of size 1 at 0x61900000a480 thread T0
    #0 0x10a752f23 in ZSTD_rollingHash_append zstd_compress_internal.h:667
    #1 0x10a74c123 in ZSTD_rollingHash_compute zstd_compress_internal.h:677
    #2 0x10a74be53 in ZSTD_ldm_fillHashTable zstd_ldm.c:231
    #3 0x10a872295 in ZSTDMT_serialState_reset zstdmt_compress.c:516
    #4 0x10a86f3bf in ZSTDMT_initCStream_internal zstdmt_compress.c:1512
    #5 0x10a64ab78 in ZSTD_compressStream2 zstd_compress.c:4000
    #6 0x10a97e014 in compress dictionary_stream_round_trip.c:123
    #7 0x10a97cc32 in LLVMFuzzerTestOneInput dictionary_stream_round_trip.c:181
    #8 0x10a97f31a in main regression_driver.c:77
    #9 0x7fff68d3d3d4 in start (libdyld.dylib:x86_64+0x163d4)

0x61900000a480 is located 0 bytes to the right of 1024-byte region [0x61900000a080,0x61900000a480)
allocated by thread T0 here:
    #0 0x10aaee053 in wrap_malloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5c053)
    #1 0x10a56631f in FUZZ_malloc fuzz_helpers.c:19
    #2 0x10a567a22 in FUZZ_train zstd_helpers.c:109
    #3 0x10a97ca34 in LLVMFuzzerTestOneInput dictionary_stream_round_trip.c:178
    #4 0x10a97f31a in main regression_driver.c:77
    #5 0x7fff68d3d3d4 in start (libdyld.dylib:x86_64+0x163d4)

SUMMARY: AddressSanitizer: heap-buffer-overflow zstd_compress_internal.h:667 in ZSTD_rollingHash_append
Shadow bytes around the buggy address:
  0x1c3200001440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3200001450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3200001460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3200001470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c3200001480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1c3200001490:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c32000014a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c32000014b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c32000014c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c32000014d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c32000014e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==25047==ABORTING

@Cyan4973
Copy link
Contributor

Cyan4973 commented May 14, 2020 •
edited
Loading

Several CI tests fail.
I was initially expecting that this is normal, as the new test is supposed to fail while the corresponding fix is not yet merged, yet, is it what happens ?
Looking at the test logs, it's unclear to me. It looks like some kind of a build failure.

@bimbashrestha
Copy link
Contributor Author

bimbashrestha commented May 14, 2020

I think I need to upload the artifacts to our https://github.com/facebook/zstd/releases/fuzz-corpora. I'll also have to merge the bug fix from Nick. Going to do those shortly and try again

@bimbashrestha
Copy link
Contributor Author

bimbashrestha commented May 14, 2020

Okay, uploaded the new fuzzer seed file to https://github.com/facebook/zstd/releases/tag/fuzz-corpora and merging the fix. It should be okay now...hopefully:)

@terrelln
Copy link
Contributor

terrelln commented May 14, 2020

I don't know why CIFuzz is failing. When you run ./fuzz.py build all is dictionary_stream_round_trip built?

@bimbashrestha
Copy link
Contributor Author

bimbashrestha commented May 14, 2020

Hmm nope. It doesn't look like its downloading the new seed file...

@bimbashrestha
Copy link
Contributor Author

bimbashrestha commented May 14, 2020

I don't know why CIFuzz is failing. When you run ./fuzz.py build all is dictionary_stream_round_trip built

Trying now

@bimbashrestha bimbashrestha mentioned this pull request May 14, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bimbashrestha @Cyan4973 @terrelln @facebook-github-bot