Skip to content

Output afl-clang-fast stuffs only if necessary #1912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jan 4, 2024

Conversation

SonicStark
Copy link
Contributor

TL;DR: If afl-clang-fast/afl-clang-fast++ were always there, calling them may finally fall back into gcc rather than clang, which is confusing.

When gcc and clang is available in current system while llvm-config not, building AFLplusplus can still be OK and afl-gcc, afl-clang and afl-clang-fast can be found there. However if afl-clang-fast were called after that, weird things would happen:

AFLplusplus/src/afl-cc.c

Lines 1578 to 1611 in 61e27c6

#if (LLVM_MAJOR >= 3)
if (strncmp(callname, "afl-clang-fast", 14) == 0) {
compiler_mode = LLVM;
} else if (strncmp(callname, "afl-clang-lto", 13) == 0 ||
strncmp(callname, "afl-lto", 7) == 0) {
compiler_mode = LTO;
} else
#endif
if (strncmp(callname, "afl-gcc-fast", 12) == 0 ||
strncmp(callname, "afl-g++-fast", 12) == 0) {
compiler_mode = GCC_PLUGIN;
} else if (strncmp(callname, "afl-gcc", 7) == 0 ||
strncmp(callname, "afl-g++", 7) == 0) {
compiler_mode = GCC;
} else if (strcmp(callname, "afl-clang") == 0 ||
strcmp(callname, "afl-clang++") == 0) {
compiler_mode = CLANG;
}

Here strncmp(callname, "afl-clang-fast", 14) would never work and compiler_mode would keep UNSET, because LLVM_MAJOR is empty without a valid llvm-config:

LLVM_MAJOR = $(shell $(LLVM_CONFIG) --version 2>/dev/null | sed 's/\..*//' )
override CXXFLAGS += -Wall -g -I ./include/ \
-DVERSION=\"$(VERSION)\" -Wno-variadic-macros -Wno-deprecated-copy-with-dtor \
-DLLVM_MINOR=$(LLVM_MINOR) -DLLVM_MAJOR=$(LLVM_MAJOR)

Then if env AFL_CC_COMPILER and AFL_LLVM_* is not set and --afl-... options are not used, afl-cc can bypass

AFLplusplus/src/afl-cc.c

Lines 2035 to 2041 in 61e27c6

if (instrument_opt_mode && instrument_mode == INSTRUMENT_DEFAULT &&
(compiler_mode == LLVM || compiler_mode == UNSET)) {
instrument_mode = INSTRUMENT_CLASSIC;
compiler_mode = LLVM;
}
and go to

AFLplusplus/src/afl-cc.c

Lines 2043 to 2063 in 61e27c6

if (!compiler_mode) {
// lto is not a default because outside of afl-cc RANLIB and AR have to
// be set to LLVM versions so this would work
if (have_llvm)
compiler_mode = LLVM;
else if (have_gcc_plugin)
compiler_mode = GCC_PLUGIN;
else if (have_gcc)
#ifdef __APPLE__
// on OSX clang masquerades as GCC
compiler_mode = CLANG;
#else
compiler_mode = GCC;
#endif
else if (have_lto)
compiler_mode = LTO;
else
FATAL("no compiler mode available");
}

Obviously have_gcc is 1 because we have as confirmed here:

AFLplusplus/src/afl-cc.c

Lines 1542 to 1547 in 61e27c6

if ((ptr = find_object("as", argv[0])) != NULL) {
have_gcc = 1;
ck_free(ptr);
}

That is to say, afl-clang-fast would finally run like afl-gcc. Isn't this counter-intuitive? I have met several people who used AFL++ for the first time, and in the above situation, they instrumented their programs and found that the target binary was very different from the one build by clang or afl-clang (especially in the linking of runtimes). This did make them confused for quite a while:joy:

@vanhauser-thc
Copy link
Member

I get your point and agree.
People execute afl-clang-fast and what is actually executed underneath could be something different.
afl-cc should be so flexible to do that, but afl-clang-fast, afl-clang-lto, afl-gcc-fast, afl-gcc and afl-clang should be doing what their name is.
would you mind changing your PR so that instead afl-clang-fast is created if afl-llvm-pass.so is present, and afl-gcc-fast if afl-gcc-pass.so?
and if you want to do the extra effort, change afl-cc to bail if afl-clang-fast is the executable name but the necessary pass is not there, and same for afl-gcc-fast, and afl-gcc baling if gcc is not there and afl-clang if there is no clang?

@andreafioraldi
Copy link
Member

afl-clang-fast can be always used in NATIVE if clang is found btw

@vanhauser-thc
Copy link
Member

@SonicStark ping

@SonicStark
Copy link
Contributor Author

SonicStark commented Dec 6, 2023

So sorry for this late replay.

afl-clang-fast can be always used in NATIVE if clang is found btw

@andreafioraldi was right. When LLVM_MAJOR < 4, afl-clang-fast should not be built. And when LLVM_MAJOR >= 4, it can be always there, with compiler_mode = CLANG as its last fallback, so if the built passes are all not available, afl-cc would go to

AFLplusplus/src/afl-cc.c

Lines 2062 to 2071 in ef706ad

/* if our PCGUARD implementation is not available then silently switch to
native LLVM PCGUARD */
if (compiler_mode == CLANG &&
(instrument_mode == INSTRUMENT_DEFAULT ||
instrument_mode == INSTRUMENT_PCGUARD) &&
find_object("SanitizerCoveragePCGUARD.so", argv[0]) == NULL) {
instrument_mode = INSTRUMENT_LLVMNATIVE;
}
and work as expected. I’ll push a new commit for this later.

would you mind changing your PR so that instead afl-clang-fast is created if afl-llvm-pass.so is present, and afl-gcc-fast if afl-gcc-pass.so?

As for afl-gcc-fast:

./afl-gcc-pass.so: instrumentation/afl-gcc-pass.so.cc | test_deps
$(CXX) $(CXXEFLAGS) $(PLUGIN_FLAGS) -shared $< -o $@
ln -sf afl-cc afl-gcc-fast
ln -sf afl-cc afl-g++-fast
ln -sf afl-cc.8 afl-gcc-fast.8
ln -sf afl-cc.8 afl-g++-fast.8
test_deps:
@echo "[*] Checking for working '$(CC)'..."
@command -v $(CC) >/dev/null 2>&1 || ( echo "[-] Oops, can't find '$(CC)'. Make sure that it's in your \$$PATH (or set \$$CC and \$$CXX)."; exit 1 )
# @echo "[*] Checking for gcc for plugin support..."
# @$(CC) -v 2>&1 | grep -q -- --enable-plugin || ( echo "[-] Oops, this gcc has not been configured with plugin support."; exit 1 )
@echo "[*] Checking for gcc plugin development header files..."
@test -d `$(CC) -print-file-name=plugin`/include || ( echo "[-] Oops, can't find gcc header files. Be sure to install 'gcc-X-plugin-dev'."; exit 1 )

It is created just after afl-gcc-pass.so, and both of them depend on target test_deps, which would run exit 1 if current gcc is not suitable. So that’s not a problem.

if you want to do the extra effort, change afl-cc to bail if afl-clang-fast is the executable name but the necessary pass is not there, and same for afl-gcc-fast, and afl-gcc baling if gcc is not there and afl-clang if there is no clang?

More graceful sanity checks and fallback actions is needed in afl-cc, and I’m glad to do sth for this, maybe in a new PR, if needed.
But I'm afraid the problem goes beyond that.
Recently during my off hours, I have trying to bring LTO mode to aflgo, hoping this can solve parallel compiling, multiple binary targets, extra-long path, and some other issues. So I read the source code of afl-cc.c first (still in progress, notes & updates locate in afl-cc-rtfsc), which is really a laborious deal, and some hidden issues came to the light one after another. For example:

if (have_instr_list) {
uses have_instr_list, but actually it is set as

AFLplusplus/src/afl-cc.c

Lines 469 to 471 in ef706ad

if (!strncmp(cur, "-fsanitize-coverage-", 20) && strstr(cur, "list=")) {
have_instr_list = 1;
when and only when process_params being called at
process_params(argc, argv);
i.e. later than the using. have_c has the same issue, used at
if (lto_mode && !have_c) {
but set later until running to
if (!strcmp(cur, "-c")) have_c = 1;
The above two can still be seen in latest dev branch. So I suspect that there may be some problems sneaking in current LTO/NATIVE mode, but I haven’t seen any issues reported about this.
One more example, the following two blocks do the same thing, and the later will be executed for sure. So they may be duplicate?

AFLplusplus/src/afl-cc.c

Lines 1128 to 1137 in ef706ad

if (lto_mode && argc > 1) {
u32 idx;
for (idx = 1; idx < argc; idx++) {
if (!strncasecmp(argv[idx], "-fpic", 5)) have_pic = 1;
}
}

AFLplusplus/src/afl-cc.c

Lines 398 to 407 in ef706ad

if (lto_mode && argc > 1) {
u32 idx;
for (idx = 1; idx < argc; idx++) {
if (!strncasecmp(argv[idx], "-fpic", 5)) have_pic = 1;
}
}

As far as I'm concerned, afl-cc.c is becoming another hulk like afl-fuzz.c in old AFL. So I hope to contribute more to afl-cc, preventing things from getting worse, but it may take a lot of efforts.

@vanhauser-thc
Copy link
Member

true. just pushed fixes for that

@SonicStark
Copy link
Contributor Author

UPDATE: I spent a few days to figure out how afl-cc works. And I have refactored it based on v4.08c, with 50 commits or so, in afl-cc-rtfsc/release-4.08c-dat. The patch for v4.08c is here: afl-cc-changes.patch

It seems that the new afl-cc works well—at least seeming good, with tests passed (ASAN_BUILD=1, both llvm-12 & llvm-16):
image

Features overview:

  • Split afl-cc.c into 8 more modules (could be more compact perhaps).
  • Hierarchical args parsing and modes choosing.
  • Safe built object finding.
  • Standalone objects linking.
  • Smoothly fix the messes mentioned before.

If you guys have interested in this, I'll push some commits in this PR to migrate the changes to dev branch. BTW I plan to run more regression tests if possible.

🎉Happy new year!!!

 - Add afl-cc-state.c
 - Strip includes, find_object, debug/be_quiet/have_*/callname setting from afl-cc.c
 - Use debugf_args in main
 - Modify execvp stuffs to fit new aflcc struct
1. compiler_mode by callname in argv[0]
2. compiler_mode by env "AFL_CC_COMPILER"
3. compiler_mode/instrument_mode by command line options "--afl-..."
4. instrument_mode/compiler_mode by various env vars including "AFL_LLVM_INSTRUMENT"
5. final checking steps
6. print "... - mode: %s-%s\n"
7. determine real argv[0] according to compiler_mode
@vanhauser-thc
Copy link
Member

can you please not split it up into multiple files? I know it is a huge file, but we try to keep all tools in a single file, with the exception of afl-fuzz.

@SonicStark
Copy link
Contributor Author

Oops. I'll merge them finally.
BTW I feel so lucky to have a nice big monitor on my desk🙃

SonicStark added a commit to SonicStark/afl-cc-rtfsc that referenced this pull request Jan 2, 2024
formated with custom-format.py
@SonicStark
Copy link
Contributor Author

Looks working well. Further regression tests is needed to be more convincing.

@vanhauser-thc
Copy link
Member

thanks! I will take a close look at this the next days

@vanhauser-thc
Copy link
Member

LGTM, thank you

@vanhauser-thc vanhauser-thc merged commit aad9ac2 into AFLplusplus:dev Jan 4, 2024
@SonicStark SonicStark deleted the dev branch January 5, 2024 00:45
vanhauser-thc added a commit that referenced this pull request Jan 18, 2024
* Output afl-clang-fast stuffs only if necessary (#1912)

* afl-cc header

* afl-cc common declarations

 - Add afl-cc-state.c
 - Strip includes, find_object, debug/be_quiet/have_*/callname setting from afl-cc.c
 - Use debugf_args in main
 - Modify execvp stuffs to fit new aflcc struct

* afl-cc show usage

* afl-cc mode selecting

1. compiler_mode by callname in argv[0]
2. compiler_mode by env "AFL_CC_COMPILER"
3. compiler_mode/instrument_mode by command line options "--afl-..."
4. instrument_mode/compiler_mode by various env vars including "AFL_LLVM_INSTRUMENT"
5. final checking steps
6. print "... - mode: %s-%s\n"
7. determine real argv[0] according to compiler_mode

* afl-cc macro defs

* afl-cc linking behaviors

* afl-cc fsanitize behaviors

* afl-cc misc

* afl-cc body update

* afl-cc all-in-one

formated with custom-format.py

* nits

---------

Co-authored-by: vanhauser-thc <vh@thc.org>

* changelog

* update grammar mutator

* lto llvm 12+

* docs(custom_mutators): fix missing ':' (#1953)

* Fix broken LTO mode and response file support (#1948)

* Strip `-Wl,-no-undefined` during compilation (#1952)

Make the compiler wrapper stripping `-Wl,-no-undefined` in addition to `-Wl,--no-undefined`.
Both versions of the flag are accepted by clang and, therefore, used by building systems in the wild (e.g., samba will not build without this fix).

* Remove dead code in write_to_testcase (#1955)

The custom_mutators_count check in if case is duplicate with if condition.
The else case is custom_mutators_count == 0, neither custom_mutator_list iteration nor sent check needed.

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>

* update qemuafl

* WIP: Add ability to generate drcov trace using QEMU backend (#1956)

* Document new drcov QEMU plugin

* Add link to lightkeeper for QEMU drcov file loading

---------

Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>

* code format

* changelog

* sleep on uid != 0 afl-system-config

* fix segv about skip_next, warn on unsupported cases of linking options (#1958)

* todos

* ensure afl-cc only allows available compiler modes

* update grammar mutator

* disable aslr on apple

* fix for arm64

---------

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Sonic <50692172+SonicStark@users.noreply.github.com>
Co-authored-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Nils Bars <nils.bars@rub.de>
Co-authored-by: Jean-Romain Garnier <7504819+JRomainG@users.noreply.github.com>
Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>
vanhauser-thc added a commit that referenced this pull request Jan 20, 2024
* Pure Python (3.6) port of benchmark.sh as benchmark.py, no other changes

* Test standard and persistent modes separately

* Add support for multi-core benchmarking

* Save the results to a json file

* Allow config of all experiment params, average across runs

* Add start_time_of_run and total_execs_per_sec, cleanup for PR

* benchmark: cleanup, add results, add a data exploration notebook

* benchmark: add a README, lower default runs from 5 to 3

* benchmark: notebook wording tweaks

* copy 'detect_leaks=0' from ASAN to LSAN

fix for issue #1733, set "detect_leaks=0" when ASAN_OPTIONS contains it and LSAN_OPTIONS are not set.

* fix of fix: make sure ASAN_OPTIONS and LSAN_OPTIONS agree on leak detection

* fix lsan fix

* clang-format 16->17

* Add missing initialisation for havoc_queued during the custom mutator's stage.

* fix dictionary and cmin

* Use direct call to write to OpenBSD

The linker on OpenBSD emits a warning when linking this file:
warning: syscall() may go away, please rewrite code to use direct calls

* Fix possible doc inconsistency for custom mutator's queue_get function.

* update todos

* benchmark: Add support for COMPARISON file

* benchmark: show the number of cores used in COMPARISON

* benchmark: lower minimum Python version to 3.8

* benchmark: use afl's execs/s; increase CPU model width

* benchmark: disallow duplicate entries for the same CPU in COMPARISON

* Update benchmark.py

* fix inf in stats

* Fix benchmark.py

* missing closing parenthesis

* Update benchmark.py

* benchmark: remove self-calculation of execs/sec

* benchmark: update COMPARISON

* benchmark: Update Jupyter notebook and results file.

* benchmark: rename afl_execs_per_sec to execs_per_sec

* benchmark: update README

* update

* add benchmark

* nits

* add benchmarks

* Update unicornafl ref

* Pass correct Nyx ID when creating a Nyx runner

* Fix typo in docker pull command, add exampe to mount current dir as volume (#1914)

* mini fix

* add custom_post_run.c

* update afl-fuzz-run

* update python module

* format code

* update

* merge function

* changes

* code format

* improve cmplog

* nit

* nit

* fix

* fix

* Stop hardcoding the path /usr/local/lib/afl in afl-ld-lto.c and respect the configured PREFIX.

* Add benchmark for Raspberry Pi 5

* ryzen 5950 benchmark

* add missing raspery5

* comparison -> comparison.md

* removing options "-Wl,-rpath" "LLVM_LIBDIR" when using gcc

* fixing -Wl,-rpath=<LLVM_LIBDIR>

* nits

* fix

* afl-cc fixes

* nit

* add n_fuzz to ignore_timeouts

* fix

* Fix #1927

* in-depth blog post

* add AFL_FUZZER_LOOPCOUNT

* AFL_FUZZER_LOOPCOUNT

* fix 2 mutation bugs

* v4.09c release

* v4.10a init

* switch to explore powerschedule as default

* fix MUT_INSERTASCIINUM

* fix MUT_STRATEGY_ARRAY_SIZE

* fix bad fix for MUT_STRATEGY_ARRAY_SIZE

* remove afl-network-client on uninstall

* update nyx

* Improve binary-only related docs

* llvm 18 build fixes.

* code format

* Fix custom_send link

Add a leading '/' to walk in the repo root instead of current dir.

* Use ../ instead

* initial simple injection detection support

* inject docs

* fix for issue #1916, iLLVM crash in split-floatingpoint-compares

* LLVM 17 bug workaround

* finish injection implementation

* remove tmp todo

* update changelog

* forgot to add the injection pass

* Output afl-clang-fast stuffs only if necessary (#1912)

* afl-cc header

* afl-cc common declarations

 - Add afl-cc-state.c
 - Strip includes, find_object, debug/be_quiet/have_*/callname setting from afl-cc.c
 - Use debugf_args in main
 - Modify execvp stuffs to fit new aflcc struct

* afl-cc show usage

* afl-cc mode selecting

1. compiler_mode by callname in argv[0]
2. compiler_mode by env "AFL_CC_COMPILER"
3. compiler_mode/instrument_mode by command line options "--afl-..."
4. instrument_mode/compiler_mode by various env vars including "AFL_LLVM_INSTRUMENT"
5. final checking steps
6. print "... - mode: %s-%s\n"
7. determine real argv[0] according to compiler_mode

* afl-cc macro defs

* afl-cc linking behaviors

* afl-cc fsanitize behaviors

* afl-cc misc

* afl-cc body update

* afl-cc all-in-one

formated with custom-format.py

* nits

---------

Co-authored-by: vanhauser-thc <vh@thc.org>

* changelog

* update grammar mutator

* lto llvm 12+

* docs(custom_mutators): fix missing ':' (#1953)

* Fix broken LTO mode and response file support (#1948)

* Strip `-Wl,-no-undefined` during compilation (#1952)

Make the compiler wrapper stripping `-Wl,-no-undefined` in addition to `-Wl,--no-undefined`.
Both versions of the flag are accepted by clang and, therefore, used by building systems in the wild (e.g., samba will not build without this fix).

* Remove dead code in write_to_testcase (#1955)

The custom_mutators_count check in if case is duplicate with if condition.
The else case is custom_mutators_count == 0, neither custom_mutator_list iteration nor sent check needed.

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>

* update qemuafl

* WIP: Add ability to generate drcov trace using QEMU backend (#1956)

* Document new drcov QEMU plugin

* Add link to lightkeeper for QEMU drcov file loading

---------

Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>

* code format

* changelog

* sleep on uid != 0 afl-system-config

* fix segv about skip_next, warn on unsupported cases of linking options (#1958)

* todos

* ensure afl-cc only allows available compiler modes

* update grammar mutator

* disable aslr on apple

* fix for arm64

* help selective instrumentation

* typos

* macos

* add compiler test script

* apple fixes

---------

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Chris Ball <chris@printf.net>
Co-authored-by: hexcoder <hexcoder-@users.noreply.github.com>
Co-authored-by: hexcoder- <heiko@hexco.de>
Co-authored-by: Manuel Carrasco <m.carrasco@imperial.ac.uk>
Co-authored-by: Jasper Lievisse Adriaanse <j@jasper.la>
Co-authored-by: ifyGecko <26214995+ifyGecko@users.noreply.github.com>
Co-authored-by: Dominik Maier <domenukk@gmail.com>
Co-authored-by: Christian Holler (:decoder) <choller@mozilla.com>
Co-authored-by: Carlo Maragno <ste.maragno@gmail.com>
Co-authored-by: yangzao <yangzaocn@outlook.com>
Co-authored-by: Romain Geissler <romain.geissler@amadeus.com>
Co-authored-by: Jakob Lell <jakob@jakoblell.com>
Co-authored-by: vincenzo MEZZELA <vincenzo.mezzela@amadeus.com>
Co-authored-by: Andrea Fioraldi <andreafioraldi@gmail.com>
Co-authored-by: Bet4 <0xbet4@gmail.com>
Co-authored-by: David Carlier <devnexen@gmail.com>
Co-authored-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Sonic <50692172+SonicStark@users.noreply.github.com>
Co-authored-by: Nils Bars <nils.bars@rub.de>
Co-authored-by: Jean-Romain Garnier <7504819+JRomainG@users.noreply.github.com>
Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>
vanhauser-thc added a commit that referenced this pull request Feb 1, 2024
* Output afl-clang-fast stuffs only if necessary (#1912)

* afl-cc header

* afl-cc common declarations

 - Add afl-cc-state.c
 - Strip includes, find_object, debug/be_quiet/have_*/callname setting from afl-cc.c
 - Use debugf_args in main
 - Modify execvp stuffs to fit new aflcc struct

* afl-cc show usage

* afl-cc mode selecting

1. compiler_mode by callname in argv[0]
2. compiler_mode by env "AFL_CC_COMPILER"
3. compiler_mode/instrument_mode by command line options "--afl-..."
4. instrument_mode/compiler_mode by various env vars including "AFL_LLVM_INSTRUMENT"
5. final checking steps
6. print "... - mode: %s-%s\n"
7. determine real argv[0] according to compiler_mode

* afl-cc macro defs

* afl-cc linking behaviors

* afl-cc fsanitize behaviors

* afl-cc misc

* afl-cc body update

* afl-cc all-in-one

formated with custom-format.py

* nits

---------

Co-authored-by: vanhauser-thc <vh@thc.org>

* changelog

* update grammar mutator

* lto llvm 12+

* docs(custom_mutators): fix missing ':' (#1953)

* Fix broken LTO mode and response file support (#1948)

* Strip `-Wl,-no-undefined` during compilation (#1952)

Make the compiler wrapper stripping `-Wl,-no-undefined` in addition to `-Wl,--no-undefined`.
Both versions of the flag are accepted by clang and, therefore, used by building systems in the wild (e.g., samba will not build without this fix).

* Remove dead code in write_to_testcase (#1955)

The custom_mutators_count check in if case is duplicate with if condition.
The else case is custom_mutators_count == 0, neither custom_mutator_list iteration nor sent check needed.

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>

* update qemuafl

* WIP: Add ability to generate drcov trace using QEMU backend (#1956)

* Document new drcov QEMU plugin

* Add link to lightkeeper for QEMU drcov file loading

---------

Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>

* code format

* changelog

* sleep on uid != 0 afl-system-config

* fix segv about skip_next, warn on unsupported cases of linking options (#1958)

* todos

* ensure afl-cc only allows available compiler modes

* update grammar mutator

* disable aslr on apple

* fix for arm64

* help selective instrumentation

* typos

* macos

* add compiler test script

* apple fixes

* bump nyx submodules (#1963)

* fix docs

* update changelog

* update grammar mutator

* improve compiler test script

* gcc asan workaround (#1966)

* fix github merge fuckup

* fix

* Fix afl-cc (#1968)

- Check if too many cmdline params here, each time before insert a new param.
 - Check if it is "-fsanitize=..." before we do sth.
 - Remove improper param_st transfer.

* Avoid adding llvmnative instrumentation when linking rust sanitizer runtime (#1969)

* Dynamic instrumentation filtering for LLVM native (#1971)

* Add two dynamic instrumentation filter methods to runtime

* Always use pc-table with native pcguard

* Add make_symbol_list.py and README

* changelog

* todos

* new forkserver check

* fix

* nyx test for CI

* improve nyx docs

* Fixes to afl-cc and documentation (#1974)

* Always compile with -ldl when building for CODE_COVERAGE

When building with CODE_COVERAGE, the afl runtime contains code that
calls `dladdr` which requires -ldl. Under most circumstances, clang
already adds this (e.g. when building with pc-table), but there are some
circumstances where it isn't added automatically.

* Add visibility declaration to __afl_connected

When building with hidden visibility, the use of __AFL_LOOP inside such
code can cause linker errors due to __afl_connected being declared
"hidden".

* Update docs to clarify that CODE_COVERAGE=1 is required for dynamic_covfilter

* nits

* nyx build script updates

* test error output

* debug ci

* debug ci

* Improve afl-cc (#1975)

* update response file support

 - full support of rsp file
 - fix some segv issues

* Improve afl-cc

 - remove dead code about allow/denylist options of sancov
 - missing `if (!aflcc->have_msan)`
 - add docs for each function
 - typo

* enable nyx

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* fix ci

* clean test script

* NO_NYX

* NO_NYX

* fix ci

* debug ci

* fix ci

* finalize ci fix

---------

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Sonic <50692172+SonicStark@users.noreply.github.com>
Co-authored-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Nils Bars <nils.bars@rub.de>
Co-authored-by: Jean-Romain Garnier <7504819+JRomainG@users.noreply.github.com>
Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>
Co-authored-by: Sergej Schumilo <sergej@schumilo.de>
Co-authored-by: Christian Holler (:decoder) <choller@mozilla.com>
vanhauser-thc added a commit that referenced this pull request Feb 3, 2024
* Output afl-clang-fast stuffs only if necessary (#1912)

* afl-cc header

* afl-cc common declarations

 - Add afl-cc-state.c
 - Strip includes, find_object, debug/be_quiet/have_*/callname setting from afl-cc.c
 - Use debugf_args in main
 - Modify execvp stuffs to fit new aflcc struct

* afl-cc show usage

* afl-cc mode selecting

1. compiler_mode by callname in argv[0]
2. compiler_mode by env "AFL_CC_COMPILER"
3. compiler_mode/instrument_mode by command line options "--afl-..."
4. instrument_mode/compiler_mode by various env vars including "AFL_LLVM_INSTRUMENT"
5. final checking steps
6. print "... - mode: %s-%s\n"
7. determine real argv[0] according to compiler_mode

* afl-cc macro defs

* afl-cc linking behaviors

* afl-cc fsanitize behaviors

* afl-cc misc

* afl-cc body update

* afl-cc all-in-one

formated with custom-format.py

* nits

---------

Co-authored-by: vanhauser-thc <vh@thc.org>

* changelog

* update grammar mutator

* lto llvm 12+

* docs(custom_mutators): fix missing ':' (#1953)

* Fix broken LTO mode and response file support (#1948)

* Strip `-Wl,-no-undefined` during compilation (#1952)

Make the compiler wrapper stripping `-Wl,-no-undefined` in addition to `-Wl,--no-undefined`.
Both versions of the flag are accepted by clang and, therefore, used by building systems in the wild (e.g., samba will not build without this fix).

* Remove dead code in write_to_testcase (#1955)

The custom_mutators_count check in if case is duplicate with if condition.
The else case is custom_mutators_count == 0, neither custom_mutator_list iteration nor sent check needed.

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>

* update qemuafl

* WIP: Add ability to generate drcov trace using QEMU backend (#1956)

* Document new drcov QEMU plugin

* Add link to lightkeeper for QEMU drcov file loading

---------

Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>

* code format

* changelog

* sleep on uid != 0 afl-system-config

* fix segv about skip_next, warn on unsupported cases of linking options (#1958)

* todos

* ensure afl-cc only allows available compiler modes

* update grammar mutator

* disable aslr on apple

* fix for arm64

* help selective instrumentation

* typos

* macos

* add compiler test script

* apple fixes

* bump nyx submodules (#1963)

* fix docs

* update changelog

* update grammar mutator

* improve compiler test script

* gcc asan workaround (#1966)

* fix github merge fuckup

* fix

* Fix afl-cc (#1968)

- Check if too many cmdline params here, each time before insert a new param.
 - Check if it is "-fsanitize=..." before we do sth.
 - Remove improper param_st transfer.

* Avoid adding llvmnative instrumentation when linking rust sanitizer runtime (#1969)

* Dynamic instrumentation filtering for LLVM native (#1971)

* Add two dynamic instrumentation filter methods to runtime

* Always use pc-table with native pcguard

* Add make_symbol_list.py and README

* changelog

* todos

* new forkserver check

* fix

* nyx test for CI

* improve nyx docs

* Fixes to afl-cc and documentation (#1974)

* Always compile with -ldl when building for CODE_COVERAGE

When building with CODE_COVERAGE, the afl runtime contains code that
calls `dladdr` which requires -ldl. Under most circumstances, clang
already adds this (e.g. when building with pc-table), but there are some
circumstances where it isn't added automatically.

* Add visibility declaration to __afl_connected

When building with hidden visibility, the use of __AFL_LOOP inside such
code can cause linker errors due to __afl_connected being declared
"hidden".

* Update docs to clarify that CODE_COVERAGE=1 is required for dynamic_covfilter

* nits

* nyx build script updates

* test error output

* debug ci

* debug ci

* Improve afl-cc (#1975)

* update response file support

 - full support of rsp file
 - fix some segv issues

* Improve afl-cc

 - remove dead code about allow/denylist options of sancov
 - missing `if (!aflcc->have_msan)`
 - add docs for each function
 - typo

* enable nyx

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* fix ci

* clean test script

* NO_NYX

* NO_NYX

* fix ci

* debug ci

* fix ci

* finalize ci fix

* Enhancement on Deterministic stage (#1972)

* fuzzer: init commit based on aflpp 60dc37a

* fuzzers: adding the skip variables and initialize

* log: profile the det/havoc finding

* log: add profile log output

* fuzzers: sperate log/skipdet module

* fuzzers: add quick eff_map calc

* fuzzers: add skip_eff_map in fuzz_one

* fuzzers: mark whole input space in eff_map

* fuzzers: add undet bit threshold to skip some seeds

* fuzzers: fix one byte overflow

* fuzzers: fix overflow

* fix code format

* add havoc only again

* code format

* remove log to INTROSPECTION, rename skipdet module

* rename skipdet module

* remove log to stats

* clean redundant code

* code format

* remove redundant code format check

* remove redundant doc

* remove redundant objects

* clean files

* change -d to default skipdet

* disable deterministic when using CUSTOM_MUTATOR

* revert fix

* final touches for skipdet

* remove unused var

* remove redundant eff struct (#1977)

* update QEMU-Nyx submodule (#1978)

* update QEMU-Nyx submodule (#1980)

* Fix type in AFL_NOOPT env variable in afl-cc help message (#1982)

* nits

* 2024 v4.10c release

* fixes

---------

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Sonic <50692172+SonicStark@users.noreply.github.com>
Co-authored-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Nils Bars <nils.bars@rub.de>
Co-authored-by: Jean-Romain Garnier <7504819+JRomainG@users.noreply.github.com>
Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>
Co-authored-by: Sergej Schumilo <sergej@schumilo.de>
Co-authored-by: Christian Holler (:decoder) <choller@mozilla.com>
Co-authored-by: Han Zheng <35988108+kdsjZh@users.noreply.github.com>
Co-authored-by: Khaled Yakdan <yakdan@code-intelligence.com>
@vanhauser-thc vanhauser-thc mentioned this pull request Feb 3, 2024
domenukk pushed a commit that referenced this pull request Feb 19, 2024
* push to stable (#1983)

* Output afl-clang-fast stuffs only if necessary (#1912)

* afl-cc header

* afl-cc common declarations

 - Add afl-cc-state.c
 - Strip includes, find_object, debug/be_quiet/have_*/callname setting from afl-cc.c
 - Use debugf_args in main
 - Modify execvp stuffs to fit new aflcc struct

* afl-cc show usage

* afl-cc mode selecting

1. compiler_mode by callname in argv[0]
2. compiler_mode by env "AFL_CC_COMPILER"
3. compiler_mode/instrument_mode by command line options "--afl-..."
4. instrument_mode/compiler_mode by various env vars including "AFL_LLVM_INSTRUMENT"
5. final checking steps
6. print "... - mode: %s-%s\n"
7. determine real argv[0] according to compiler_mode

* afl-cc macro defs

* afl-cc linking behaviors

* afl-cc fsanitize behaviors

* afl-cc misc

* afl-cc body update

* afl-cc all-in-one

formated with custom-format.py

* nits

---------

Co-authored-by: vanhauser-thc <vh@thc.org>

* changelog

* update grammar mutator

* lto llvm 12+

* docs(custom_mutators): fix missing ':' (#1953)

* Fix broken LTO mode and response file support (#1948)

* Strip `-Wl,-no-undefined` during compilation (#1952)

Make the compiler wrapper stripping `-Wl,-no-undefined` in addition to `-Wl,--no-undefined`.
Both versions of the flag are accepted by clang and, therefore, used by building systems in the wild (e.g., samba will not build without this fix).

* Remove dead code in write_to_testcase (#1955)

The custom_mutators_count check in if case is duplicate with if condition.
The else case is custom_mutators_count == 0, neither custom_mutator_list iteration nor sent check needed.

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>

* update qemuafl

* WIP: Add ability to generate drcov trace using QEMU backend (#1956)

* Document new drcov QEMU plugin

* Add link to lightkeeper for QEMU drcov file loading

---------

Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>

* code format

* changelog

* sleep on uid != 0 afl-system-config

* fix segv about skip_next, warn on unsupported cases of linking options (#1958)

* todos

* ensure afl-cc only allows available compiler modes

* update grammar mutator

* disable aslr on apple

* fix for arm64

* help selective instrumentation

* typos

* macos

* add compiler test script

* apple fixes

* bump nyx submodules (#1963)

* fix docs

* update changelog

* update grammar mutator

* improve compiler test script

* gcc asan workaround (#1966)

* fix github merge fuckup

* fix

* Fix afl-cc (#1968)

- Check if too many cmdline params here, each time before insert a new param.
 - Check if it is "-fsanitize=..." before we do sth.
 - Remove improper param_st transfer.

* Avoid adding llvmnative instrumentation when linking rust sanitizer runtime (#1969)

* Dynamic instrumentation filtering for LLVM native (#1971)

* Add two dynamic instrumentation filter methods to runtime

* Always use pc-table with native pcguard

* Add make_symbol_list.py and README

* changelog

* todos

* new forkserver check

* fix

* nyx test for CI

* improve nyx docs

* Fixes to afl-cc and documentation (#1974)

* Always compile with -ldl when building for CODE_COVERAGE

When building with CODE_COVERAGE, the afl runtime contains code that
calls `dladdr` which requires -ldl. Under most circumstances, clang
already adds this (e.g. when building with pc-table), but there are some
circumstances where it isn't added automatically.

* Add visibility declaration to __afl_connected

When building with hidden visibility, the use of __AFL_LOOP inside such
code can cause linker errors due to __afl_connected being declared
"hidden".

* Update docs to clarify that CODE_COVERAGE=1 is required for dynamic_covfilter

* nits

* nyx build script updates

* test error output

* debug ci

* debug ci

* Improve afl-cc (#1975)

* update response file support

 - full support of rsp file
 - fix some segv issues

* Improve afl-cc

 - remove dead code about allow/denylist options of sancov
 - missing `if (!aflcc->have_msan)`
 - add docs for each function
 - typo

* enable nyx

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* fix ci

* clean test script

* NO_NYX

* NO_NYX

* fix ci

* debug ci

* fix ci

* finalize ci fix

* Enhancement on Deterministic stage (#1972)

* fuzzer: init commit based on aflpp 60dc37a

* fuzzers: adding the skip variables and initialize

* log: profile the det/havoc finding

* log: add profile log output

* fuzzers: sperate log/skipdet module

* fuzzers: add quick eff_map calc

* fuzzers: add skip_eff_map in fuzz_one

* fuzzers: mark whole input space in eff_map

* fuzzers: add undet bit threshold to skip some seeds

* fuzzers: fix one byte overflow

* fuzzers: fix overflow

* fix code format

* add havoc only again

* code format

* remove log to INTROSPECTION, rename skipdet module

* rename skipdet module

* remove log to stats

* clean redundant code

* code format

* remove redundant code format check

* remove redundant doc

* remove redundant objects

* clean files

* change -d to default skipdet

* disable deterministic when using CUSTOM_MUTATOR

* revert fix

* final touches for skipdet

* remove unused var

* remove redundant eff struct (#1977)

* update QEMU-Nyx submodule (#1978)

* update QEMU-Nyx submodule (#1980)

* Fix type in AFL_NOOPT env variable in afl-cc help message (#1982)

* nits

* 2024 v4.10c release

* fixes

---------

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Sonic <50692172+SonicStark@users.noreply.github.com>
Co-authored-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Nils Bars <nils.bars@rub.de>
Co-authored-by: Jean-Romain Garnier <7504819+JRomainG@users.noreply.github.com>
Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>
Co-authored-by: Sergej Schumilo <sergej@schumilo.de>
Co-authored-by: Christian Holler (:decoder) <choller@mozilla.com>
Co-authored-by: Han Zheng <35988108+kdsjZh@users.noreply.github.com>
Co-authored-by: Khaled Yakdan <yakdan@code-intelligence.com>

* replaced unicornafl with unicorn

The submodule of unicorn cannot be imported through unicornafl.*_const here. If we want to use the *_const module, we should reference `from unicorn.*_const import *` like this instead of importing the entire contents of the *_const module via unicornafl。

---------

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: van Hauser <vh@thc.org>
Co-authored-by: Sonic <50692172+SonicStark@users.noreply.github.com>
Co-authored-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Nils Bars <nils.bars@rub.de>
Co-authored-by: Jean-Romain Garnier <7504819+JRomainG@users.noreply.github.com>
Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>
Co-authored-by: Sergej Schumilo <sergej@schumilo.de>
Co-authored-by: Christian Holler (:decoder) <choller@mozilla.com>
Co-authored-by: Han Zheng <35988108+kdsjZh@users.noreply.github.com>
Co-authored-by: Khaled Yakdan <yakdan@code-intelligence.com>
domenukk pushed a commit that referenced this pull request Feb 27, 2024
* push to stable (#1983)

* Output afl-clang-fast stuffs only if necessary (#1912)

* afl-cc header

* afl-cc common declarations

 - Add afl-cc-state.c
 - Strip includes, find_object, debug/be_quiet/have_*/callname setting from afl-cc.c
 - Use debugf_args in main
 - Modify execvp stuffs to fit new aflcc struct

* afl-cc show usage

* afl-cc mode selecting

1. compiler_mode by callname in argv[0]
2. compiler_mode by env "AFL_CC_COMPILER"
3. compiler_mode/instrument_mode by command line options "--afl-..."
4. instrument_mode/compiler_mode by various env vars including "AFL_LLVM_INSTRUMENT"
5. final checking steps
6. print "... - mode: %s-%s\n"
7. determine real argv[0] according to compiler_mode

* afl-cc macro defs

* afl-cc linking behaviors

* afl-cc fsanitize behaviors

* afl-cc misc

* afl-cc body update

* afl-cc all-in-one

formated with custom-format.py

* nits

---------

Co-authored-by: vanhauser-thc <vh@thc.org>

* changelog

* update grammar mutator

* lto llvm 12+

* docs(custom_mutators): fix missing ':' (#1953)

* Fix broken LTO mode and response file support (#1948)

* Strip `-Wl,-no-undefined` during compilation (#1952)

Make the compiler wrapper stripping `-Wl,-no-undefined` in addition to `-Wl,--no-undefined`.
Both versions of the flag are accepted by clang and, therefore, used by building systems in the wild (e.g., samba will not build without this fix).

* Remove dead code in write_to_testcase (#1955)

The custom_mutators_count check in if case is duplicate with if condition.
The else case is custom_mutators_count == 0, neither custom_mutator_list iteration nor sent check needed.

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>

* update qemuafl

* WIP: Add ability to generate drcov trace using QEMU backend (#1956)

* Document new drcov QEMU plugin

* Add link to lightkeeper for QEMU drcov file loading

---------

Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>

* code format

* changelog

* sleep on uid != 0 afl-system-config

* fix segv about skip_next, warn on unsupported cases of linking options (#1958)

* todos

* ensure afl-cc only allows available compiler modes

* update grammar mutator

* disable aslr on apple

* fix for arm64

* help selective instrumentation

* typos

* macos

* add compiler test script

* apple fixes

* bump nyx submodules (#1963)

* fix docs

* update changelog

* update grammar mutator

* improve compiler test script

* gcc asan workaround (#1966)

* fix github merge fuckup

* fix

* Fix afl-cc (#1968)

- Check if too many cmdline params here, each time before insert a new param.
 - Check if it is "-fsanitize=..." before we do sth.
 - Remove improper param_st transfer.

* Avoid adding llvmnative instrumentation when linking rust sanitizer runtime (#1969)

* Dynamic instrumentation filtering for LLVM native (#1971)

* Add two dynamic instrumentation filter methods to runtime

* Always use pc-table with native pcguard

* Add make_symbol_list.py and README

* changelog

* todos

* new forkserver check

* fix

* nyx test for CI

* improve nyx docs

* Fixes to afl-cc and documentation (#1974)

* Always compile with -ldl when building for CODE_COVERAGE

When building with CODE_COVERAGE, the afl runtime contains code that
calls `dladdr` which requires -ldl. Under most circumstances, clang
already adds this (e.g. when building with pc-table), but there are some
circumstances where it isn't added automatically.

* Add visibility declaration to __afl_connected

When building with hidden visibility, the use of __AFL_LOOP inside such
code can cause linker errors due to __afl_connected being declared
"hidden".

* Update docs to clarify that CODE_COVERAGE=1 is required for dynamic_covfilter

* nits

* nyx build script updates

* test error output

* debug ci

* debug ci

* Improve afl-cc (#1975)

* update response file support

 - full support of rsp file
 - fix some segv issues

* Improve afl-cc

 - remove dead code about allow/denylist options of sancov
 - missing `if (!aflcc->have_msan)`
 - add docs for each function
 - typo

* enable nyx

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* fix ci

* clean test script

* NO_NYX

* NO_NYX

* fix ci

* debug ci

* fix ci

* finalize ci fix

* Enhancement on Deterministic stage (#1972)

* fuzzer: init commit based on aflpp 60dc37a

* fuzzers: adding the skip variables and initialize

* log: profile the det/havoc finding

* log: add profile log output

* fuzzers: sperate log/skipdet module

* fuzzers: add quick eff_map calc

* fuzzers: add skip_eff_map in fuzz_one

* fuzzers: mark whole input space in eff_map

* fuzzers: add undet bit threshold to skip some seeds

* fuzzers: fix one byte overflow

* fuzzers: fix overflow

* fix code format

* add havoc only again

* code format

* remove log to INTROSPECTION, rename skipdet module

* rename skipdet module

* remove log to stats

* clean redundant code

* code format

* remove redundant code format check

* remove redundant doc

* remove redundant objects

* clean files

* change -d to default skipdet

* disable deterministic when using CUSTOM_MUTATOR

* revert fix

* final touches for skipdet

* remove unused var

* remove redundant eff struct (#1977)

* update QEMU-Nyx submodule (#1978)

* update QEMU-Nyx submodule (#1980)

* Fix type in AFL_NOOPT env variable in afl-cc help message (#1982)

* nits

* 2024 v4.10c release

* fixes

---------

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Sonic <50692172+SonicStark@users.noreply.github.com>
Co-authored-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Nils Bars <nils.bars@rub.de>
Co-authored-by: Jean-Romain Garnier <7504819+JRomainG@users.noreply.github.com>
Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>
Co-authored-by: Sergej Schumilo <sergej@schumilo.de>
Co-authored-by: Christian Holler (:decoder) <choller@mozilla.com>
Co-authored-by: Han Zheng <35988108+kdsjZh@users.noreply.github.com>
Co-authored-by: Khaled Yakdan <yakdan@code-intelligence.com>

* Fix dump_regs() type errors in Python

TypeError will occur as gdb api return some strange type of values that json can't serialize, this would fix this issue

* Update reg_val is None condition

---------

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: van Hauser <vh@thc.org>
Co-authored-by: Sonic <50692172+SonicStark@users.noreply.github.com>
Co-authored-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Nils Bars <nils.bars@rub.de>
Co-authored-by: Jean-Romain Garnier <7504819+JRomainG@users.noreply.github.com>
Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>
Co-authored-by: Sergej Schumilo <sergej@schumilo.de>
Co-authored-by: Christian Holler (:decoder) <choller@mozilla.com>
Co-authored-by: Han Zheng <35988108+kdsjZh@users.noreply.github.com>
Co-authored-by: Khaled Yakdan <yakdan@code-intelligence.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants