FuzzBench: 2020-09-07 report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplus_optimal 98.61
honggfuzz 95.37
entropic 93.75
lafintel 91.53
libfuzzer 91.47
aflsmart 90.35
afl 89.89
mopt 89.60
aflfast 87.67
fairfuzz 84.74
eclipser 76.68
By avg. rank
average rank
fuzzer
aflplusplus_optimal 2.48
afl 4.72
honggfuzz 4.85
aflsmart 4.95
entropic 5.10
mopt 5.40
lafintel 6.32
libfuzzer 6.70
fairfuzz 7.48
aflfast 7.55
eclipser 10.45
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median coverages on each benchmark
    fuzzer afl aflfast aflplusplus_optimal aflsmart eclipser entropic fairfuzz honggfuzz lafintel libfuzzer mopt
    benchmark
    bloaty_fuzz_target 8476 8469 8911 8664 6723 7097 9015 8230 7781 6875 9206
    curl_curl_fuzzer_http 17659 17482 18117 17659 15699 16686 16824 17987 17497 16218 17630
    freetype2-2017 20394 20142 26053 20375 19020 21206 20348 21638 19569 19528 20416
    harfbuzz-1.3.2 8363 8317 8875 8377 7545 8404 8008 8595 8331 8341 8348
    jsoncpp_jsoncpp_fuzzer 639 639 639 639 621 641 639 641 639 641 639
    lcms-2017-03-21 3119 2553 3604 3133 2412 3368 3156 2749 2865 3434 2923
    libjpeg-turbo-07-2017 3778 3769 3780 3752 2857 3754 3035 3757 3473 3751 3776
    libpcap_fuzz_both 101 101 3912 83 2929 3721 101 3828 3796 3592 101
    libpng-1.2.56 1944 1943 2004 2128 1530 2000 1945 2153 2001 1999 1944
    libxml2-v2.9.2 12768 12045 13026 12833 5455 12453 10792 13096 12145 11985 12195
    mbedtls_fuzz_dtlsclient 8244 8122 8280 8278 7656 8182 8259 8032 8149 8039 8238
    openssl_x509 13784 13781 13784 13784 13713 13759 13708 13734 13783 13758 13784
    openthread-2019-12-23 5849 5248 5865 5826 5714 5902 4143 5769 5268 5899 5827
    proj4-2017-08-14 5340 5150 6864 5271 429 7866 5202 7360 5336 7726 5206
    re2-2014-12-09 3505 3504 3516 3504 3115 3518 3504 3545 3496 3554 3502
    sqlite3_ossfuzz 35615 34785 36671 35709 16206 23867 26835 30271 27808 22491 35230
    systemd_fuzz-link-parser 640 640 640 640 637 639 639 639 640 619 640
    vorbis-2017-12-11 2166 2164 2172 2166 2057 2157 2165 2155 2167 1830 2177
    woff2-2016-05-06 1864 1830 1871 1859 1376 1760 1654 1839 1863 1653 1853
    zlib_zlib_uncompress_fuzzer 963 942 963 961 938 966 964 964 964 985 960

bloaty_fuzz_target summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    mopt 82800 15.0 9080.800000 356.353997 8176.0 9146.5 9206.0 9256.0 9303.0
    fairfuzz 82800 15.0 8765.866667 530.225810 7597.0 8672.0 9015.0 9116.5 9210.0
    aflplusplus_optimal 82800 15.0 8859.666667 190.021678 8271.0 8846.5 8911.0 8955.5 9051.0
    aflsmart 82800 15.0 8570.200000 316.670446 7788.0 8542.0 8664.0 8733.0 8963.0
    afl 82800 15.0 8425.866667 405.172250 7552.0 8125.0 8476.0 8782.5 8904.0
    aflfast 82800 15.0 8248.666667 370.935625 7692.0 7980.5 8469.0 8502.0 8841.0
    honggfuzz 82800 15.0 8270.866667 292.969737 7741.0 8156.0 8230.0 8482.0 8709.0
    lafintel 82800 15.0 7768.066667 338.518509 7339.0 7507.5 7781.0 7835.0 8512.0
    entropic 82800 15.0 7075.133333 96.223599 6810.0 7069.5 7097.0 7124.5 7212.0
    libfuzzer 82800 15.0 6826.666667 300.778672 6260.0 6665.5 6875.0 7035.5 7281.0
    eclipser 82800 15.0 6781.533333 315.770004 6397.0 6534.0 6723.0 6972.5 7414.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

curl_curl_fuzzer_http summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 15.0 18106.866667 112.170066 17868.0 18082.5 18117.0 18166.5 18271.0
    honggfuzz 82800 15.0 17977.666667 83.986960 17809.0 17929.0 17987.0 18036.0 18104.0
    afl 82800 15.0 17663.200000 89.892952 17533.0 17614.5 17659.0 17695.5 17857.0
    aflsmart 82800 15.0 17671.333333 66.908964 17529.0 17645.5 17659.0 17719.5 17771.0
    mopt 82800 15.0 17610.466667 73.523433 17373.0 17588.5 17630.0 17650.5 17672.0
    lafintel 82800 15.0 17502.800000 88.094268 17395.0 17424.0 17497.0 17546.5 17711.0
    aflfast 82800 15.0 17486.666667 82.772206 17343.0 17430.5 17482.0 17537.0 17654.0
    fairfuzz 82800 15.0 16745.000000 401.353781 15851.0 16595.5 16824.0 16983.5 17393.0
    entropic 82800 15.0 16572.266667 437.361809 15775.0 16438.5 16686.0 16782.0 17286.0
    libfuzzer 82800 15.0 16010.533333 881.148347 14328.0 15288.5 16218.0 16490.0 17253.0
    eclipser 82800 15.0 15695.600000 104.549920 15485.0 15674.5 15699.0 15757.5 15877.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

freetype2-2017 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 15.0 25902.933333 694.827776 24311.0 25490.5 26053.0 26274.5 26899.0
    honggfuzz 82800 15.0 21573.533333 972.553184 20000.0 20668.5 21638.0 22013.0 23443.0
    entropic 82800 15.0 20917.400000 1064.393577 18379.0 20311.5 21206.0 21433.5 22337.0
    mopt 82800 15.0 20416.000000 148.270024 20062.0 20339.0 20416.0 20512.5 20651.0
    afl 82800 15.0 20465.066667 181.433761 20215.0 20331.0 20394.0 20556.5 20850.0
    aflsmart 82800 15.0 20336.733333 187.658301 20034.0 20252.0 20375.0 20435.5 20723.0
    fairfuzz 82800 15.0 20293.333333 353.150042 19104.0 20279.5 20348.0 20438.5 20672.0
    aflfast 82800 15.0 20155.266667 152.163665 19908.0 20090.5 20142.0 20219.5 20520.0
    lafintel 82800 15.0 19682.933333 539.058898 18840.0 19230.5 19569.0 20146.0 20515.0
    libfuzzer 82800 15.0 19563.333333 867.757180 18425.0 18728.0 19528.0 20253.0 20777.0
    eclipser 82800 15.0 18842.733333 639.960319 17404.0 18737.5 19020.0 19296.5 19538.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

harfbuzz-1.3.2 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 15.0 8815.666667 173.619315 8409.0 8803.5 8875.0 8896.5 9022.0
    honggfuzz 82800 15.0 8597.133333 58.980465 8472.0 8570.0 8595.0 8624.0 8715.0
    entropic 82800 15.0 8451.266667 112.748435 8306.0 8371.0 8404.0 8547.5 8625.0
    aflsmart 82800 15.0 8407.933333 78.578865 8338.0 8357.5 8377.0 8427.0 8557.0
    afl 82800 15.0 8388.733333 74.380745 8315.0 8353.5 8363.0 8388.0 8596.0
    mopt 82800 15.0 8366.666667 64.996337 8316.0 8331.5 8348.0 8360.5 8547.0
    libfuzzer 82800 15.0 8358.800000 93.456789 8212.0 8324.0 8341.0 8387.0 8555.0
    lafintel 82800 15.0 8349.533333 73.158211 8254.0 8293.0 8331.0 8403.0 8492.0
    aflfast 82800 15.0 8356.533333 83.153788 8274.0 8293.0 8317.0 8424.5 8494.0
    fairfuzz 82800 15.0 7971.466667 245.100815 7458.0 7829.5 8008.0 8124.0 8316.0
    eclipser 82800 15.0 7556.000000 94.376904 7463.0 7482.0 7545.0 7590.5 7827.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 15.0 641.000000 0.000000 641.0 641.0 641.0 641.0 641.0
    honggfuzz 82800 15.0 641.000000 0.000000 641.0 641.0 641.0 641.0 641.0
    libfuzzer 82800 15.0 641.000000 0.000000 641.0 641.0 641.0 641.0 641.0
    afl 82800 15.0 639.000000 0.000000 639.0 639.0 639.0 639.0 639.0
    aflfast 82800 15.0 638.866667 0.351866 638.0 639.0 639.0 639.0 639.0
    aflplusplus_optimal 82800 15.0 639.000000 0.000000 639.0 639.0 639.0 639.0 639.0
    aflsmart 82800 15.0 639.000000 0.000000 639.0 639.0 639.0 639.0 639.0
    fairfuzz 82800 15.0 639.000000 0.000000 639.0 639.0 639.0 639.0 639.0
    lafintel 82800 15.0 639.000000 0.000000 639.0 639.0 639.0 639.0 639.0
    mopt 82800 15.0 639.000000 0.000000 639.0 639.0 639.0 639.0 639.0
    eclipser 82800 15.0 618.866667 6.010309 608.0 615.0 621.0 623.0 627.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

lcms-2017-03-21 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 15.0 3638.800000 123.929819 3454.0 3549.5 3604.0 3746.5 3872.0
    libfuzzer 82800 15.0 3409.733333 184.749818 3114.0 3282.5 3434.0 3540.5 3755.0
    entropic 82800 15.0 3378.066667 130.459883 3152.0 3307.5 3368.0 3454.5 3647.0
    fairfuzz 82800 15.0 3110.400000 164.377962 2817.0 3010.0 3156.0 3218.0 3393.0
    aflsmart 82800 15.0 3090.800000 244.239695 2735.0 2881.5 3133.0 3224.0 3494.0
    afl 82800 15.0 3058.333333 168.875130 2740.0 2939.0 3119.0 3189.0 3241.0
    mopt 82800 15.0 2963.866667 238.113678 2594.0 2792.0 2923.0 3085.5 3439.0
    lafintel 82800 15.0 2922.400000 337.748596 2568.0 2592.5 2865.0 3059.0 3580.0
    honggfuzz 82800 15.0 2714.200000 78.986617 2551.0 2739.0 2749.0 2754.0 2774.0
    aflfast 82800 15.0 2359.200000 531.380171 1231.0 2433.0 2553.0 2632.0 2904.0
    eclipser 82800 15.0 1989.266667 644.781188 1176.0 1370.5 2412.0 2487.0 2844.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

libjpeg-turbo-07-2017 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 15.0 3765.933333 82.582830 3472.0 3778.5 3780.0 3793.0 3820.0
    afl 82800 15.0 3757.466667 97.626743 3411.0 3762.0 3778.0 3798.0 3815.0
    mopt 82800 15.0 3759.400000 97.593764 3411.0 3773.0 3776.0 3789.0 3818.0
    aflfast 82800 15.0 3767.200000 15.288651 3747.0 3752.5 3769.0 3776.0 3798.0
    honggfuzz 82800 15.0 3749.266667 90.717353 3431.0 3755.0 3757.0 3794.0 3817.0
    entropic 82800 15.0 3678.933333 201.566532 3180.0 3753.0 3754.0 3756.0 3760.0
    aflsmart 82800 15.0 3639.066667 167.234423 3406.0 3457.0 3752.0 3770.0 3803.0
    libfuzzer 82800 15.0 3562.200000 252.473252 3171.0 3315.5 3751.0 3754.5 3760.0
    lafintel 82800 15.0 3592.333333 168.010912 3408.0 3435.0 3473.0 3758.5 3794.0
    fairfuzz 82800 15.0 3022.400000 227.443494 2477.0 2925.5 3035.0 3124.0 3347.0
    eclipser 82800 15.0 2889.333333 131.483658 2567.0 2840.5 2857.0 2953.0 3152.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

libpcap_fuzz_both summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 15.0 2978.533333 1815.790928 83.0 1878.0 3912.0 4125.0 4346.0
    honggfuzz 82800 15.0 3810.000000 364.704381 3029.0 3733.5 3828.0 4000.0 4551.0
    lafintel 82800 15.0 3684.400000 348.778849 3005.0 3411.5 3796.0 3872.5 4255.0
    entropic 82800 15.0 3783.200000 179.156436 3537.0 3652.5 3721.0 3932.0 4069.0
    libfuzzer 82800 15.0 3572.600000 105.968189 3403.0 3489.5 3592.0 3663.0 3708.0
    eclipser 82800 15.0 2770.066667 271.280421 2332.0 2521.0 2929.0 2980.5 3112.0
    afl 82800 15.0 99.933333 4.712698 83.0 101.0 101.0 101.0 103.0
    aflfast 82800 15.0 95.000000 8.783101 83.0 83.0 101.0 101.0 101.0
    fairfuzz 82800 15.0 199.400000 177.413883 83.0 101.0 101.0 282.0 508.0
    mopt 82800 15.0 92.600000 9.295160 83.0 83.0 101.0 101.0 101.0
    aflsmart 82800 15.0 85.400000 6.333584 83.0 83.0 83.0 83.0 101.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

libpng-1.2.56 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 15.0 2142.866667 30.319411 2085.0 2137.5 2153.0 2160.5 2189.0
    aflsmart 82800 15.0 2128.333333 22.021634 2087.0 2116.5 2128.0 2134.5 2177.0
    aflplusplus_optimal 82800 15.0 2002.933333 3.326660 1997.0 2000.5 2004.0 2005.5 2008.0
    lafintel 82800 15.0 2002.333333 3.518658 1999.0 2000.0 2001.0 2003.0 2010.0
    entropic 82800 15.0 2000.733333 3.750556 1994.0 1998.5 2000.0 2003.0 2008.0
    libfuzzer 82800 15.0 1985.466667 24.354133 1945.0 1972.5 1999.0 2000.0 2002.0
    fairfuzz 82800 15.0 1944.666667 2.023669 1941.0 1943.5 1945.0 1946.0 1948.0
    afl 82800 15.0 1943.866667 1.846490 1941.0 1942.0 1944.0 1945.0 1947.0
    mopt 82800 15.0 1943.800000 2.305273 1940.0 1942.5 1944.0 1945.0 1949.0
    aflfast 82800 15.0 1939.400000 12.385475 1896.0 1940.5 1943.0 1945.0 1946.0
    eclipser 82800 15.0 1516.600000 53.917662 1408.0 1487.5 1530.0 1554.0 1592.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

libxml2-v2.9.2 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 15.0 13011.200000 222.029985 12492.0 12958.0 13096.0 13144.0 13240.0
    aflplusplus_optimal 82800 15.0 13033.466667 230.908104 12594.0 12955.5 13026.0 13123.5 13587.0
    aflsmart 82800 15.0 12741.333333 330.732232 12148.0 12465.0 12833.0 12973.5 13181.0
    afl 82800 15.0 12690.000000 331.888536 12044.0 12430.5 12768.0 12994.5 13051.0
    entropic 82800 15.0 12389.333333 377.688962 11933.0 12067.5 12453.0 12598.0 13044.0
    mopt 82800 15.0 12240.066667 380.990526 11707.0 11889.0 12195.0 12461.0 12843.0
    lafintel 82800 15.0 12172.200000 372.165708 11608.0 11843.5 12145.0 12512.5 12666.0
    aflfast 82800 15.0 12027.466667 201.546828 11685.0 11902.5 12045.0 12173.5 12385.0
    libfuzzer 82800 15.0 12110.533333 270.261214 11760.0 11940.5 11985.0 12173.5 12759.0
    fairfuzz 82800 15.0 10542.800000 957.878623 8380.0 10106.5 10792.0 11004.5 12488.0
    eclipser 82800 15.0 5417.066667 113.901252 5092.0 5415.5 5455.0 5486.0 5525.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

mbedtls_fuzz_dtlsclient summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 15.0 8494.600000 776.461829 8243.0 8266.0 8280.0 8322.0 11297.0
    aflsmart 82800 15.0 8478.400000 819.326710 8200.0 8229.0 8278.0 8303.0 11436.0
    fairfuzz 82800 15.0 8249.733333 53.524983 8172.0 8202.0 8259.0 8289.0 8358.0
    afl 82800 15.0 8445.866667 763.926031 8193.0 8214.0 8244.0 8279.0 11203.0
    mopt 82800 15.0 8440.400000 751.318346 8198.0 8219.0 8238.0 8261.0 11152.0
    entropic 82800 15.0 8190.266667 56.049043 8113.0 8155.5 8182.0 8209.5 8310.0
    lafintel 82800 15.0 8148.933333 24.998476 8113.0 8132.0 8149.0 8161.5 8197.0
    aflfast 82800 15.0 8105.533333 50.006952 8017.0 8075.0 8122.0 8146.0 8174.0
    libfuzzer 82800 15.0 7928.733333 243.913763 7567.0 7674.5 8039.0 8141.0 8208.0
    honggfuzz 82800 15.0 8051.266667 35.473263 8014.0 8026.0 8032.0 8075.5 8138.0
    eclipser 82800 15.0 7653.266667 57.196986 7562.0 7612.0 7656.0 7704.0 7736.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

openssl_x509 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 13784.400000 1.549193 13784.0 13784.0 13784.0 13784.0 13790.0
    aflplusplus_optimal 82800 15.0 13784.533333 0.915475 13784.0 13784.0 13784.0 13785.0 13786.0
    aflsmart 82800 15.0 13784.000000 0.000000 13784.0 13784.0 13784.0 13784.0 13784.0
    mopt 82800 15.0 13780.800000 8.444779 13760.0 13784.0 13784.0 13784.0 13784.0
    lafintel 82800 15.0 13780.466667 8.365462 13759.0 13783.0 13783.0 13784.0 13785.0
    aflfast 82800 15.0 13775.533333 12.223319 13748.0 13770.5 13781.0 13783.0 13784.0
    entropic 82800 15.0 13762.000000 9.433981 13749.0 13758.0 13759.0 13761.0 13785.0
    libfuzzer 82800 15.0 13762.133333 11.933546 13749.0 13757.0 13758.0 13766.5 13783.0
    honggfuzz 82800 15.0 13736.133333 8.822590 13716.0 13734.0 13734.0 13737.0 13760.0
    eclipser 82800 15.0 13711.466667 16.374487 13692.0 13697.0 13713.0 13728.5 13735.0
    fairfuzz 82800 15.0 13710.066667 25.350308 13686.0 13689.5 13708.0 13714.0 13768.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

openthread-2019-12-23 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 15.0 5901.000000 12.845233 5869.0 5894.5 5902.0 5910.0 5918.0
    libfuzzer 82800 15.0 5897.933333 13.285151 5867.0 5891.0 5899.0 5906.0 5920.0
    aflplusplus_optimal 82800 15.0 5875.000000 24.068355 5841.0 5858.0 5865.0 5891.0 5912.0
    afl 82800 15.0 5767.666667 210.995825 5256.0 5819.5 5849.0 5859.0 5903.0
    mopt 82800 15.0 5754.000000 211.689463 5227.0 5775.0 5827.0 5867.0 5889.0
    aflsmart 82800 15.0 5742.066667 209.235160 5208.0 5801.5 5826.0 5838.5 5849.0
    honggfuzz 82800 15.0 5766.733333 157.388721 5243.0 5745.5 5769.0 5860.0 5887.0
    eclipser 82800 15.0 5616.266667 194.494754 5237.0 5661.5 5714.0 5724.5 5779.0
    lafintel 82800 15.0 5356.133333 199.196194 5234.0 5261.5 5268.0 5270.0 5817.0
    aflfast 82800 15.0 5325.200000 213.487437 5203.0 5243.5 5248.0 5251.5 5864.0
    fairfuzz 82800 15.0 4668.333333 703.738452 4143.0 4143.0 4143.0 5157.0 5876.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

proj4-2017-08-14 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 15.0 7864.133333 52.546397 7752.0 7828.5 7866.0 7896.0 7975.0
    libfuzzer 82800 15.0 7735.800000 54.786860 7663.0 7697.0 7726.0 7785.0 7827.0
    honggfuzz 82800 15.0 7345.733333 176.881101 7032.0 7265.0 7360.0 7469.0 7582.0
    aflplusplus_optimal 82800 15.0 6797.000000 351.867061 5591.0 6800.5 6864.0 6889.0 7147.0
    afl 82800 15.0 5342.600000 85.340327 5183.0 5296.0 5340.0 5381.5 5517.0
    lafintel 82800 15.0 5321.866667 69.141539 5191.0 5271.0 5336.0 5374.0 5431.0
    aflsmart 82800 15.0 5356.200000 171.155401 5230.0 5247.5 5271.0 5391.5 5846.0
    mopt 82800 15.0 5211.133333 73.925509 5066.0 5169.0 5206.0 5252.5 5350.0
    fairfuzz 82800 15.0 5099.933333 166.501165 4799.0 4947.5 5202.0 5225.5 5274.0
    aflfast 82800 15.0 5191.866667 113.676777 5058.0 5135.5 5150.0 5227.5 5460.0
    eclipser 82800 15.0 439.066667 248.735507 160.0 190.0 429.0 538.5 1117.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

re2-2014-12-09 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 15.0 3551.666667 9.278752 3519.0 3552.0 3554.0 3555.0 3559.0
    honggfuzz 82800 15.0 3532.800000 17.251087 3511.0 3516.0 3545.0 3548.5 3551.0
    entropic 82800 15.0 3527.733333 24.075674 3476.0 3515.0 3518.0 3553.5 3559.0
    aflplusplus_optimal 82800 15.0 3515.133333 3.979710 3506.0 3514.0 3516.0 3518.0 3520.0
    afl 82800 15.0 3505.600000 6.905484 3498.0 3503.5 3505.0 3506.0 3529.0
    aflfast 82800 15.0 3503.600000 4.420730 3497.0 3500.5 3504.0 3506.0 3511.0
    aflsmart 82800 15.0 3505.600000 7.632637 3495.0 3502.0 3504.0 3507.0 3529.0
    fairfuzz 82800 15.0 3499.266667 28.431538 3413.0 3502.5 3504.0 3508.0 3530.0
    mopt 82800 15.0 3502.533333 5.755329 3493.0 3497.5 3502.0 3506.0 3511.0
    lafintel 82800 15.0 3497.933333 7.878240 3486.0 3493.0 3496.0 3505.0 3514.0
    eclipser 82800 15.0 3109.666667 22.771118 3061.0 3104.0 3115.0 3122.0 3137.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

sqlite3_ossfuzz summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 15.0 36630.733333 271.948384 36149.0 36423.0 36671.0 36811.0 37116.0
    aflsmart 82800 15.0 35697.733333 254.316863 35357.0 35499.0 35709.0 35940.0 36098.0
    afl 82800 15.0 35515.600000 305.387529 34863.0 35342.5 35615.0 35693.5 36047.0
    mopt 82800 15.0 35203.200000 337.791609 34501.0 35088.0 35230.0 35429.0 35684.0
    aflfast 82800 15.0 34810.533333 276.699906 34341.0 34625.0 34785.0 34906.0 35342.0
    honggfuzz 82800 15.0 30209.133333 437.798856 29423.0 29920.0 30271.0 30387.0 31140.0
    lafintel 82800 15.0 27782.266667 1111.512963 25021.0 27312.5 27808.0 28594.5 29188.0
    fairfuzz 82800 15.0 26268.333333 3341.803441 21554.0 22788.0 26835.0 28699.5 31180.0
    entropic 82800 15.0 23734.533333 893.845454 21344.0 23342.5 23867.0 24108.0 25055.0
    libfuzzer 82800 15.0 22539.600000 863.774508 20732.0 22186.5 22491.0 22871.0 24508.0
    eclipser 82800 15.0 16416.400000 812.950078 16200.0 16203.0 16206.0 16210.5 19355.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

vorbis-2017-12-11 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    mopt 82800 15.0 2174.666667 5.037951 2166.0 2170.0 2177.0 2178.5 2183.0
    aflplusplus_optimal 82800 15.0 2172.333333 4.011887 2167.0 2169.0 2172.0 2174.0 2181.0
    lafintel 82800 15.0 2166.666667 6.113996 2150.0 2165.0 2167.0 2168.5 2179.0
    afl 82800 15.0 2168.866667 5.591660 2163.0 2165.0 2166.0 2174.5 2179.0
    aflsmart 82800 15.0 2168.533333 5.396648 2164.0 2165.0 2166.0 2169.5 2183.0
    fairfuzz 82800 15.0 2160.133333 15.999405 2110.0 2162.5 2165.0 2166.5 2175.0
    aflfast 82800 15.0 2166.133333 5.235411 2161.0 2163.0 2164.0 2167.0 2179.0
    entropic 82800 15.0 2155.600000 4.032015 2143.0 2155.0 2157.0 2157.5 2160.0
    honggfuzz 82800 15.0 2155.800000 5.239956 2146.0 2153.0 2155.0 2157.5 2167.0
    eclipser 82800 15.0 2055.800000 29.301877 2011.0 2033.5 2057.0 2072.0 2113.0
    libfuzzer 82800 15.0 1937.466667 157.549478 1829.0 1830.0 1830.0 2144.0 2164.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

woff2-2016-05-06 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: eclipser.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 15.0 1872.400000 18.538762 1844.0 1856.0 1871.0 1890.5 1895.0
    afl 82800 15.0 1864.133333 18.070760 1838.0 1846.0 1864.0 1881.0 1888.0
    lafintel 82800 15.0 1863.933333 14.032954 1836.0 1855.0 1863.0 1877.5 1882.0
    aflsmart 82800 15.0 1862.400000 15.117634 1844.0 1850.0 1859.0 1873.0 1889.0
    mopt 82800 15.0 1854.800000 9.420343 1841.0 1849.0 1853.0 1857.0 1882.0
    honggfuzz 82800 15.0 1847.666667 25.855827 1818.0 1827.5 1839.0 1876.5 1886.0
    aflfast 82800 15.0 1834.866667 16.308922 1811.0 1827.0 1830.0 1840.5 1869.0
    entropic 82800 15.0 1743.266667 53.408756 1641.0 1699.5 1760.0 1776.0 1816.0
    fairfuzz 82800 15.0 1643.400000 31.470621 1600.0 1613.0 1654.0 1667.0 1693.0
    libfuzzer 82800 15.0 1667.933333 39.966891 1603.0 1639.0 1653.0 1703.5 1732.0
    eclipser 82800 9.0 1389.222222 24.076845 1376.0 1376.0 1376.0 1391.0 1445.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 15.0 976.400000 10.356502 965.0 966.0 985.0 986.0 986.0
    entropic 82800 15.0 974.733333 10.102805 965.0 966.0 966.0 985.0 986.0
    fairfuzz 82800 15.0 967.933333 9.794070 956.0 963.0 964.0 973.5 984.0
    honggfuzz 82800 15.0 967.266667 8.819351 956.0 963.5 964.0 965.0 985.0
    lafintel 82800 15.0 966.333333 10.932695 943.0 963.0 964.0 965.0 985.0
    afl 82800 15.0 965.666667 10.438026 945.0 962.0 963.0 964.0 984.0
    aflplusplus_optimal 82800 15.0 964.666667 7.843711 960.0 960.0 963.0 964.0 984.0
    aflsmart 82800 15.0 962.000000 2.138090 960.0 960.0 961.0 964.0 965.0
    mopt 82800 15.0 962.400000 6.310988 956.0 960.0 960.0 963.0 984.0
    aflfast 82800 15.0 945.933333 11.877149 925.0 942.0 942.0 960.0 960.0
    eclipser 82800 15.0 941.533333 10.259955 920.0 937.0 938.0 942.0 963.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique regions covered
    Each bar shows the total number of regions found by a given fuzzer. The colored area shows the number of unique regions (i.e., regions that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of regions covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 6dff14dd2a3efe05c5689e21784bb002b743a923