Fuzzing Guided by Bayesian Program Analysis (Paper Artifact)

Fuzzing Guided by Bayesian Program Analysis (Paper Artifact) All scripts and source code of this artifact are released under the MIT License. List of Claims We provide complete reproduction steps for the following illustrations in the paper: Figure 11 Figure 12 Figure 13 Figure 14 The complete table, that contains Table 3 and Table 4, presents the median TTE (time-toexposure) of each fuzzer over 10 runs on 24 real-world programs We also provide detailed information about each newly discovered vulnerability that has not yet been assigned a CVE, which has been submitted to the developers and is presented in Table 5 and Table 6. Download, Installation, and Sanity-testing Due to the efficiency requirements of fuzzing, our artifact currently only supports running on Linux x86-64/AMD. We provide a Docker image that you can pull and rename using the following command: docker pull jjleo/bayzzer-artifact:latest docker tag jjleo/bayzzer-artifact:latest bayzzer-artifact:latest Since fuzzing generates a large amount of log information, we recommend setting a limit for the log size. Please add the following content to the Docker configuration file  /etc/docker/daemon.json: { "log-driver": "json-file", "log-opts": {"max-size": "10m", "max-file": "3"}, } Please make sure to restart Docker for this configuration to take effect: sudo systemctl restart docker You also need to install the following Python packages: pip3 install numpy matplotlib scipy Please download our scripts archive  bayzzer-artifact-scripts.tar.gz from Zenodo and extract it: tar -xzvf bayzzer-artifact-scripts.tar.gz Please note that all scripts must be executed from within the  bayzzer-artifact-scripts directory. You need to execute the following commands to ensure that the fuzzer runs properly: echo core | sudo tee /proc/sys/kernel/core_pattern echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor Next, enter the script folder and use the following command to test whether fuzzing can run normally in the container (this process will use CPU cores 0-3 and the entire runtime is expected to take no more than 1 hour): cd bayzzer-artifact-scripts python3 run.py all all 10 1 4 sudo chown -R $USER:$USER output If the program runs without errors and non-empty  fuzzer_stats files exist in all  output/runtime_1/out/<benchmark>/<fuzzer>/default/ folders, then the test is running normally. Next, run the following command to ensure that the vulnerability reproduction scripts can run normally: python3 copy_results.py output results python3 copy_overhead.py output results/overhead python3 reproduce.py results 1 The program will enter an interactive Docker container. Enter the following command: /reproduce/start.sh exit This will exit the interactive Docker container. Enter the following command: sudo chown -R $USER:$USER results If the above programs run without errors and the content of  results/log/0/gif2tga.san is similar to the following (it won't be exactly the same, but there should not be a large number of  <null>): { "bayzzer": { "1488": ["heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif"], ... }, "prospector": { "1474": ["heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif"], ... }, "ffapp": { "1386": ["heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif"], ... }, "funfuzz": { "1686": ["heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif"], ... }, "aflpp": { "1663": ["heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif"], ... }, "ablation": { "1468": ["heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif"], ... } } Then it indicates that the vulnerability reproduction scripts can run normally. Finally, run the following command to ensure that the illustrations generation scripts can run normally: python3 RQ1.py results-origin/log python3 RQ2.py results-origin/overhead python3 RQ3.py results-origin/log If the program runs without errors (there may be font missing issues, which is normal) and successfully generates  figure_11.pdf,  figure_12.pdf,  figure_13.pdf,  figure_14.pdf, and  complete_table_3_4.csv in the  bayzzer-artifact-scripts directory with the same data as in the paper, then it indicates that the illustrations generation scripts can run normally. Evaluation Instructions Please note that all scripts must be executed from within the  bayzzer-artifact-scripts directory. Our experiments set a time limit of 60 hours for each fuzzing run and repeated it 10 times. If you need to reproduce all experimental results, you can run: python3 run.py all all 216000 10 <CPU> sudo chown -R $USER:$USER output Here,  <CPU> indicates the number of CPU cores to be used for parallel experiments (from core 0 to core  <CPU>-1). Reproducing all experiments as reported in the paper will require approximately 9.86 CPU-years of computation. Next, run the following command: python3 copy_results.py output results python3 copy_overhead.py output results/overhead python3 reproduce.py results 10 The program will enter an interactive Docker container. Enter the following command: /reproduce/start.sh exit This will exit the interactive Docker container. Enter the following command: sudo chown -R $USER:$USER results Finally, use the following command to generate illustrations: python3 RQ1.py results/log python3 RQ2.py results/overhead python3 RQ3.py results/log This script will generate  figure_11.pdf,  figure_12.pdf,  figure_13.pdf,  figure_14.pdf, and  complete_table_3_4.csv in the  bayzzer-artifact-scripts directory. We provide the original experimental data in  results-origin. You can use the following command to generate illustrations based on the original data: python3 RQ1.py results-origin/log python3 RQ2.py results-origin/overhead python3 RQ3.py results-origin/log Finally, we provide specific information on each newly discovered vulnerability that has not yet been assigned a CVE in Table 5 and Table 6. Vulnerabilities that have already been assigned CVEs can be searched directly online. Here, we only present the Bug Only vulnerabilities. Program Vulnerabilities nasm https://bugzilla.nasm.us/show_bug.cgi?id=3392931 draco https://github.com/google/draco/issues/1100 https://github.com/google/draco/issues/1102 https://github.com/google/draco/issues/1103 https://github.com/google/draco/issues/1105 bloaty https://github.com/google/bloaty/issues/398 libyaml https://github.com/yaml/libyaml/issues/312 librdkafka https://github.com/confluentinc/librdkafka/issues/5091 https://github.com/confluentinc/librdkafka/issues/5092 Additional Artifact Description We will first provide a detailed introduction to the architectural organization and usage documentation of the scripts in the artifact, then introduce the architectural organization of the Docker image and how to modify it to test new programs. Scripts We first introduce the usage of each script in  bayzzer-artifact-scripts: ├── analysis.py # used for reproducing crashes ├── asan.crash.json # used for reproducing crashes ├── bug_id.csv # used for classifying crashes ├── copy_overhead.py # used for copying overhead logs ├── copy_results.py # used for copying fuzzing results ├── reproduce.py # used for reproducing crashes ├── results-origin # original data from the paper ├── RQ1.py # used for generating illustrations in RQ1 ├── RQ2.py # used for generating illustrations in RQ2 ├── RQ3.py # used for generating illustrations in RQ3 ├── run.py # used for running fuzzing ├── runtime # corpus and scripts for running fuzzing └── venn.py # used for generating Venn graphs Next, we will explain the core scripts used in the reproduction process. Please note that all scripts must be executed from within the  bayzzer-artifact-scripts directory. Reproducing Fuzzing Process Use the following command to reproduce the fuzzing process in the experiment: python3 run.py <fuzzer> <benchmark> <time> <repeat> <CPU> where  <fuzzer> can be one of:  Bayzzer,  Prospector,  FishFuzz,  FunFuzz,  AFL++,  Ablation, or  all to run all fuzzers sequentially.  <benchmark> can be one of  exiv2,  tiffsplit,  mp3gain,  pdftotext,  flvmeta,  tcpdump,  cflow,  lame,  jhead,  mujs,  wav2swf,  objdump,  sqlite3,  gdk-pixbuf-pixdata,  MP4Box,  lou_checktable,  nasm,  gif2tga,  tcpprep,  tic,  mp42aac,  nm-new,  imginfo,  jq or  all to run all benchmarks sequentially.  <time> indicates the time limit (in seconds),  <repeat> indicates the number of repetitions, and  <CPU> indicates the number of CPU cores to be used for parallel experiments (from core 0 to core  <CPU>-1). Each run is assigned a dedicated Docker container bound to a specific CPU core. For the  i-th run of  <fuzzer> on  <program>, the corresponding container is named  <fuzzer>_<program>_<i>. Suppose the minimum value of  <i> among all current container names is , then this means that rounds of repeated experiments have been completed so far. After the run completes, the results will be saved in the  output folder under the  bayzzer-artifact-scripts directory. The results from the  x-th run (with  x starting from 1) of x x − 1  <fuzzer> on  <benchmark> will be stored in  output/runtime_<x>/out/<benchmark>/<fuzzer>/default. Since the Docker container uses the root user, you need to use the following command to restore permissions: sudo chown -R $USER:$USER output If you used Ctrl+C to interrupt  run.py, please use the following command to remove all containers that are still running: docker rm -f $(docker ps -aq --filter ancestor=bayzzer-artifact:latest) If you want to restart the experiment, the results will overwrite the previous results in the  output folder. Please make sure to use the following command to restore the permissions of the  output folder: sudo chown -R $USER:$USER output Copy Crashing Inputs Use the following command to copy crashing inputs during fuzzing process: python3 copy_results.py <output> <results> Here,  <output> and  <results> denote the fuzzing output directory and the directory to which the target is copied, respectively. The results from the  i-th run (with  i starting from 0) of  <fuzzer> on  <benchmark> will be copied to  <results>/<fuzzer>/<benchmark>/<i>/crashes. Copy Overhead Logs Use the following command to copy overhead logs during fuzzing process of Bayzzer: python3 copy_overhead.py <output> <results> Here,  <output> and  <results> denote the fuzzing output directory and the directory to which the target is copied, respectively. The results from the  i-th run (with  i starting from 0) of Bayzzer on  <benchmark> will be copied to  <results>/<benchmark>/<i>.log. Reproducing Crashing Inputs Use the following command to reproduce crashing inputs during fuzzing process: python3 reproduce.py <results> <repeat> Here,  <results> and  <repeat> denote the copied crashing inputs and the number of repetitions, respectively. Please note that  <results> must be located in the  bayzzer-artifact-scripts directory. This script will create a Docker container for reproducing the crashing inputs. The program will enter this interactive container, where you can input the following command to start the reproduction process and exit the container once reproduction is complete: /reproduce/start.sh exit The reproduction logs will be stored in  <results>/log. The results from the  x-th run (with x starting from 0) of each fuzzer on  <benchmark> are stored in  <results>/log/<x>/<benchmark>.san. Since fuzzing generates a large number of crashing inputs, this process may last for a very long time. Generating Illustrations in RQ1 Use the following command to generate Figure 11, Figure 12, and the complete table (including Table 3 and Table 4) presents the median TTE (time-to-exposure) of each fuzzer over 10 runs on 24 real-world programs: python3 RQ1.py <log> Here,  <log> denote the reproduction logs. This script will generate  figure_11.pdf,  figure_12.pdf, and  complete_table_3_4.csv in the  bayzzer-artifact-scripts directory. This script uses fuzzing result information from  Bayzzer,  Prospector,  FishFuzz,  FunFuzz, and  AFL++. Generating Illustrations in RQ2 Use the following command to generate Figure 13: python3 RQ2.py <log> Here,  <log> denote the overhead logs. This script will generate  figure_13.pdf in the  bayzzer-artifact-scripts directory. This script uses fuzzing result information from  Bayzzer. Generating Illustrations in RQ3 Use the following command to generate Figure 14: python3 RQ3.py <log> Here,  <log> denote the reproduction logs. This script will generate  figure_14.pdf in the  bayzzer-artifact-scripts directory. This script uses fuzzing result information from  Bayzzer and  Ablation. Original Data We stored the original data used in the paper in the  results-origin directory, including the crashing inputs of each fuzzing process ( results-origin/<fuzzer>), Bayzzer's overhead logs ( results-origin/overhead), the reproduction logs of each fuzzing process ( results-origin/log), as well as all illustrations ( results-origin/*.pdf and  results-origin/*.csv). All of these data were generated by the aforementioned scripts and can serve as valuable references. Docker Image First, we show the key files and directory structure inside the  bayzzer-artifact Docker image: |-- AFL++ # source code of AFL++ |-- Bayzzer # source code of Bayzzer |-- Bayzzer-ablation # source code of the ablation experiment |-- Fish++ # source code of FishFuzz |-- Prospector # source code of Prospector |-- SVF-taint # source code of SVF for taint analysis |-- benchmark # benchmark binaries built by each fuzzer | |-- build_bench_bayzzer.sh # build scripts for Bayzzer | |-- project # build scripts for each benchmark | | |-- cflow-1.6 # an exmaple benchmark | | | |-- build.sh # an exmaple bulid script |-- funfuzz-artifacts # source code of FunFuzz |-- libdai # source code of libDAI for Bayeisan inference Next, we explain how to use Bayzzer to test new programs. Compiling New Programs We take the existing benchmark  cflow as an example. Its build script is located at  /benchmark/project/cflow-1.6/build.sh: