Fuzzing Guided by Bayesian Program Analysis (Paper Artifact)

Fuzzing Guided by Bayesian Program Analysis (Paper Artifact) List of Claims We provide complete reproduction steps for the following illustrations in the paper: Figure 9 Figure 10 Figure 11 Figure 12 The complete table including Table 3 and Table 4 We also provide specific information for each new vulnerability submitted to developers in Table 5 and Table 6. Download, Installation, and Sanity-testing Due to the efficiency requirements of fuzzing, our artifact currently only supports running on Linux x86-64/AMD. We provide a Docker image that you can pull and rename using the following command: docker pull jjleo/bayzzer-artifact:latest docker tag jjleo/bayzzer-artifact:latest bayzzer-artifact:latest You also need to install the following Python packages: pip3 install numpy matplotlib scipy Please download our scripts archive  bayzzer-artifact-scripts.tar.gz from Zenodo and extract it: tar -xzvf bayzzer-artifact-scripts.tar.gz Next, enter the script folder and use the following command to test whether fuzzing can run normally in the container (this process will use CPU cores 0-3 and the entire runtime is expected to take no more than 1 hour): cd bayzzer-artifact-scripts python3 run.py all all 10 1 4 sudo chown -R $USER:$USER output If the program runs without errors and non-empty  fuzzer_stats files exist in all output/runtime_1/out/<benchmark>/<fuzzer>/default/ folders, then the test is running normally. Next, run the following command to ensure that the vulnerability reproduction scripts can run normally: python3 copy_results.py output results python3 copy_overhead.py output results/overhead python3 reproduce.py results 1 The program will enter an interactive Docker container. Enter the following command: /reproduce/start.sh exit This will exit the interactive Docker container. Enter the following command: sudo chown -R $USER:$USER results If the above programs run without errors and the content of results/log/0/gif2tga.san is similar to the following (it won't be exactly the same, but there should not be a large number of ): { "bayzzer": { "1488": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "12567": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "12888": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "12977": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] }, "prospector": { "1474": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "12514": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "12834": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "12921": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] }, "ffapp": { "1386": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "12278": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "12599": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "12686": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] }, "funfuzz": { "1686": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "13000": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "13446": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "13553": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] }, "aflpp": { "1663": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "12989": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "13457": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "13588": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] }, "ablation": { "1468": [ "heap-buffer-overflow", "WritePixel->DecodeGifImg->LoadGif" ], "12522": [ "heap-buffer-overflow", "WritePixels->DecodeGifImg->LoadGif" ], "12853": [ "heap-buffer-overflow", "GifIndexToTrueColor->WritePixels->DecodeGifImg" ], "12945": [ "stack-overflow", "DecodeGifImg->LoadGif->main" ] } } Then it indicates that the vulnerability reproduction scripts can run normally. Finally, run the following command to ensure that the illustrations generation scripts can run normally: python3 RQ1.py results-origin/log python3 RQ2.py results-origin/overhead python3 RQ3.py results-origin/log If the program runs without errors (there may be font missing issues, which is normal) and successfully generates  figure_9.pdf,  figure_10.pdf, figure_11.pdf,  figure_12.pdf, and  complete_table_3_4.csv with the same data as in the paper, then it indicates that the illustrations generation scripts can run normally. Evaluation Instructions Our experiments set a time limit of 60 hours for each fuzzing run and repeated it 10 times. If you need to reproduce all experimental results, you can run: python3 run.py all all 216000 10 <CPU> sudo chown -R $USER:$USER output This will take approximately 9.86 CPU-years to complete. If you do not have sufficient CPU cores to reproduce all experiments within the artifact evaluation timeframe, please let us know, and we can provide a server with 256 cores for reproduction. Since our method performs better over longer periods (after 40 hours), reducing the time limit may impact the experimental results. Next, run the following command: python3 copy_results.py output results python3 copy_overhead.py output results/overhead python3 reproduce.py results 10 The program will enter an interactive Docker container. Enter the following command: /reproduce/start.sh exit This will exit the interactive Docker container. Enter the following command: sudo chown -R $USER:$USER results Finally, use the following command to generate illustrations: python3 RQ1.py results/log python3 RQ2.py results/overhead python3 RQ3.py results/log This script will generate  figure_9.pdf,  figure_10.pdf,  figure_11.pdf, figure_12.pdf, and  complete_table_3_4.csv. We provide the original experimental data in  results-origin. You can use the following command to generate illustrations based on the original data: python3 RQ1.py results-origin/log python3 RQ2.py results-origin/overhead python3 RQ3.py results-origin/log Finally, we provide specific information for each vulnerability submitted to developers in Table 5 and Table 6. For vulnerabilities that have already been assigned CVEs, they can be searched directly online. Here we only show the Bug Only** vulnerabilities: Program Vulnerabilities draco https://github.com/google/draco/issues/1100 https://github.com/google/draco/issues/1102 https://github.com/google/draco/issues/1103 Program Vulnerabilities https://github.com/google/draco/issues/1105 bloaty https://github.com/google/bloaty/issues/398 libyaml https://github.com/yaml/libyaml/issues/312 librdkafka https://github.com/confluentinc/librdkafka/issues/5091 https://github.com/confluentinc/librdkafka/issues/5092 Additional Artifact Description We first introduce the contents of each script in  bayzzer-artifact-scripts: ├── analysis.py # used for reproducing crashes ├── asan.crash.json # used for reproducing crashes ├── bug_id.csv # used for classifying crashes ├── copy_overhead.py # used for copying overhead logs ├── copy_results.py # used for copying fuzzing results ├── reproduce.py # used for reproducing crashes ├── results-origin # original data from the paper ├── RQ1.py # used for generating illustrations in RQ1 ├── RQ2.py # used for generating illustrations in RQ2 ├── RQ3.py # used for generating illustrations in RQ3 ├── run.py # used for running fuzzing ├── runtime # corpus and scripts for running fuzzing └── venn.py # used for generating Venn graphs Use the following command to reproduce the fuzzing process in the experiment: python3 run.py <fuzzer> <benchmark> <time> <repeat> <CPU> where  <fuzzer> can be one of:  Bayzzer,  Prospector,  FishFuzz,  FunFuzz, AFL++,  Ablation, or  all to run all fuzzers sequentially.  <benchmark> can be