Assignment: FUZZ

Changelog:

In this assignment, you will use a “whitebox fuzzing” tool to find memory errors in software in combination with a generic memory error detector.

Assignment resources

  1. Documentation for the two tools we will be using:

    • AddressSanitizer: a memory error detector integrated with recent versions of the C compilers GCC and Clang.
    • american fuzzy lop: a whitebox fuzzing tool written by Michał Zalewski.

  2. Neystadt, “Automated Penetration Testing with White-Box Fuzzing”.

Assignment details

  1. We will be testing a slighly modified version of FreeBSD’s indent program. Download the source code for it here.

    This is this code from FreeBSD’s git repository with these changes applied to it. The changes let the program compile on Linux and makes it ignore non-ASCII input.

  2. Download american fuzzy lop from here or a local copy.

Build american fuzzy lop

  1. Extract the tar file you downloaded with a command like:

    tar -zxvf afl-latest.tgz

    (where afl-latest.tgz is the file you downloaded). This will create a directory named something like afl-2.39b. In this directory run:

    make

Build indent with AFL and AddressSanitizer support

  1. Unpack fbsd-indent.tar.gz using a command like:

    tar -zxvf fbsd-indent.tar.gz

    Then in the fbsd-indent directory run:

    AFL_USE_ASAN=1 /path/to/afl-2.39b/afl-gcc -m32 -fno-omit-frame-pointer -fsanitize=address -g -O *.c -o indent

    where /path/to/afl-2.39b/afl-gcc is the afl-gcc program in the directory you ran make in when building american fuzzy lop. You can find the path to the directory using the pwd command while inside it.

    • The option -m32 builds a 32-bit binary instead of a 64-bit one. This is necessary because AFL has a hard time limiting the memory usage of a 64-bit program that uses AddressSanitizer. See also the documentation in docs/notes_for_asan.txt in the AFL directory.

    • The options AFL_USE_ASAN=1 and -fsanitize=address make this program use AddressSanitizer, which is a memory error detector. -fsanitize=address is the option that GCC uses to enable sanitization. We also tell AFL we are building with ASAN support so it can adjust how it instruments the application.

    • The -fno-omit-frame-pointer option makes GCC use a frame pointer in the generated code. This will allow AddressSanitizer to generate good stack traces.

    • Building with afl-gcc instead of gcc adds tracking of what path is taken through the program that AFL can use later on to generate test cases.

    Alternately, you may also build indent without AddressSanitizer support to perform the fuzzing, then rebuild it with AddressSanitizer to diagnose the memory errors found by the fuzzing. See the instructions under the Hints below. This will allow you to build without -m32, if that is a problem on your system. This option makes fuzzing faster, but identifies memory errors less consistently.

  2. Verify that indent works by running it on a simple C file. For example, you can try running it on the io.c that comes with indent using:

    indent <io.c >io.c.indented

    then compare the two files (io.c and io.c.indented) in a text editor.

    Note that if indent is given a filename as an argument it modifies that file.

Setup to run AFL

  1. By default Ubuntu runs a special program to handle applications that crash. We want to disable this feature to run american fuzzy lop, because it wants to monitor how the program it tests crashes accurately. To do this run

    sudo bash -c 'echo core >/proc/sys/kernel/core_pattern'

    Note that you will need to rerun this command if you reboot your VM.

Run the AFL fuzzer on indent

  1. american fuzzy lop will take an initial test case and make random changes to it. It will combine different random changes based on which changes seem to cause the program to execute more different paths.

  2. You need to first supply one or more initial test cases. Create a directory called testcases. Inside it create one or more small C files to use to test indent. american fuzzy lop will be able to find more useful test cases if your C files include things that a C indenting program will need to handle. Otherwise, the fuzzer will need to “discover” what indent will recognize by testing randomly.

  3. Now that you have the initial input. Run afl-fuzz on indent with a command like

    /path/to/afl-2.39b/afl-fuzz -m 700 -i testcases -o findings ./indent

    The options here:

    • The -m 700 says to make up to 700 MB of memory available to indent. It won’t actually use that much memory, but AddressSanitizer will reserve a lot of memory.
    • The -o findings option specifies the directory to write output to and to use to store temporary files.
    • The -i testcases option specifies where to find the initial test cases.

    This memory AddressSanitizer reserves is use store a table with one entry for every 8 bytes of memory indicating whether the memory is valid to access. AddressSanitizer reserves space for all possible addresses in this table, but leaves parts of the table which correspond to invalid addresses as invalid rather actually requiring that space to be allocated by the operating system.

  4. As the fuzzer runs, you will see a status screen which is described here.

  5. Wait until the fuzzer reports finding at least two unique crashes (and preferably at least three or four). In my testing, this took substantially less than 15 minutes. If it does not, then you should consider whether your initial cases are too simple (don’t include enough C syntax that indent will need to process) or too long (take up many kilobytes, making each test run slower and giving the tool more “boring” variations to try). If you still have difficulty after adjusting your test cases, please contact the instructor.

  6. After the fuzzer generates some crashing test cases, stop it with control-C. If we were doing this for real, we’d keep running this until AFL completed a complete “cycle”, but we will not make you wait that long.

  7. Make a copy of the status screen (screenshot or text copy) and save it to a file for later.

Examine a Crashing Test Case

  1. Look in the findings directory that was generated by running AFL. Among this folder there are directories called crashes and hangs. These contain the test cases that match the unique crashes and/or hangs reported in the status screen.

  2. View one of the crashing test cases. Note that the test case may contain some non-ASCII characters, so you may wish to check using a hex editor or od -c file rather than just viewing the test case in a text editor.

  3. Make a copy of the crashing test case and trying running the indent program on it was ./indent <testcase.dat. You should notice a crash. Most likely this crash will be from a memory error. Since we built our program with AddressSanitizer, rather than being a normal segfault, the memory error will have been caught by extra code added by AddressSanitizer. See below under “Hints” for how to interpret this output.

  4. Rather than working with this complex test case, we will run an utility afl-tmin that comes with american fuzzy lop to minimize it. It will try to “fuzz” the given test case slightly without changing what path it takes through the program in order to make it shorter. Run this utility with

    /path/to/afl-2.39b/afl-tmin -m 700 -i input-file -o output-file ./indent

    to produce a minimized version of the testcase in input-file in output-file.

  5. Verify that the minimized test case produces a similar crash.

  6. Examine the program around where AddressSanitizer places the crash.

Questions

Answer the following questions in file called answers.txt:

  1. Where did the crash occur in our first test case? If the cause was accessing an array out of bounds, what variable did the program neglect to keep in bounds? If the cause was something else, explain briefly.

  2. Examine some other crashing test cases and try running them with the program. Do they represent different bugs? Explain briefly how you determined this.

Submission

  1. Submit the following on Collab:

    • The file answers.txt.

    • A screenshot or text copy of the AFL fuzzer status message in a file called status.txt or status.png

    • An unminimized crashing test case in a file called testcase.dat.

    • A minimized version of that test case in a file called testcase.dat.min.

    • One of the other crashing test cases you used to answer question 2 in a file called testcase2.dat.

    • Your initial test cases that you used to generate testcase.dat.

Hints and Debugging and Alternatives

Possible error messages

  1. If you get an error like

     args.c:42:23: fatal error: sys/cdefs.h: No such file or directory

    then run

     sudo apt install libc6-dev-i386

Understanding AddressSanitizer output

  1. AddressSanitizer output includes the following information:

    • A stack trace indicating where the memory error occured. This is the location of the bad read or write, which may not be where the bug needs to be fixed in the program.

    • If the memory error was because of trying to access the heap, the location of the code that most recently freed or malloc’d something near that the location that was accessed.

    • If the memory error was because of trying to access the stack or global data, the variables that are closest in memory to the location that was accessed.

    • “Shadow bytes”, which show AddressSanitizer’s internal data structure for keeping track of the state of memory around the accessed address. This has one byte of data for every eight bytes of program memory, indicating whether that part of program memory is valid or invalid and, if it is invalid, why it is invalid. In this data structure “red zones” are memory regions allocated around objects to catch accesses just outside of an object.

Fuzzing Without AddressSanitizer

  1. It’s also possible to disable AddressSanitizer when fuzzing, but use it only to diagnose the crashes the fuzzer finds. This will make fuzzing faster at the cost finding a few less crashes.

    In this case, compile without -m32 or -fsanitize=address or AFL_USE_ASAN=1. Then, after getting fuzzing results, build indent again with AddressSanitizer by compiling with gcc instead of afl-gcc and adding back the -fsanitize=address option. Then, take the crashing test cases the fuzzer finds --- or other test cases in the queue` subdirectory, which might also crash but only under AddressSanitizer.