fuzz: execute each file in dir without fuzz engine #24472

pull ajtowns wants to merge 1 commits into bitcoin:master from ajtowns:202203-phuzztesting changing 1 files +67 −5

ajtowns commented at 4:29 AM on March 4, 2022: contributor

Phony fuzzing (phuzzing)! Run the fuzz testing code against known inputs to detect errors. Advantage is you can easily test using the existing qa-assets datasets without having to compile with fuzzing enabled; disadvantage is that it doesn't do any actual fuzzing.

Example usage:

$ for a in ${QA_ASSETS}/fuzz_seed_corpus/*; do echo ${a##*/}; done | xargs -P8 -I {} /bin/sh -c "FUZZ={} test/fuzz/fuzz ${QA_ASSETS}/fuzz_seed_corpus/{}"
No fuzzer for address_deserialize.
No fuzzer for addrdb.
No fuzzer for banentry_deserialize.
addition_overflow: succeeded against 848 files in 0s.
asmap: succeeded against 981 files in 0s.
checkqueue: succeeded against 211 files in 0s.
...

(-P8 says run 8 of the tasks in parallel)

If there are failures, the first one will be reported and the program will abort with output like:

fuzz: test/fuzz/versionbits.cpp:336: void (anonymous namespace)::versionbits_fuzz_target(FuzzBufferType): Assertion `exp_state != ThresholdState::FAILED' failed.
Error processing seed "corpus/versionbits/35345ae8e722234095810b1117a29b63af7621af"

Rebase of #22763, which was a rebase of #21496, but also reports the name of the fuzzer and the time taken.

Fixes #21461

ajtowns commented at 4:33 AM on March 4, 2022: contributor

cc @AnthonyRonning @prakash1512 @sipa @practicalswift
DrahtBot added the label Tests on Mar 4, 2022
MarcoFalke cross-referenced this on Mar 4, 2022 from issue fuzz: execute each file in dir without fuzz engine by prakash1512
laanwj commented at 10:48 AM on March 7, 2022: member

Concept ACK.

ghost commented at 5:06 AM on March 9, 2022: none

ACK.

Looks good to me, sorry for dropping that originally. Ran through some of the same tests I was doing before, all working as expected.

Example of that:

FUZZ=process_messages src/test/fuzz/fuzz ../qa-assets/fuzz_seed_corpus/process_messages/
process_messages: succeeded against 14024 files in 153s.


FUZZ=process_messages src/test/fuzz/fuzz ../qa-assets/fuzz_seed_corpus/process_messages/11265df2b1a0c93629515a880a2a851d6ceb133c
process_messages: succeeded against 1 files in 0s.

for a in ../qa-assets/fuzz_seed_corpus/*; do echo ${a##*/}; done | xargs -P8 -I {} /bin/sh -c "FUZZ={} src/test/fuzz/fuzz ../qa-assets/fuzz_seed_corpus/{}"
addition_overflow: succeeded against 1047 files in 0s.
address_deserialize_v1_withtime: succeeded against 420 files in 0s.
address_deserialize_v1_notime: succeeded against 400 files in 0s.
addr_info_deserialize: succeeded against 1232 files in 1s.
address_deserialize_v2: succeeded against 949 files in 1s.
...

in src/test/fuzz/fuzz.cpp:64 in 80dd3013e0 outdated

  60 | @@ -59,6 +61,7 @@ void FuzzFrameworkRegisterTarget(std::string_view name, TypeTestOneInput target,
  61 |      Assert(it_ins.second);
  62 |  }
  63 |  
  64 | +std::string_view g_fuzz_target;

MarcoFalke commented at 5:32 PM on March 16, 2022:

nit:

static std::string_view g_fuzz_target;

in src/test/fuzz/fuzz.cpp:102 in 80dd3013e0 outdated

 100 | -    Assert(it != FuzzTargets().end());
 101 | +    g_fuzz_target = Assert(std::getenv("FUZZ"));
 102 | +    const auto it = FuzzTargets().find(g_fuzz_target);
 103 | +    if (it == FuzzTargets().end()) {
 104 | +        std::cerr << "No fuzzer for " << g_fuzz_target << "." << std::endl;
 105 | +        exit(1);

MarcoFalke commented at 5:35 PM on March 16, 2022:

nit:

        std::exit(EXIT_FAILURE);

https://en.cppreference.com/w/cpp/utility/program/EXIT_status https://en.cppreference.com/w/cpp/utility/program/exit

in src/test/fuzz/fuzz.cpp:196 in 80dd3013e0 outdated

 193 | +        }
 194 | +        test_one_input(buffer);
 195 |          return 0;
 196 |      }
 197 | -    test_one_input(buffer);
 198 | +    signal(SIGABRT, signal_handler);

MarcoFalke commented at 5:48 PM on March 16, 2022:

nit: Will have to call std::signal, if the signal calls std::_Exit, otherwise it might be UB?

in src/test/fuzz/fuzz.cpp:144 in 80dd3013e0 outdated
```
 139 | +void signal_handler(int signal)
 140 | +{
 141 | +    if (signal == SIGABRT) {
 142 | +        std::cerr << "Error processing seed " << g_seed_path << std::endl;
 143 | +    } else {
 144 | +        std::cerr << "Unexpected signal " << signal << " received\n";
```
MarcoFalke commented at 5:48 PM on March 16, 2022:

Unclear if this stl call is allowed? Might be UB, but I guess it doesn't matter either way?

ajtowns commented at 9:12 PM on March 16, 2022:

It matches the code in https://en.cppreference.com/w/cpp/utility/program/abort so should be mostly okay hopefully?

I suppose in theory you could setup worker threads, and have a monitor thread watch for them aborting -- then you could do multiple inputs in parallel, and report on multiple failures rather than exiting after the first one.

in src/test/fuzz/fuzz.cpp:142 in 80dd3013e0 outdated

 137 | +#if defined(PROVIDE_FUZZ_MAIN_FUNCTION) && !defined(__AFL_LOOP)
 138 | +fs::path g_seed_path;
 139 | +void signal_handler(int signal)
 140 | +{
 141 | +    if (signal == SIGABRT) {
 142 | +        std::cerr << "Error processing seed " << g_seed_path << std::endl;

MarcoFalke commented at 5:49 PM on March 16, 2022:

nit: I think when simply iterating over inputs, they are not called seeds?

        std::cerr << "Error processing input " << g_input_path << std::endl;

MarcoFalke commented at 5:56 PM on March 16, 2022:

( https://github.com/google/fuzzing/blob/master/docs/glossary.md#test-input )

MarcoFalke approved
MarcoFalke commented at 5:51 PM on March 16, 2022: member

review ACK

Left some nits
MarcoFalke cross-referenced this on Mar 16, 2022 from issue fuzz: Make it possible to execute each file in a directory without fuzz engine by MarcoFalke
fuzz: execute each file in dir without fuzz engine
Co-Authored-By: Anthony Ronning <anthonyronning@gmail.com>
f59bee3fb2
ajtowns force-pushed on Mar 16, 2022
ajtowns commented at 9:30 PM on March 16, 2022: contributor

Renamed seed to input, added static/std, switched from signal.h to csignal header.
MarcoFalke approved
MarcoFalke merged this on Mar 17, 2022
MarcoFalke closed this on Mar 17, 2022
sidhujag referenced this in commit 16b7f5f572 on Mar 18, 2022
bitcoin locked this on Mar 17, 2023

Contributors

Labels

Linked (view graph)

#21461 fuzz: Make it possible to execute each file in a directory without fuzz engine #21496 fuzz: execute each file in dir without fuzz engine #22763 fuzz: execute each file in dir without fuzz engine