fuzz: Corpus sharding in parallel runs #36121

pull maflcko wants to merge 2 commits into bitcoin:master from maflcko:2608-fuzz-sharding changing 2 files +70 −29
  1. maflcko commented at 2:07 PM on August 29, 2026: member

    Currently, the fuzz runner accepts a --par option to schedule fuzz runs in parallel. This is fine. However, when only a single target is selected, --par will not speed up the run. Moreover, when multiple targets are selected, the run-time of the longest target dominates.

    Fix both issues by splitting the corpus to into --par equal-sized shards by default. The sharding can be disabled, if needed.

    Can be tested e.g. by running a single target:

    time ./bld-cmake/test/fuzz/test_runner.py --par 1  -l DEBUG ./qa-assets/fuzz_corpora/ utxo_snapshot  # slow
time ./bld-cmake/test/fuzz/test_runner.py --par 99 -l DEBUG ./qa-assets/fuzz_corpora/ utxo_snapshot  # fast
  2. DrahtBot added the label Fuzzing on Aug 29, 2026
  3. DrahtBot commented at 2:07 PM on August 29, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/36121.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    Concept ACK jeanpablojp

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #35417 (ci: isolate container test networks by willcl-ark)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  5. jeanpablojp commented at 1:47 PM on August 31, 2026: contributor

    Concept ACK

    I had a look at the logs. Looks like on macOS the pool was already saturated. Summing the process times and dividing by --par gives 613s there, against 200s for the slowest single target, so there was no tail to cut. On Windows and MSan it's the other way round, and that is where the win comes from.

    Almost all of the macOS side comes from one target. utxo_total_supply builds a ChainTestingSetup inside the target body, so a datadir per input, and three shards of 680 take around 240s each against 200s for all 2040 in one process. Same on Windows.

    Is it worth gating this on something, or does the win on the other jobs already outweigh it?

  6. fuzz: Corpus sharding in parallel runs
    To better utilize the given --par option, split each corpus into shards by default.

The diff can be reviewed via --ignore-all-space
    fa2d43f7ea
  7. in test/fuzz/test_runner.py:69 in d0000027db outdated 
      64 | @@ -64,6 +65,11 @@ def main():
  65 |          default=4,
  66 |          help='How many targets to merge or execute in parallel.',
  67 |      )
  68 | +    parser.add_argument(
  69 | +        '--corpus-shards',
    jeanpablojp commented at 1:47 PM on August 31, 2026:

    nit: --help ends up with two defaults, the one in the help string and a (default: None) that ArgumentDefaultsHelpFormatter appends.

    maflcko commented at 9:04 AM on September 1, 2026:

    Yeah, saw that too. Not sure how to fix. I guess it is fine to leave as-is.

  8. in test/fuzz/test_runner.py:341 in d0000027db 
     347 | -        ]
 348 | -        empty_dir = not any(corpus_path.iterdir())
 349 | -        if using_libfuzzer:
 350 | -            if empty_min_time and empty_dir:
 351 | -                args += [f"-max_total_time={empty_min_time}"]
 352 | +    with tempfile.TemporaryDirectory(dir=corpus.parent, prefix='.fuzz-shards-') as shard_root:
    jeanpablojp commented at 1:47 PM on August 31, 2026:

    This gets created before we know whether there will be any sharding, so with a read-only parent the run dies in mkdtemp, and --corpus-shards 1 doesn't avoid it. It also survives a SIGTERM, left next to the corpus.

    And since the with wraps the collection loop too, the removal only happens after the last job, with everything else idle. 31s on macOS.

    maflcko commented at 9:04 AM on September 1, 2026:

    read-only parent should be rare, but I think I fixed this in the latest push by having a temp dir for each job.

    SIGTERM will probably still leave them around.

    The creation/removal is now interleaved, but I don't expect this to improve the runtime, unless there is over-subscription, but then there could be even more storage thrashing...

  9. in test/fuzz/test_runner.py:348 in d0000027db 
     354 | +            corpus_path = corpus / t
 355 | +            os.makedirs(corpus_path, exist_ok=True)
 356 | +            corpus_files = sorted(path for path in corpus_path.iterdir() if path.is_file())
 357 | +            empty_dir = not corpus_files
 358 | +
 359 | +            if corpus_shards > 1 and len(corpus_files) > 1:
    jeanpablojp commented at 1:47 PM on August 31, 2026:

    This hits every target that has a corpus, none has fewer than three inputs. cmpctblock sometimes does the same kind of per-input work through ResetChainmanAndMempool.

    The hard links are one per corpus input, so that part doesn't shrink with more workers. On macOS that takes 155s before the first job gets collected.

    maflcko commented at 9:04 AM on September 1, 2026:

    Hmm, I guess this is a combination of some OSs not having a tmpfs at all and the GHA VMs having a slow storage. :(

  10. in test/fuzz/test_runner.py:394 in d0000027db 
     439 | +            except subprocess.CalledProcessError as e:
 440 | +                if e.stdout:
 441 | +                    logging.info(e.stdout)
 442 | +                if e.stderr:
 443 | +                    logging.info(e.stderr)
 444 | +                logging.info(f"⚠️ Failure generated from target with exit code {e.returncode}: {result.args}")
    jeanpablojp commented at 1:47 PM on August 31, 2026:

    nit: this path is already gone by the time anyone reads the log, it's inside the shard dir. The file name survives, the path doesn't.

    And with the sys.exit(1) in here, the dir disappears under the shards that are still running, so on the standalone builds they die on the read_file Assert instead of on the input that actually broke.

    maflcko commented at 9:04 AM on September 1, 2026:

    Hmm, the file disappearing should be fine, because the basename of the file (sha1 hash) should be important/sufficient.

    The read_file assertions aren't great. I guess this is fixed by having per-job tmp dirs for the shards.

  11. in test/fuzz/test_runner.py:399 in d0000027db 
     444 | +                logging.info(f"⚠️ Failure generated from target with exit code {e.returncode}: {result.args}")
 445 | +                sys.exit(1)
 446 | +            if using_libfuzzer:
 447 | +                done_stat = [l for l in output.splitlines() if "DONE" in l]
 448 | +                assert len(done_stat) == 1
 449 | +                stats.append((target, done_stat[0]))
    jeanpablojp commented at 1:47 PM on August 31, 2026:

    Summary goes from 237 lines to 3754 on your asan run, and cov: becomes per process. coins_view_db reports sixteen values between 12033 and 13709 where master reports 14509, and since it's a union there's no way to put it back together. So you can't compare coverage across a change any more. Could the shards be folded back into one line per target?

    maflcko commented at 9:04 AM on September 1, 2026:

    I don't think it is possible to recover the coverage from the shards. I can only see compiling with coverage instrumentation and then using that. This could possibly also help with https://github.com/bitcoin-core/qa-assets/issues/166

  12. maflcko force-pushed on Sep 1, 2026
  13. maflcko commented at 9:04 AM on September 1, 2026: member

    I had a look at the logs. Looks like on macOS the pool was already saturated. Summing the process times and dividing by --par gives 613s there, against 200s for the slowest single target, so there was no tail to cut.

    Hmm, if the slowest single target happened to be scheduled toward the end, there would be a tail to cut, no?

    Almost all of the macOS side comes from one target. utxo_total_supply builds a ChainTestingSetup inside the target body, so a datadir per input, and three shards of 680 take around 240s each against 200s for all 2040 in one process. Same on Windows.

    Hmm, I guess this means the bottleneck is storage IO and not CPU, so this pull request won't help and may even make it worse due to storage thrashing.

    In theory, all of the fuzzing should be in a tmpfs (default on modern Linux), but neither macOS nor Windows have that option :(

    Is it worth gating this on something, or does the win on the other jobs already outweigh it?

    in theory, the macOS config (and possibly Windows) could disable it, but if this gets too complicated, I may close this pull.

  14. maflcko marked this as a draft on Sep 1, 2026
  15. ci: tmpfs 633456aa8b
  16. maflcko commented at 12:55 PM on September 1, 2026: member

    Ok, so tmpfs helped a bit on macos, but still not enough. On Linux, it didn't really make a difference, it seems.

  17. jeanpablojp commented at 3:28 PM on September 1, 2026: contributor

    Hmm, if the slowest single target happened to be scheduled toward the end, there would be a tail to cut, no?

    You're right. utxo_total_supply is the last to finish in all 20 master runs I checked, and it runs for the last 86 to 214s of them, so my sentence was wrong.

    Ok, so tmpfs helped a bit on macos, but still not enough.

    macOS on master runs anywhere from 632 to 976s and this head is 767s, so it is inside what master does on its own. I guess two runs can't really separate that.

Contributors
maflckoDrahtBotjeanpablojp
Labels
Fuzzing
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-09-09 07:56 UTC