During IBD, when doing pruning, prune 10% extra to avoid pruning again soon after #11658

pull luke-jr wants to merge 1 commits into bitcoin:master from luke-jr:ibd_prune_extra changing 1 files +9 −0

luke-jr commented at 9:08 AM on November 11, 2017: member

Pruning forces a chainstate flush, which can defeat the dbcache and harm performance significantly.

Alternative to #11359

During IBD, when doing pruning, prune 10% extra to avoid pruning again soon after

Pruning forces a chainstate flush, which can defeat the dbcache and harm performance significantly.

ac51a26bdc

fanquake added the label Validation on Nov 11, 2017

in src/validation.cpp:3399 in ac51a26bdc

3390 | @@ -3391,6 +3391,15 @@ void FindFilesToPrune(std::set<int>& setFilesToPrune, uint64_t nPruneAfterHeight
3391 |      int count=0;
3392 |  
3393 |      if (nCurrentUsage + nBuffer >= nPruneTarget) {
3394 | +        // On a prune event, the chainstate DB is flushed.
3395 | +        // To avoid excessive prune events negating the benefit of high dbcache
3396 | +        // values, we should not prune too rapidly.
3397 | +        // So when pruning in IBD, increase the buffer a bit to avoid a re-prune too soon.
3398 | +        if (IsInitialBlockDownload()) {
3399 | +            // Since this is only relevant during IBD, we use a fixed 10%

promag commented at 4:14 PM on November 15, 2017:

BTW, why 10%, in IBD could be even more no?

Sjors commented at 11:29 AM on December 7, 2017: member

Some IRC discussion.
Sjors referenced this in commit 03c20f3cc5 on Feb 10, 2018
Sjors cross-referenced this on Feb 10, 2018 from issue Prune more aggressively during IBD by Sjors
Sjors referenced this in commit c3eea611eb on Feb 10, 2018
Sjors referenced this in commit a9e8bb4640 on Feb 10, 2018
Sjors referenced this in commit a3716db4c4 on Feb 10, 2018
Sjors referenced this in commit 132c88e64f on Feb 19, 2018
Sjors referenced this in commit d49bab490a on Feb 19, 2018
Sjors referenced this in commit 86bef23e65 on Feb 19, 2018
Sjors referenced this in commit 22b81deb24 on Mar 12, 2018
Sjors referenced this in commit 82efbf1e8a on Mar 26, 2018
Sjors commented at 11:06 AM on May 28, 2018: member

tACK ac51a26 (although I rebased it on master).

This performs better than master in my tests so far: #12404 (comment) @promag wrote:

BTW, why 10%, in IBD could be even more no?

I tried this in #12404 and there seems to be negative side effects that I can't explain. I'd say this PR is a safer choice for now. @luke-jr you may want to borrow the release note I added to my PR, though that can be done later.
Sjors commented at 12:06 PM on June 14, 2018: member

I'm currently racing two Orange Pi's with 2 GB memory. This PR means cache rarely grows beyond 300 MB, but it's still a bit faster than master; after 5 days it's at 2018-01-27 vs. master is still at 2017-11-25.
laanwj added this to the "Blockers" column in a project

sipa commented at 3:05 AM on July 6, 2018: member

utACK ac51a26bdc69dc35e1f4f89b62c3134047e93bc1. This should be a safe first improvement. We can later make the overshoot value configurable or smarter, but this should already be a lot better.
MarcoFalke added the label Needs release notes on Jul 7, 2018
MarcoFalke removed the label Needs release notes on Jul 7, 2018
promag commented at 10:04 PM on July 7, 2018: member

utACK ac51a26. Improvement looks good.
ajtowns commented at 6:50 AM on July 12, 2018: contributor

utACK ac51a26bdc69dc35e1f4f89b62c3134047e93bc1
MarcoFalke cross-referenced this on Jul 12, 2018 from issue Benchmark IBD/reindex with pruning/txindex enabled by MarcoFalke
achow101 commented at 10:56 PM on July 13, 2018: member

utACK ac51a26bdc69dc35e1f4f89b62c3134047e93bc1
sipa merged this on Jul 14, 2018
sipa closed this on Jul 14, 2018
sipa referenced this in commit 9a1ad2c5cb on Jul 14, 2018
fanquake removed this from the "Blockers" column in a project

Sjors cross-referenced this on Jul 14, 2018 from issue Add a pruning 'high water mark' to reduce the frequency of pruning events by esotericnonsense
ajtowns commented at 7:40 AM on July 16, 2018: contributor

Post-merge tested ACK.

I ran this on some memory-constrained Linodes (dbcache=1000, prune=5000, and both connect= to a single nearby fully synced node; 2GB memory, 50GB disk), and saw a reasonably consistent 15% performance increase compared to not having this patch.
martinpgaunt commented at 7:48 AM on July 16, 2018: none

How use this ?
Bushstar cross-referenced this on Jul 16, 2018 from issue commits from bitcoin/master by Bushstar
bitcoin deleted a comment on Jul 16, 2018
Sjors commented at 8:30 AM on July 16, 2018: member

@martinpgaunt it just works, if you download Bitcoin Core 0.17 when it comes out in a couple of weeks (or a bit later).
jasonbcox referenced this in commit 966fa58918 on Sep 13, 2019
dagurval cross-referenced this on Oct 16, 2019 from issue Improve IBD when pruning by ptschip
Sjors cross-referenced this on Nov 19, 2019 from issue Flush without erasing cache during periodic and pruning flushes by sdaftuar
jonspock referenced this in commit 518957562a on Dec 22, 2019
proteanx referenced this in commit e8a3e85e10 on Dec 23, 2019
PastaPastaPasta referenced this in commit d7b71db2b9 on Jul 19, 2020
PastaPastaPasta referenced this in commit 0c7ba8e4c9 on Jul 24, 2020
PastaPastaPasta referenced this in commit 474e20ff38 on Jul 27, 2020
UdjinM6 referenced this in commit d3f7be5b84 on Jul 27, 2020
UdjinM6 referenced this in commit 0eb337028b on Jul 27, 2020
bitcoin locked this on Sep 8, 2021

Contributors

Labels

Linked (view graph)

#11359 Add a pruning 'high water mark' to reduce the frequency of pruning events #12404 Prune more aggressively during IBD #15265 Flush without erasing cache during periodic and pruning flushes