Compensate for memory peak at flush time #10126

1. 
   sipa commented at 7:08 PM on March 30, 2017: member

   It seems that our conversion from CCoinsCacheDB into a CDBBatch, and the following processing of the batch by LevelDB causes a spike in memory usage. In various experiments (all on x86_64, though), I've determined that this spike is between x1.5 and x1.9.

   Take this into account ahead of time before deciding whether to flush or not.

2. Compensate for memory peak at flush time 7228ce853d
3. laanwj added this to the milestone 0.14.1 on Mar 30, 2017
4. laanwj added the label Resource usage on Mar 30, 2017
5. 
   gmaxwell commented at 7:09 PM on March 30, 2017: contributor

   Please tag as 0.14.1.

6. laanwj added the label UTXO Db and Indexes on Mar 30, 2017
7. laanwj added this to the "For backport" column in a project

---

8. 
   theuni commented at 11:56 PM on March 30, 2017: member

   @sipa are any of your experiments relatively simple to reproduce? I'd like to step through the high usage spots to get a better feel for what's happening.

9. 
   sipa commented at 12:02 AM on March 31, 2017: member

   @theuni Using the top commit at https://github.com/sipa/bitcoin/commits/repmemusage I've done tests with reindexes and syncs from scratch, killing and restarting bitcoind at certain memory usage points. Then looked at (MAX_RSS - 280MB) / (last_reported_cache=_line).

10. 
    laanwj commented at 9:43 AM on March 31, 2017: member

    utACK 7228ce8

11. laanwj merged this on Mar 31, 2017
12. laanwj closed this on Mar 31, 2017

    ---

13. laanwj referenced this in commit 9ab26bf7e4 on Mar 31, 2017
14. laanwj referenced this in commit 0b5e162b84 on Mar 31, 2017
15. 
    morcos commented at 12:01 PM on March 31, 2017: member

    @sipa I still object to this PR without at least some increase in the default dbcache. I believe that in the event the mempool is full and there is no loan available then performance will be unnecessarily poor. I think 300 MB (with this new calculation) is approximately the bare minimum we can achieve without likely exceeding it by accident when connecting a block. It is not appropriate for a default.

16. morcos cross-referenced this on Mar 31, 2017 from issue Clean up calculations of pcoinsTip memory usage by morcos
17. laanwj removed this from the "For backport" column in a project

---

18. sipa deleted the branch on Jun 23, 2017
19. codablock referenced this in commit 9ab1787a83 on Sep 27, 2017
20. codablock referenced this in commit cfeab93642 on Oct 12, 2017
21. codablock referenced this in commit cfefd34f45 on Oct 24, 2017
22. UdjinM6 referenced this in commit f10760c621 on Nov 8, 2017
23. PastaPastaPasta referenced this in commit dbe610d8a6 on May 20, 2019
24. bitcoin locked this on Sep 8, 2021

Contributors

sipagmaxwelltheunilaanwjmorcos

Milestone
0.14.1