Always gets 'Corrupted block database detected' with DB cache set to 1024MB #6502

bitcoin-qt version: 0.11.0 DB cache set from GUI: 1024MB (the issue does not happen with DB cache set to 512MB) OS: Win8.1 32bit

Scenario: Upgrade from an older 0.9.x version, with latest block about 32 weeks old. After start of the new 0.11.0 it started to fetch newer blocks, but get stuck after while. Graceful shutdown initiated manually and completed properly. During next start there is a message "Corrupted block database detected", rebuild chosen, causing to start fetch whole blockchain from the very beginning (6yrs 28wks). While it get to 2yrs and a few weeks it got stuck again. Graceful shutdown initiated and completed. During next start there is a message "Corrupted block database detected". There is a locked wallet. It is running on PC behind NAT. The configuration change is DB cache size set to 1024MB. Other programs incl. Litecoin client run well, no overheating, no suspected HW memory issues, enough free memory. There is a lot (hundreds) of messages like following in debug.log prior the graceful shutdown:

2015-07-31 23:18:01 ERROR: AcceptToMemoryPool: non-final

Everything goes wrong after this exception occures:

2015-08-01 08:17:19 
************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
C:\Program Files\BitcoinCore\bin\bitcoin-qt.exe in ProcessMessages()

What further information can I provide?

I gave it another try. Deleted all files from BitcoinCore data directory, but encrypted wallet (wallet.dat). Ran from cmd prompt, kept cache size at 1024MB, added '-debug' and '-txindex' flags: C:\Program Files\BitcoinCore>"C:\Program Files\BitcoinCore\bin\bitcoin-qt.exe" -debug -dbcache=1024 -txindex

The issue reproduced - CPU/HDD/NET utilization went to 0%, number of processed blocks got stuck, after graceful shutdown of BitcoinCore the block database is corrupted. It gets always stuck roughly after the same time, but not at the same place - once it is while processing blk00063.dat, another time it is while processing blk00057.dat.

Abnormality found in logfile:

2015-08-01 11:24:34 UpdateTip: new best=000000000000004bf315b7e079bc523f28e71318596e50f763c89c0596aaf038  height=233734  log2_work=69.897754  tx=16927445  date=2013-04-29 11:42:20 progress=0.093981  cache=598.4MiB(1792121tx)
2015-08-01 11:24:34   - Connect postprocess: 1.00ms [810.26s]
2015-08-01 11:24:34 - Connect block: 13.00ms [5739.49s]
2015-08-01 11:24:34   - Load block from disk: 3.00ms [964.21s]
2015-08-01 11:24:34       - Connect 178 transactions: 12.00ms (0.067ms/tx, 0.029ms/txin) [811.70s]
2015-08-01 11:24:34     - Verify 409 txins: 13.00ms (0.032ms/txin) [998.36s]
2015-08-01 11:24:34     - Index writing: 4.00ms [1498.18s]
2015-08-01 11:24:34     - Callbacks: 0.00ms [195.56s]
2015-08-01 11:24:34   - Connect total: 20.00ms [3370.03s]
2015-08-01 11:24:34   - Flush: 3.00ms [416.42s]
2015-08-01 11:24:34   - Writing chainstate: 1.00ms [178.60s]
2015-08-01 11:24:34 UpdateTip: new best=0000000000000042c685d3eec403f9e709d2db2d6030e0ae2a222bfb8fd185c9  height=233735  log2_work=69.897804  tx=16927623  date=2013-04-29 11:46:30 progress=0.093982  cache=598.4MiB(1792145tx)
2015-08-01 11:24:34   - Connect postprocess: 2.00ms [810.26s]
2015-08-01 11:24:34 - Connect block: 29.00ms [5739.52s]
2015-08-01 11:24:56 

************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
C:\Program Files\BitcoinCore\bin\bitcoin-qt.exe in ProcessMessages()       

2015-08-01 11:24:56 ProcessMessages(block, 99333 bytes) FAILED peer=15
2015-08-01 11:24:56 Reducing block download timeout for peer=15 block=00000000000001c164d7d8e85cde09c0f1bf269f02409e7941d662afa1dcd874, orig=1438496319431991 new=1438491896121087
2015-08-01 11:24:56 Requesting block 0000000000000050f52572c8ab7db497b76e771dcd966ada56e8344b1c9e7a37 (234174) peer=15
2015-08-01 11:24:56 Requesting tx 79c8f13fa691a245d1d0811f8d5614c3e319a8d29e2dba88a060d14c8c684a53 peer=15
2015-08-01 11:24:56 Requesting tx 265737edcfdf579b25523fdfb86724fcfec8e9a71c6a77bf4e1354714ddb410b peer=15
2015-08-01 11:24:56 Requesting tx 32f5916aed09b98859b015522341dcd4c08cc2765c1cce5b5130e745b62e8eec peer=15
2015-08-01 11:24:56 Requesting tx fc1616ee2b3a454bc90c8e509baba484269f7590ac69ae0667a1da5385f75a91 peer=15
2015-08-01 11:24:56 Requesting tx b366b2dfdb6f37478fc502fb554d29720298e4fb079498ed47a5194ac482cc06 peer=15
2015-08-01 11:24:56 Requesting tx e2d1cf9ce516eee08ee5c7bdb6d47c3a3a618a5540343188c6fd7af2221be94a peer=15
2015-08-01 11:24:56 Requesting tx 7befb9bef7a805a77a6265709a39cbf036cf06ab10f44d1de60b37f1ebcff508 peer=15
2015-08-01 11:24:56 Requesting tx 9409a5bb4ee2f435f976ae3a3538c1daa471fbef7c112b03fe25c0ab83d7cca5 peer=15
2015-08-01 11:24:56 Requesting tx 8467ed955d84a17f3e5f2717b2b54d900169708338aecfa8ad1969e2c0b2658b peer=15
2015-08-01 11:24:56 Requesting tx 4717af60416128e72837bffea64de3c09baf505665b96244a87aab3aa9a92291 peer=15
2015-08-01 11:24:56 Requesting tx 18a4a6fabeb66bc8cd92dd919f72b7aa8f528ed441949facb97d49f3fe1c5288 peer=15
2015-08-01 11:24:56 Requesting tx be31bf3b68ffa2022880823b3c377ac2dea5605768e023ce6c903c36d84b98ec peer=15
2015-08-01 11:24:56 Requesting tx 775a846b8bde4d4cfe5a100fd4ddec5717187877923aea3a2bd7ca2e8d6d10aa peer=15
2015-08-01 11:24:56 Requesting tx b31e5003200c760084cbe86466303ceb9b6412cd046c114282adb356f766c414 peer=15
2015-08-01 11:24:56 Requesting tx aefe102571de0e411be720955c370cad2ef849cc9303d2a6d51fbbf7eb6cfa91 peer=15
2015-08-01 11:24:56 Requesting tx b847951f627c47b65a7f839d193368f9e98614311ed36ce75936593c6ffb6f67 peer=15
2015-08-01 11:24:56 Requesting tx 441cee102650ca3697dc1bdcd76bad0cefd4dffc5ad2d0c4edad61684b62e6b9 peer=15
2015-08-01 11:24:56 Requesting tx ce42252f12249c80528ca754b5632bc9f7a974b773ec0f4f5cb375b61285d102 peer=15
2015-08-01 11:24:56 Requesting tx fbfb4c58d78241350ab9611a3b2f65fe3bece6c199a57f1706bb9ce99c33e4f6 peer=15
2015-08-01 11:24:56 Requesting tx 4886cef2279b5ed120c30824e4db81f828adb32a41741ccdfaf7a6bc764d6c9a peer=15
2015-08-01 11:24:56 Requesting tx 1f3e5d516297c3127ff64c35bf0b8267eaedf02b1ec1f4a1a4cb4e1ca7bd0a87 peer=15
2015-08-01 11:24:56 Requesting tx f76c84d97c565d3f2b608f7133801422aca495ea88ccb3b7f13bfb1ce8ea32b9 peer=15
2015-08-01 11:24:56 Requesting tx d9463d03339cf59acf849bb7c7d3d1711559900ac553a1ddbf9c296de9f14f20 peer=15
2015-08-01 11:24:56 Requesting tx 18a336c347ce53db5393243f611b22a85faf4cbac07c7ed6e38d1b2934a11c1a peer=15
2015-08-01 11:24:56 Requesting tx c6d143c1e87f14b681d6210f72d81afd8494a87a0fdaa61b104602be8800c393 peer=15
2015-08-01 11:24:56 Requesting tx 57f8569b99eb290b15d6b17ca733c59cca8d6dd027e5896ad55082b178b07f51 peer=15
2015-08-01 11:24:56 Requesting tx cd732e3b1550e02bb11acd2594fbdf7bf759bbf35d5d328df2660196c3021bfb peer=15
2015-08-01 11:24:56 sending: getdata (1009 bytes) peer=15
2015-08-01 11:24:56 received: tx (225 bytes) peer=16
2015-08-01 11:24:56 stored orphan tx d39e3f9393204711f667438f1dec473510c76ac882eea667765dbb3ef601b390 (mapsz 101 prevsz 203)
2015-08-01 11:24:56 mapOrphan overflow, removed 1 tx
2015-08-01 11:24:56 Requesting tx 7dba05627832250582ab1339d0b5474b37596c225d9052ce3f98063d260e014b peer=16
2015-08-01 11:24:56 Requesting tx 88df131651d49557a58b97d719092302af63089c3b121320020dd23e6b2d6914 peer=16
2015-08-01 11:24:56 Requesting tx cfeb61aa8e27a31a995d18c24d7a541081a45180981338048de7d10762b90640 peer=16
2015-08-01 11:24:56 Requesting tx d4f7013a1447dc0a7015cf06a72b027444ff03bbb0b03e7052dd9201ebbef734 peer=16
2015-08-01 11:24:56 Requesting tx b16e4623e8b9a42799913cc6b2909b6968d66c624b2bc548bcbd396f317e68a8 peer=16
2015-08-01 11:24:56 Requesting tx dffb058f663713d5656c85be492f3c443e9195ec343321f3c1fc07184bba4c45 peer=16
2015-08-01 11:24:56 Requesting tx 735bee94ef23a1e8ea80c259a4470818bebbbae6e05ced6cc62e47b7c74dcbcf peer=16
2015-08-01 11:24:56 Requesting tx d11000af0035a2386e7b3fdbbe6f008689f740cbd8b14baf2a9c3a61f122e876 peer=16
2015-08-01 11:24:56 Requesting tx fc670165ea30b1e887459b7a52f7581658c2dc4b39a9423fa4913cfdd391b59e peer=16
2015-08-01 11:24:56 Requesting tx 1e064f563a9514a706da006038b0305a0e9934cf939f9d196c31894c113d71b6 peer=16
2015-08-01 11:24:56 Requesting tx 136455d7e63ad18dadedf1d88bc1faa39359aec88b279869e2f21744f5013758 peer=16
2015-08-01 11:24:56 sending: getdata (397 bytes) peer=16
2015-08-01 11:24:56 received: tx (191 bytes) peer=1
2015-08-01 11:24:56 stored orphan tx c5e41ac83fdac5c7485d6d3d784eaf07003cb8dc9a480eb43fb490756b754856 (mapsz 101 prevsz 203)
2015-08-01 11:24:56 mapOrphan overflow, removed 1 tx
2015-08-01 11:24:56 Requesting tx 998e33544291d3037d6290a44019c5a2be5385735da2b5763ebf730f54da39a8 peer=1
2015-08-01 11:24:56 Requesting tx 6c6d686aea636cfffc0610bff9a3cb466f926d93db738387336d3400b442ef55 peer=1
2015-08-01 11:24:56 Requesting tx 427c276c2344875bcf846f5615a589b844718e157492b322a1bbdcb4876d2947 peer=1
2015-08-01 11:24:56 Requesting tx 6680ad94a108ca05e969ab93c920ec5baa07a457669d34d8b7736c663d2e4158 peer=1
2015-08-01 11:24:56 Requesting tx d69e450356b00fdbbcb9124afda250a94169c68ecfec39913fa9814b29c99a38 peer=1
2015-08-01 11:24:56 Requesting tx 8ed8b853d18103cb1d1076742aa22d95b1aabd1fa878a516a1c694686c8c093b peer=1
2015-08-01 11:24:56 Requesting tx a2991aa65cfdb8f95773bb5223efcc463778b46494fba0d7ac197e25f4b1ee67 peer=1
2015-08-01 11:24:56 Requesting tx 7c70359c8d78ea74ddf7742e3037421f2a4022a6fdf493cfd5c3b499329d7995 peer=1
2015-08-01 11:24:56 Requesting tx 4e22c7f361906c8b09f5208458850419904610eeb975cda7e981161435ac6c58 peer=1
2015-08-01 11:24:56 Requesting tx f494678858fc3651122578c75b36de7bcb19cda456d3288a1fa827acb8409fff peer=1
2015-08-01 11:24:56 Requesting tx e2bc2dc0e34d3b46522f086230fc4940da4151af3111de33f4f2c9db235ee051 peer=1
2015-08-01 11:24:56 Requesting tx e0f0dbb4a895ae1fdb840f85faf41d9ec39d7841d84ed082e747299c6ec1abf4 peer=1
2015-08-01 11:24:56 Requesting tx 11f6e5b9f8aa0327c5be39420b6b09e0e36c62a512679783f57eb5a860437741 peer=1
2015-08-01 11:24:56 sending: getdata (469 bytes) peer=1
2015-08-01 11:24:56 received: block (249162 bytes) peer=2
2015-08-01 11:24:56 received block 000000000000019d22642f4ce446bf587f00d430088a81c497a7a95f7b22abfa peer=2
2015-08-01 11:24:56 Reducing block download timeout for peer=2 block=00000000000000358cc742d7905ab9f480b527becbbaf55155b1a6295599197e, orig=1438496344467600 new=1438491896212093
2015-08-01 11:24:56 Requesting block 000000000000011687504e5081b778e751da3270f562c3eebd7d3be5fa6f127d (234175) peer=2
2015-08-01 11:24:56 Requesting tx 84d350519211aecf22133c9936a09f2b9dcb4caa0bfe563241e1fd86f5697e97 peer=2
2015-08-01 11:24:56 Requesting tx 46def80301f978621b74f8d3645fa452f0eb4ae7170dbaae3421af35df6dde43 peer=2
2015-08-01 11:24:56 Requesting tx 52829534d73de8ceb352fa2db5cf207e1e46b53f5d37c62137b8b0524e5689b2 peer=2
2015-08-01 11:24:56 Requesting tx 365748ba0e00db5195a892d9ce3f9dddde881e25ca7e131cb299df75bd97d59f peer=2
2015-08-01 11:24:56 Requesting tx 03ea98e5a460f4e1ea07ab860f68552126ff8077ae2804f0ecdcecbd3c5cd69d peer=2
2015-08-01 11:24:56 Requesting tx e0fab82c2e29a48cca035c24895dec57bf1b29e3654bbb7089da6288be0e3947 peer=2
2015-08-01 11:24:56 Requesting tx 7f454a436036b9929c475beed2ac407618d71f0178e16ac28a7ca5ca92511bb2 peer=2
2015-08-01 11:24:56 Requesting tx 62f1b581dcf9c7dc0cbff7a2f25ed1df57fe30851376cecac1e6a4d84fb98eb7 peer=2
2015-08-01 11:24:56 Requesting tx 6835d3773400f71395e77d289b703e83d4aac08ff84732b325b079070e2352b3 peer=2
2015-08-01 11:24:56 Requesting tx dd580e9c87c667598f11dcebe52f762007d6765e184c5aec3c553aa6dadf4880 peer=2
2015-08-01 11:24:56 Requesting tx 54c56613d2e1bab7e962f2d0917ce02c6eacbf4ed35c4b5cfde87e2760f5885f peer=2
2015-08-01 11:24:56 Requesting tx 05a81c3a08d7895bb8751f3da4f240fdb53dd1f437943be277506f4c9c2dcf60 peer=2
2015-08-01 11:24:56 sending: getdata (469 bytes) peer=2
2015-08-01 11:24:56 received: block (248823 bytes) peer=3
2015-08-01 11:24:56 received block 00000000000000750f638700ded83f72a88cb1ce95e7d4c4df8f2e08b930a0e2 peer=3
2015-08-01 11:24:56  25: For conf success > 0.85 need FeeRate >:           -1 from buckets 2.1e+015 - 2.1e+015  Cur Bucket stats  -1.#J%       0.0/(0.0+0 mempool)
2015-08-01 11:24:56   - Load block from disk: 23.00ms [964.23s]
2015-08-01 11:24:56 ERROR: ConnectBlock(): inputs missing/spent
2015-08-01 11:24:56 Misbehaving: 108.44.238.26:8333 (0 -> 100) BAN THRESHOLD EXCEEDED
2015-08-01 11:24:56 InvalidChainFound: invalid block=00000000000000750f638700ded83f72a88cb1ce95e7d4c4df8f2e08b930a0e2  height=233736  log2_work=69.897855  date=2013-04-29 12:12:55
2015-08-01 11:24:56 InvalidChainFound:  current best=0000000000000042c685d3eec403f9e709d2db2d6030e0ae2a222bfb8fd185c9  height=233735  log2_work=69.897804  date=2013-04-29 11:46:30
2015-08-01 11:24:56 ERROR: ConnectTip(): ConnectBlock 00000000000000750f638700ded83f72a88cb1ce95e7d4c4df8f2e08b930a0e2 failed
2015-08-01 11:24:56 InvalidChainFound: invalid block=00000000000000750f638700ded83f72a88cb1ce95e7d4c4df8f2e08b930a0e2  height=233736  log2_work=69.897855  date=2013-04-29 12:12:55
2015-08-01 11:24:56 InvalidChainFound:  current best=0000000000000042c685d3eec403f9e709d2db2d6030e0ae2a222bfb8fd185c9  height=233735  log2_work=69.897804  date=2013-04-29 11:46:30
2015-08-01 11:24:56 sending: reject (70 bytes) peer=3
2015-08-01 11:24:56 received: tx (1693 bytes) peer=8
2015-08-01 11:24:56 stored orphan tx 2933693a390aed29a60e7ba08166cdac30524656d271e20f377af8591a883e68 (mapsz 101 prevsz 211)
2015-08-01 11:24:56 mapOrphan overflow, removed 1 tx
2015-08-01 11:24:56 Reducing block download timeout for peer=8 block=00000000000000b2ae4111d46f836bb6e876c115214b4274f69ae0aa8a276fe2, orig=1438491871636002 new=1438491596336784
2015-08-01 11:24:56 Requesting tx 7c846fbe0ea817526e03274e50dcaea0319510833428a400f84cec0b20eaba86 peer=8
2015-08-01 11:24:56 Requesting tx c400a14f03fc078f4e1762b026a62da486c71640313d9c955862925f593294ba peer=8
2015-08-01 11:24:56 Requesting tx 1046d206a170fe90c62bab71cbc166af156d9fc4e8ef2dd13886ead8852c4ce7 peer=8
2015-08-01 11:24:56 Requesting tx 9018138a5146484ff50ce1d5471554d6d8f46121db19b21c2982c6cacf3d0acb peer=8
2015-08-01 11:24:56 disconnecting peer=3

Database corruption reported during subsequent BitcoinCore start (same command line used as above for starting BitcoinCore):

2015-08-01 13:31:19 Bitcoin version v0.11.0 (2015-07-10 19:23:55 +0200)
2015-08-01 13:31:19 Using OpenSSL version OpenSSL 1.0.1k 8 Jan 2015
2015-08-01 13:31:19 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2015-08-01 13:31:19 Default data directory C:\Users\Miroslav\AppData\Roaming\Bitcoin
2015-08-01 13:31:19 Using data directory C:\Users\Miroslav\AppData\Roaming\Bitcoin
2015-08-01 13:31:19 Using config file C:\Users\Miroslav\AppData\Roaming\Bitcoin\bitcoin.conf
2015-08-01 13:31:19 Using at most 125 connections (2048 file descriptors available)
2015-08-01 13:31:19 Using 2 threads for script verification
2015-08-01 13:31:19 Using wallet wallet.dat
2015-08-01 13:31:19 scheduler thread start
2015-08-01 13:31:19 init message: Verifying wallet...
2015-08-01 13:31:19 CDBEnv::Open: LogDir=C:\Users\Miroslav\AppData\Roaming\Bitcoin\database ErrorFile=C:\Users\Miroslav\AppData\Roaming\Bitcoin\db.log
2015-08-01 13:31:19 Bound to [::]:8333
2015-08-01 13:31:19 Bound to 0.0.0.0:8333
2015-08-01 13:31:19 Cache configuration:
2015-08-01 13:31:19 * Using 128.0MiB for block index database
2015-08-01 13:31:19 * Using 232.0MiB for chain state database
2015-08-01 13:31:19 * Using 664.0MiB for in-memory UTXO set
2015-08-01 13:31:19 init message: Loading block index...
2015-08-01 13:31:19 Opening LevelDB in C:\Users\Miroslav\AppData\Roaming\Bitcoin\blocks\index
2015-08-01 13:31:23 Opened LevelDB successfully
2015-08-01 13:31:23 Opening LevelDB in C:\Users\Miroslav\AppData\Roaming\Bitcoin\chainstate
2015-08-01 13:31:34 Opened LevelDB successfully
2015-08-01 13:31:54 LoadBlockIndexDB: last block file = 57
2015-08-01 13:31:54 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=565, size=93038652, heights=233487...234175, time=2013-04-28...2013-05-02)
2015-08-01 13:31:54 Checking all blk files are present...
2015-08-01 13:31:54 LoadBlockIndexDB: transaction index enabled
2015-08-01 13:31:54 LoadBlockIndexDB: hashBestChain=0000000000000042c685d3eec403f9e709d2db2d6030e0ae2a222bfb8fd185c9 height=233735 date=2013-04-29 11:46:30 progress=0.093969
2015-08-01 13:31:54 init message: Verifying blocks...
2015-08-01 13:31:54 Verifying last 288 blocks at level 3
2015-08-01 13:31:54 ERROR: DisconnectBlock(): added transaction mismatch? database corrupted
2015-08-01 13:31:54 ERROR: ApplyTxInUndo: undo data adding output to missing transaction
2015-08-01 13:31:54 ERROR: ApplyTxInUndo: undo data adding output to missing transaction
2015-08-01 13:31:54 ERROR: ApplyTxInUndo: undo data adding output to missing transaction

I can make the complete logfile available via AWS S3, uncompressed size ~380MB. Also, I can make available the corrupted DB files.

If it was Linux, I would attach gdb and produce backtrace and coredump of the process. However, it is Windows and I do not know what more information to collect. I can reproduce the issue again and follow instructions for further information gathering, if Bitcoin developers want me to do so.

As it seems as a memory allocation issue, maybe it cannot handle that large cache, I will try to make the cache of half size (512MB instead of 1024MB) and run again. Let me highlight that RAM is not exhausted, computer has 4GB (usable 3.5GB as of 32bit Windows) + swap and less than 50% of physical RAM is utilized.

UPDATE: With DB cache set to 512MB the issue does not happen.

bad_alloc is an out of memory error. dbcache of 1 GB apparently uses too much memory for your system.

2015-07-31 23:18:01 ERROR: AcceptToMemoryPool: non-final is a harmless message, see #5794.

Wladimir, thanks for your answer.

bad_alloc is an out of memory error. dbcache of 1 GB apparently uses too much memory for your system.

I still believe it is a software issue, as my system has a plenty of physical RAM available prior and during the occurence of the issue. As mentioned, let me highlight that RAM was not exhausted, computer has 4GB (usable 3.5GB as of 32bit Windows) + swap and less than 50% of physical RAM was utilized.

bad_alloc means out of virtual memory, which is not directly related to physical RAM, and is limited to 3 GB on 32bit Windows, I believe.

Right. And some virtual memory is used for other purposes, such as mmap'ing databases. 1GB of buffers just isn't going to work on 32 bit.

Windows task manager reported that the process was utilizing around 970MB at the time of issue. The memory allocation was growing steadily since start of the blockchain fetch. User-mode virtual address space for each 32-bit process is 2 GB [1], can increase to 3GB under special circumstances, not applicable in my case. Is the BitcoinCore 32bit application compiled with /LARGEADDRESSAWARE linker option ? If yes, it will be able to address memory up to 4GB on 64bit system and then I will retry on 64bit Windows to determine whether the issue is caused by 2GB memory limitation.

Reference: [1] https://msdn.microsoft.com/en-us/library/aa366778.aspx

Any reason for not just using the 64-bit version?

No. I just wanted to make sure that the issue is caused by the address space limitation and not a bug. If you think it is not worth spending further effort, let close the issue. I believe it would help to users if it was not possible to set 1GB DB cache on 32bit Windows system.

I have been challenged with this problem now for about six weeks. I am on Ubuntu 12.04 32-Bit with 1.7 gigs of RAM. Problem has been experienced with 10.2 and 11.0. Based on the discussion, I suspect Bitcoin Core does not work any longer for typical 32-Bit machines.

I kept getting the "Do you want to rebuild the block database now?" message. I tried -reindex with no solution.

I have an old Windows XP Box for which I have ran a node for years. Now, it always fails. Since it was not relevant to my concerns, I basically ignored it without diagnosis.

With the default dbcache size it works fine on 32-bit machines (I have several nodes on 32-bit). It should even work better with 0.11 as dbcache now correctly estimates the memory usage, whereas it used to underestimate it.

That isn't the case for me. Private bytes on a Win32 node rises over time from a startup value of ~230,000 K and eventually leads to a crash after a day or so of uptime, with something like this:

    MinGW Runtime Assertion
    Assertion failed!
    Program: C:\Program Files\Utilities\Bitcoin\bitcoin-qt.exe
    File: main.cpp, Line 1789
    Expression: hashPrevBlock == view.GetBestBlock()

or an "Error reading from database: Database I/O error". I suspect it doesn't have sufficient contiguous VM to allocate new objects or buffers.

In the first case above there was also an exception in the debug log:

    EXCEPTION: St9bad_alloc       
    std::bad_alloc       
    C:\Program Files\Utilities\Bitcoin\bitcoin-qt.exe in ProcessMessages()       
    ProcessMessages(block, 749190 bytes) FAILED peer=16309

I use Windows Vista x86 with the increaseuserva 2624 boot setting (i.e. VM is ~2624MB, not 2GB). I have downloaded the complete blockchain; I'm not using any parameters, just setting idle priority: C:\Windows\System32\cmd.exe /c start "Bitcoin Core" /LOW /MIN "C:\Program Files\Utilities\Bitcoin\bitcoin-qt.exe"

bitcoin-qt 0.11.0 manages to get up to over 2,000,000 K private bytes within 16 hours, and rises a further 300,000 K overnight, maintaining the same number of connections (50-60). By this time I have sent 12GB, seen 15,000+ peers, and received 1GB. Outbound traffic is highly variable, it is often around 1-200KByte/sec, but it can spike up to the full ~750KByte/sec line speed.

At this point bitcoin-qt goes into irregular periods of high CPU usage (one full core) for up to half a minute. When I break the process in a debugger, the thread concerned has the following on the top of the stack, which suggests to me that it's having trouble allocating memory:

    >   ntdll.dll!@RtlpLowFragHeapAllocFromContext@8()  + 0xd4 bytes    
        ntdll.dll!_RtlAllocateHeap@12()  + 0xaf bytes   
        msvcrt.dll!_malloc()  + 0x57 bytes  
        bitcoin-qt.exe!0158443f() 

In the debug log I get regular "free transaction rejected by rate limiter" and "inputs already spent", but I think that's usual. Perhaps this is useful: "UpdateTip: new best=00000000000000000e87ff685ac893b1c7b4a7159a365bbff0d35b18f3e1691f height=378137 log2_work=83.450831 tx=87128200 date=2015-10-09 13:46:19 progress=0.999999 cache=50.2MiB(7407tx)" [cache varies, this is about as high as it gets, and was just before a crash]

There's often "receive version message" coming every few seconds, e.g.: "/BitCoinJ:0.12SNAPSHOT/Satoshi:0.11.0/: version 70001, blocks=378123, us=127.0.0.1:8333, peer=14753"

There is always a "database Error" sandwiched between some other log messages on startup:

    2015-10-08 11:49:56 init message: Verifying wallet...
    2015-10-08 11:49:56 CDBEnv::Open: LogDir=C:\Users\GreenReaper\AppData\Roaming\Bitcoin\database ErrorFile=C:\Users\GreenReaper\AppData\Roaming\Bitcoin\db.log
    2015-10-08 11:49:56 scheduler thread start

However, this does not seem to stop Bitcoin working.

The system is on a private address with router DMZ and UPnP both available and working for IPv4, and a working IPv6 tunnel via SIxXS (used mostly by bitnodes.21.co, it seems).