speed up initial sync

[abitrolly]

Syncing to current network is super slow:

...
2017-12-15 09:31:01.811 [INF] BMGR: Processed 2 blocks in the last 46.04s (2339 transactions, height 378002, 2015-10-08 13:45:41 +0000 UTC)
2017-12-15 09:31:12.670 [INF] BMGR: Processed 1 block in the last 10.85s (1367 transactions, height 378003, 2015-10-08 13:54:24 +0000 UTC)
2017-12-15 09:31:57.465 [INF] BMGR: Processed 1 block in the last 44.79s (1448 transactions, height 378004, 2015-10-08 14:07:13 +0000 UTC)
2017-12-15 09:32:31.195 [INF] BMGR: Processed 1 block in the last 33.72s (882 transactions, height 378005, 2015-10-08 14:14:45 +0000 UTC)

Is it okay to be that slow - 4 blocks/minute? Why is it so slow and how to profile it?

[abitrolly]

Is it possible to reuse bitcoin Merkle tree to avoid validating every block on initial sync?

[okdas]

The initial sync is really slow for me as well. Running for a week on six cores/SSD instance and I'm on 2016-05-07. I have addrindex enabled. The btcd process is always writing heavily to the disk. Obviously, there is a lot of room for optimization. Currently, the only way to speed sync up is to use SSD and a lot of CPU. We could try to upgrade the leveldb too, but I'm under impression that optimizations need to be done on the btcd side. I haven't tried to use addblock — theoretically, that can help to reuse .dat blocks.

[2pac1]

it is super slow and if it encounters ABC node it halts, and you need to stop it and re-start it again

[l0k18]

I dunno if there might be scope for automating this, but using btcctl and pointing it at a nearby, already synced bitcoin node using should help speed things along. The peer selection system in the peer library is quite conservative. I would wonder if you might see improvements by raising the number of connected nodes from 8 to maybe something like 16. Each connected peer mandates some amount of wait time for response timeouts and in this period the slot is occupied, hence my suggestion to maybe increase the number of active connections it makes... Assuming your network won't make this even worse with increased handshake chatter.

btcctl addpeer x.x.x.x:XXX add