database: investigate replacing leveldb with pebble

[Roasbeef]

Today we use a pure Go implementation of leveldb for the main database and relevant indexes. The DB has been able to keep up pretty well over the years, but it's maintained by more or less a single person, and hasn't received any significant commits over the past year or so. As a result, we should consider moving to a better maintained database that can be a drop in replacement for our current usage.

Enter pebble as pure Go KV database inspired by rocksdb (which was inspired by leveldb). One can view this as the 3rd generation of LSM (Log-Structured-Merge) Tree based KV stores. Compared to goleveldb pebble is actively maintained, and has been in used by Cockroachdb in production for ~3 years. The API is very similar to goleveldb, which means we won't need a significant overhaul.

Rather than replace it all at once, we should first add it as a new database option, with no migration path. After enough time, we can add a migration path, t hen eventually deprecate the old database type. At a glance, pebble also appears to be more performant than goleveldb.

[kcalvinalvin]

FWIW leveldb itself hasn't had that much development either. I don't think it necessarily means lack of maintenance.

That being said, I'm not opposed to having pebble as an option. If it ends up being more performant, that's a win. Though maybe have a big warning that pebble itself may not be consensus compatible with leveldb.

[Eoous]

If no enough performance improvement, i think there is no need to replace leveldb with pebble. To replace, we must do a lot of test for pebble to check whether there are internal bugs like missing data or incorrect values when insert or get.

Another point, maybe keepping building pebble means that there are bugs. leveldb is enough stable and safe.