LevelDB has been used as the default database to store the blockchain data for a long time for BSC network. And recently its repo is receiving very limited maintenance. And we had tried to use PebbleDB to replace it, but the testing result showed that the older version PebbleDB performance was not as good as LevelDB. Recently it is heard that PebbleDB has made some improvements with the new version. So we restarted to research if PebbleDB can be a good option. And below is the chasing block comparison testing we did with PebbleDB and LevelDB on BSC.
- Machine configuration
- EC2: m6i.4xlarge
- CPU: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz 16 core
- Mem: 64G
- SSD: GP3 3000 IOPS, 250M/S
./geth --db.engine pebble(or leveldb) --syncmode full
- LevelDB version: v1.0.1-0.20210819022825-2ae1ddf74ef7
- PebbleDB version: v0.0.0-20230928194634-aa077af62593
- Increased SST file size to improve block cache
The change is advised by the Pebble community. After adjusting the configuration it takes about 3 hours to warm up the read performance while 8 hours before.
The yellow one is LevelDB while the green one is PebbleDB.
Pebble avg: 371ms (avg value of 24 hours)
Leveldb avg: 380 ms (avg value of 24 hours)
Pebble has two significant sources of memory usage:MemTables and the Block Cache.Originally, Pebble used regular GO memory allocation for the memory backing both MemTables and the Block Cache. It puts significant pressure on the GO GC. An “allocation cache” was introduced to lessen the pressure on the GO GC. It really produced a dramatic reduction in GC pressure and a measurable performance improvement.
- CPU profile when using PebbleDB
PebbleDB pacing mechanism uses separate rate limiters for flushes and compactions. It works by attempting to flush and compact only as fast as needed and no faster. While LevelDB has no throttle the rate of flushes and compactions, runs as fast as possible, which easily results in huge latency spikes for writes and reads.
PebbleDB with the new version is nearly the same speed with LevelDB with less memory usage and compaction cost. It is really one better option. You can try with the BSC new release.