BSC blockchain ecosystem has proved itself to be the most accessible and the most accessed platform for decentralized applications and open finance. But the large amount of accounts, contracts and transactions on BSC have brought challenges in scaling up. It is hard to keep nearly 18TB data of a BSC archive node on one machine with the Geth client. Many community members met difficulties when running their own archive node.
Introduction to Erigon
Erigon originally started as turbo-geth and an improvement of Geth. Later it has become to “erigon” after significant re-writing of the database structure, data model, and sync process.
Here is the key features of Erigon:
- More efficient state storage, less space for state data, uses a key-value database and stores accounts and storage in a simple way.
- Faster initial sync, uses a rearchitected full sync algorithm from Geth that is split into “stages”.
- Independent JSON-RPC daemon, RPC calls are extracted out of the main binary into a separate daemon supporting both local or remote DBs.
- Fewer Read/Write operations with the database when interacting with state data.
With the help of community developers Erigon has supported BSC by the end of last year.Below is our experience to run an archive node of BSC from the genesis block with Erigon client.
Hardware Specs
The AWS EC2 server we used includes:
- 16 vCPU
- 64 GB memory
- 50 GB disk for OS
- 8T GB nvme disk for BSC mainnet archive data
Building Erigon & RpcDaemon
Unlike Go-Ethreum’s client Geth, erigon has a separate RPC service which is called rpcdaemon. So we need to download the repo from github and checkout the devel branch(some bug fixes needed on this branch), then make to get the erigon and rpcdaemon binary separately.
cd /server
git clone GitHub - ledgerwatch/erigon: Ethereum implementation on the efficiency frontier
cd erigon
git checkout devel
make erigon
make rpcdaemon
Setting up archive services
It is simple to run an erigon archive node. Erigon runs in archive mode by default while you need to add “–gcmode=archive” for Geth. Erigon’s rpcdaemon runs in its own separate process.This brings many benefits including easier development, the ability to run multiple daemons at once, and the ability to run the daemon remotely. It is possible to run the daemon locally as well (read-only) if both processes have access to the data folder.We selected local DB mode, which RPC daemon runs on the same computer as Erigon.This mode uses shared memory access to the database of Erigon, which has better performance than accessing via TPC socket.
Provide both --datadir and --private.api.addr options when running commands:
- Run Erigon for BSC mainnet
nohup ./build/bin/erigon --datadir ./data --private.api.addr=localhost:9090 --networkid 56 --nat any --chain=bsc --staticpeers enode://22455eaa4eaf4c12dc2386840a590fdc632ef669f86a8e1adf8d1421ca2c2f69d6cef6ecc8b976f945d1fcbc3c8181edf75494b1ecb28eaf15f8b21413644385@54.217.65.254:30311,enode://c6e730d262ee0fe4e747423708c1553798c2d04e6002c3e29dc3cb46ef264f35e982ba1351d74a9cbff7c6da2b268c80bee62f8f27006b375639958cb2c28e26@3.114.158.196:30311,enode://1841077024720c251f58e6eeb10c2a3846db3610b2f4e8210e7035d0623f4ab6caef94c3bf215cb548e7c7e41d2755da33b63685de425e07aeb5cef017ea8cb5@52.51.36.24:30311,enode://d509ceb8272be80280cc8a8dd9cd01f99b2ba37c611218476794d61fc08b114f30031e1a9eb95ae1323e354b6f2f4d2a9bfb7207a48ef1b4fb84182ab21d1a89@52.213.84.200:30311,enode://d257b0f8d362161bb4a18b13253b1f6abb273a89d85fed73458f84dbf52d14ce556fd32ac07cbc49cd69d99c03ceddeccab919e10351c9d4e795a4c36404aef7@3.218.112.8:30311,enode://eab705bab55816ad0a39ac2080d413fc337117413f76eb5bbf4505edf7545eaed13b7562657310831b86a2ea1e04ee34c8fce8eb19cca2aaeb6ddd61759e68fb@35.72.50.231:30311,enode://2523b1dfdd9a70aa806fd558eebf7e6dbe90882178a4eb0325336277e415e44cef1aae8ecf494313f43b9eabae4416ac7b7281af027763af26cb821d63ae844e@54.247.157.151:30311,enode://38144f02adbe8817b2b0f241f921491bc612dd8cd603fa9cd3faa31c76697ad5e00297cba38d0a8f594752c56c9a7e84f53d7f71c0b37b5680cea5e732b6b946@34.243.164.24:30311,enode://2aadea249d51e076060c97cbc293a2d90052d1786062f137aa8f4afe06996b674ecdbe660d3033b3d46d9cdaf2c3e3db6c031e218edc5a2613caa7f9705774f9@34.193.176.125:30311,enode://7cf8161c39e500f91d0311ad639d87fe52e938e6f3e64f381b0f0bb6f1f24161c65b548e406853e7e3c684bf65fe39d7aa8350b62106da91a301ef2044a78208@35.75.218.3:30311,enode://bcccac7dfbd21bd1dbbb7bf64ef7af6986520091f00320905e5919d12167fc8c94698099946933747ad2ea24745d35ebc980f9ba1475a839c83574d5ccf318d9@34.196.94.250:30311,enode://e8a97e90dae05940a04a37ddad445b1d1f270e1a0ec1caf31f77ecd9a664db3c0f4750914afbb6f017d214ecef4a5f991b2f16987a76ec38ceb06f7f8f2f9f76@52.17.96.222:30311,enode://4b79ca4c3457c689edd4d1739a8bd0b0e410824ef4f177db574c7f854404b4728c21a832378ef4d754901c1fd20d3c491325adecff1f6ffa79f511c59ab5b4ab@18.200.9.45:30311,enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@13.114.81.214:30311 --metrics 2>&1 &
- Run rpcdaemon
./build/bin/rpcdaemon --datadir=/server/erigon/build/bin/node --private.api.addr=localhost:9090 --http.api=eth,erigon,web3,net,debug,trace,txpool
- Updating node
Updating erigon is fairly straightforward, simply pull the latest changes from github, run make, and restart our services.
I would suggest paying more attention when updating. Although the Erigon team does their best to ensure that Erigon is safe and stable, it’s always best to be safe. Also, Some updates might not be reversible, when that happens, the startup of your node after updating will include a migration of data. However, if you want to downgrade, you would likely have to re-sync.
Disk space usage
It took about two months for erigon to catch up with the latest block of BSC mainnet. And the disk space used is about 4.8TB (much smaller than 18TB with Geth client) when running to 15243567 block height for BSC mainnet.
Performance
- 20,000qps stress test, most of the delays are around 5 milliseconds, tested with 3 rpc methods(eth_call/eth_getBalance/eth_getTransactionCount):
Issues met
Below are the issues we met with Erigon client:
- 4TB size limitation
Erigon crashed when running to the block height of 141335260. It can be seen that the error “map size limit reached” was hit when the data size is nearly 4TB.So we modified the map size and restarted the erigon client. Another coredump was hit in mdbx library.We raised one issue for mdbx: 4Tb assert · Issue #260 · erthink/libmdbx · GitHub with the community developers, we reproduced the issue locally with the debug version provided by the community. The issue was located smoothly with our coredump file. It is one problem of mdbx-go. And libmdbx developers have released a new fix for it in version 0.11.4. And also the fix has been merged to erigon “devel” branch now.
- OOM issues
Our archive nodes got OOM during block execution. One is “block=13506637”, the other is “13562001”. Talking with community developers, it might be a memory leak issue in erigon or an issue in libmdbx. It is suggested that we could bypass it by resetting the process. And it seems it’s not critical and doesn’t affect the sync process. The issue was not hit after that, but needs to be investigated later.