Idea: Faster P2P Network for Validators

BNB Smart Chain
1

1.Background

Low Latency is one of the important Goals of BNB Chain in 2025. According to the latest BSC hard fork plan, BSC will gradually upgrade the block interval from 3 seconds to 0.75 seconds in 2025.

With shorter block interval, the BSC network layer would face higher challenge on its reliability to deliver blocks/txs/votes in a shorter time. This post proposes an idea to alleviate these challenges by upgrading validator’s network topology.

2.Network Performance Requirement

Before introducing the proposal, let’s first analyze the network performance required for the future 0.75s block interval.

BSC’s block interval can be divided into the following time parts:

  1. When continuous block generation:
BlockInterval = Block Generate + Block broadcast
Block Generate = 2/3*BlockInterval = 2/3 * 3s = 2s
  1. When the proposer switches:
BlockInterval = Block Generate + Block broadcast + Block Validation
Block Generate = 1/2*BlockInterval = 1/2 * 3s = 1.5s

In the current 140M Gas block, assuming that the validator needs about 0.75s of validation time, then at 0.75s block, the time parts is expected to change proportionally to:

# Continuous block generation:
2s + 1s --> 0.5s + 0.25s

# When the node switches:
1.5s+0.75s+0.75s --> 0.375s+0.1875s+0.1875s

So with a block interval of 0.75s, the expected block propagation time is ~0.2s.

The above is the ideal time parts, which is dynamically adjusted when the actual chain is running. The only determined value is BlockHeader.Time:

  1. Block Generation increases, which is the source of delay broadcast that is often discussed. Validators selfishly conduct longer MEV activities or pack more profitable blocks;
  2. Block Broadcast increases, which is usually caused by network issues, low bandwidth or high latency network environment, message forwarding routing, and other issues;
  3. Block Validation increases, which is usually caused by machine performance issues or client versions;

FF vote broadcast is ignored because it can be executed in parallel with block generation.

Of course, all validators also need to optimize their connections and networks. Currently, we could start with p2p network optimization and monitoring to continuously promote it.

3.Specification

image
image1261×955 31.2 KB

Establish a standalone validator network for exchange block, voting, and other messages, with a priority broadcast mechanism to ensure that messages are forwarded through at most 1 hop and at most 1 cross-region;

Validators need to connect with each other, and should not be behind the sentry node. DDOS issue can be resolved by the service provider’s anti-DDOS product + rate limiting + whitelist.

As the block size increases, this solution becomes more suitable.

3.1.Validator Standalone Bootnode

The Validator Relay is just a normal P2P node, with some additional features.

  1. Only accept validator connections
    • Using Consensus Key Authorize, it is suitable for validator connection.
  2. Node DIscovery mechanism
    • Devp2p discv5, topic discovery, it’s used in Eth2.0 with tens of thousands of nodes;
    • Simple gossip+peers response, it’s used in solana;
    • libp2p stack, it’s used in IPFS, Eth2.0, polkadot;

3.2.Validator Nodes

  1. Validator finds peers from public P2P network and finds validator from Validator Standalone Network too.
  2. Validator sends block/vote to the next N proposers first then other left validators, and do not forward it again. If the timeout expires, query from the proposer.
  3. If the Network is abnormal, it will wait to obtain the required message from the public P2P network.

3.3.Message Types

The current standalone Network contains the following message types:

  1. Block Message.
  2. Vote Message.

In the future, we may consider broadcasting Transaction Messages on the network, but there are many scenarios to consider:

  1. Incoming Tx From RPC/P2P;
  2. Transactions that were not packaged in time by the previous Proposer;
  3. MEV Bundler Txs;

3.4.Delay Evaluation

avgDelay = Latency+1*(PriorityPeers*BlockSize/Bandwidth)
=0.15s+1*(1*0.5MB/125MB/s)=0.154 s

maxDelay = Latency+1*(Peers*BlockSize/Bandwidth)
=0.2s+1*(21*0.5MB/125MB/s)=0.284 s

It can be found that when the network block is small, latency/forwarding hops are the biggest issue, and when the network message size is larger, bandwidth becomes the new bottleneck.

4.FAQ

  1. Is it required that the validator uses signature verification? If so, does it need to expose services and security risks such as DDOS?
    Signature verification is still required. DDOS is a common security risk, but many measures can be taken to reduce the damage caused by the attack.

  2. In the case of proposer switching, should the block packaging time be reduced and the broadcast time be increased?
    The problem should not be serious. Longer continuous block generation reduces the risk.

  3. With a shorter block generation interval, are you worried that the delay broadcast problem will be more serious and the reorg problem will be more serious?
    The continuous block generation can be larger, like 16, which reduces the probability of reorg overall.

  4. Network latency issue
    It is best to avoid large network delays. The AWS network delay in EU and US is relatively low, at around 70ms.

    Of course, all validators also need to optimize their connections and networks.

  5. Does txs need to be sent to the next in-turn validator?
    This issue needs to be discussed. It is true that directly sharing the memory pool between validators has certain benefits, but it may conflict with the interests of current validators, MEV, etc.

2 Likes