We are experiencing a behaviour on our standalone/parachain node where the
Network Bps on our cloud provider linearly increases throughout the lifecycle of our protocol. The increase continues without restraint until there's a tremendous amount of outbound/inbound traffic, accelerating the costs of the service.
Overview of protocol
Every session (6 hours) we run a certain multi-party computation (t-ECDSA DKG) offchain and begin using Substrate's network layer
GossipEngine to gossip messages as part of the protocol to other peers on the network. This leads us to believe that we should be seeing a constant outbound
Bps sent since we clean up the existing system by replacing it with the execution of a new MPC. However, the graph indicates this is not the case. Instead, it seems like we are not cleaning things up and instead every DKG's outbound traffic is compounding on one another.
Where we gossip messages
We gossip messages in 3 different places for some auxiliary protocols we run.
- For gossiping messages related to the DKG/Sign protocol
- For gossiping DKG public key for on-chain purposes
- For gossiping misbehaviour reports
In this latest graph we are using the
GossipValidator::validate function with
ProcessAndDiscard in hopes of avoiding any unnecessary outbound re-broadcasting. Nonetheless this hasn't fixed the situation.
- Perhaps there's parts of the gossip engine I haven't grasped yet but otherwise what are some recommended ways of handling/debugging these situations?
- Are there side-effects I should be aware for the gossip engine?