Issue
We are experiencing a behaviour on our standalone/parachain node where the Network Bps on our cloud provider linearly increases throughout the lifecycle of our protocol. The increase continues without restraint until there's a tremendous amount of outbound/inbound traffic, accelerating the costs of the service.
Overview of protocol
Every session (6 hours) we run a certain multi-party computation (t-ECDSA DKG) offchain and begin using Substrate's network layer GossipEngine to gossip messages as part of the protocol to other peers on the network. This leads us to believe that we should be seeing a constant outbound Bps sent since we clean up the existing system by replacing it with the execution of a new MPC. However, the graph indicates this is not the case. Instead, it seems like we are not cleaning things up and instead every DKG's outbound traffic is compounding on one another.
Where we gossip messages
Gossiping
We gossip messages in 3 different places for some auxiliary protocols we run.
- For gossiping messages related to the DKG/Sign protocol
- For gossiping DKG public key for on-chain purposes
- For gossiping misbehaviour reports
Validating
In this latest graph we are using the GossipValidator::validate function with ProcessAndDiscard in hopes of avoiding any unnecessary outbound re-broadcasting. Nonetheless this hasn't fixed the situation.
Questions
- Perhaps there's parts of the gossip engine I haven't grasped yet but otherwise what are some recommended ways of handling/debugging these situations?
- Are there side-effects I should be aware for the gossip engine?
