Over the past few days, we have encountered an issue with block finalization on the Ternoa testnet. It appears that the finalization process stopped, possibly due to one of the validators being offline. Subsequently, we noticed a significant reduction in block production time, which seems to align with the design outlined in the substrate code (https://github.com/paritytech/substrate/blob/027fc0666fa2f981a693a394f76dc9f4b87209a4/client/consensus/slots/src/lib.rs#L728).
Initially, the problem arose when there were two validators in the network. In an attempt to address the issue, we made a change to have only one validator active. While this seemed to have some impact, we believe there might be a more comprehensive approach to resolve this issue.
After careful analysis, we came across the grandpa.noteStalled extrinsic, which appears to be a potential solution for recovering from this situation. However, we also want to ensure that implementing this extrinsic does not carry any significant risks or implications for the network's stability.
Therefore, we would like to seek your insights and expertise on the following points:
Is the grandpa.noteStalled extrinsic the recommended approach to recover from the block finalization issue?
Are there any potential risks or downsides associated with calling the grandpa.noteStalled extrinsic?
What measures can we take to prevent similar issues from occurring in the future?
Explorer link - https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Falphanet.ternoa.com#/explorer