5

What are the steps to take to restart a chain when finality is lagging behind and block productions is also slowing down.

In the logs there are prints like this:

Backing off claiming new slot for block authorship: finality is lagging.

This network is a testnet, and it is only using two validators.

I have the Sudo key for this chain what can I do to revive it?

(This is related to Recovering from stalled finality (babe/grandpa) and How to use noteStalled to recover from a finality lagging chain? but I'm looking for a checklist to recover the chain when I make another mistake)

Improve this question
1
  • more context is needed: how many validators are you using? can you share your source code? is ths a test network or production?
    – Nuke
    Commented Mar 17, 2022 at 21:04

1 Answer 1

Reset to default
5

The root cause of the issue here is stalled finality. The block production will deliberately slow down whenever finality is lagging for too long, this is to avoid creating long unfinalized forks. As such the steps to recovering from this would be the same as described in the two questions you linked.

Still, this "backoff" logic is configurable and as such you can either disable it for your chain or increase the amount of unfinalized blocks that are tolerated. As an example from the node service: https://github.com/paritytech/substrate/blob/027fc0666fa2f981a693a394f76dc9f4b87209a4/bin/node/cli/src/service.rs#L367-L368 this is one of the arguments that must be passed when starting the BABE worker and it controls the backoff logic. If you pass None then no backoff will ever happen, or you can also fine-tune the finality backoff: https://github.com/paritytech/substrate/blob/027fc0666fa2f981a693a394f76dc9f4b87209a4/client/consensus/slots/src/lib.rs#L720-L736. As you can see by default the backoff will trigger after 50 unfinalized blocks.

Improve this answer
2
  • 1
    Thanks this explains what is happening, I have still no idea how to restart the finality once it is lost.
    – Simson
    Commented Mar 18, 2022 at 12:59
  • 1
    If finality is stalled the reason must be that at least 1/3 of the validators are not voting, presumably because they are either not online or because they don't have the necessary keys to vote. The first step to recovering finality is figuring out who these validators are and why they aren't voting, this can be debugged using the grandpa_roundState RPC as explained in previous questions. Assuming that the current state is irrecoverable, e.g. keys have been deleted and are irrecoverable, then you can "hard-fork" the GRANDPA validator set using the noteStalled runtime API.
    – André
    Commented Mar 18, 2022 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.