8

While experimenting with a relay chain testnet, Decentration and I ran into this issue with multiple validators on the same node, stalled finality and got the chain into slow block-making mode.

We tried to restart finality with api.tx.grandpa.noteStalled and then managed to stall block production completely with this error, trying and failing to build the next block once a minute or so:

🙌 Starting consensus session on top of parent 0xdf1c27f8e8c9210f224b6849cead0e9ab257f73b78444007c746ae8c42cf5410
🎁 Prepared block for proposing at 24410 (6 ms) [hash: 0xcddfad6034e4e777627004e33da3fdb98ef3c0787182f1aefe6677f83be8726c; parent_hash: 0xdf1c…5410; extrinsics (2): [0xca0f…d245, 0x15a4…b00e]]
🔖 Pre-sealed block for proposal at 24410. Hash now 0x9dd9e87a0d6e77516f656eee094897f0ad64ef1cf4d0a62269ba617f0340d99b, previously 0xcddfad6034e4e777627004e33da3fdb98ef3c0787182f1aefe6677f83be8726c.
👶 New epoch 72 launching at block 0x9dd9…d99b (block slot 274112089 >= start slot 274112018).
👶 Next epoch starts at slot 274112618
Not applying authority set change forced at block #24405, due to pending standard change at block #23434
Error with block built on 0xdf1c27f8e8c9210f224b6849cead0e9ab257f73b78444007c746ae8c42cf5410: ClientImport("A pending forced authority set change could not be applied since it must be applied after the pending standard change at #23434")
💤 Idle (11 peers), best: #24409 (0xdf1c…5410), finalized #23324 (0x4169…831c), ⬇ 0.9kiB/s ⬆ 1.4kiB/s

Was this the right method but we got the parameters wrong, or should we have used something else?

Improve this question

1 Answer 1

Reset to default
7

The issue lacks details to know exactly what went wrong in the first place (why did finality stop?). That said here are the steps you should have taken in order to debug the problem.

Use the grandpa_roundState RPC to get the current state of GRANDPA, this should allow identifying who are the validators that have not voted yet. You could then use this information to look at the validators deployment and figure out what was wrong: were they running? did they have the appropriate keys in the keystore?

Assuming that the situation was irrecoverable (e.g. the keys were lost), then you could use the api.tx.grandpa.noteStalled extrinsic to forcefully change the validator set. This takes two parameters: the delay in blocks after which the forced validator set change happens (the change itself only happens at end of session, so this delay is on top of that), and the highest block that was finalized by the network so far.

This last parameter is of extreme importance since this is the block at which finality will be restarted, failure to provide a sensible block number here (i.e. a block number that was actually finalized) can lead to data inconsistency and bricking finality. I suppose this might have happened in your case, as the node is refusing to import a block with the error A pending forced authority set change could not be applied since it must be applied after the pending standard change at #23434, which implies that a call to noteStalled was done where the finalized number was higher or equal to #23434, yet the node is reporting #23324 as its best finalized number.

(FWIW I will improve the docs of grandpa.noteStalled in a follow-up PR).

Improve this answer
5
  • Thanks Andre. Sorry - my link to the multiple validator on a node issue probably wasn't enough description. Essentially, I was running 3 sets of validator keys on the same single node - and another 3 sets on a second node. The network had 9 validators and the multi-key nodes were all performing babe correctly. What I hadn't realised (until I found that link) was that they could only perform grandpa signatures for one for their three sets of keys, which meant that there were fewer than 2/3 of the validators finalising, which stalled finality. Should I expand on that in the question?
    – JelliedOwl - Paul
    Commented Feb 21, 2022 at 20:50
  • 1
    I believe #23324 was the block that finality stalled at due to my grandpa key mistake. After removing the additional broken validators form the elected set, we tried grandpa.noteStalled(5, 24359) - we were just after #24359 at the time but hadn't noted that it does nothing until the next era anyway.
    – JelliedOwl - Paul
    Commented Feb 21, 2022 at 21:00
  • From what you say though, I'm a bit surprised that finality didn't recover when I trimmed the elected set at the next era, but maybe I shouldn't be. Unfortunately, we don't have the broken database any more, so I can't recheck which validators had and hadn't grandpa-ed - but it's definitely useful information for next time!
    – JelliedOwl - Paul
    Commented Feb 21, 2022 at 21:02
  • 3
    Since finality got stalled at #23324 this is the block you should have used in noteStalled, since that was the best finalized block. The call should have been grandpa.noteStalled(5, 23324). Another alternative would have been to just temporarily move the existing keys to 9 validators (without the multi-key setup), which should have restored finality and then you could reduce the validator set to 3 authorities. Forced validator set changes should be a last measure resort imo.
    – André
    Commented Feb 21, 2022 at 21:53
  • Ah, so we completely misunderstood the call and how to use it. Thank you very much for clarifying!
    – JelliedOwl - Paul
    Commented Feb 21, 2022 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.