1

Following a relay runtime upgrade for westend 103000 (which contained a broken migration of the beefy keys), our chain stopped finalizing blocks (all validators, except 1, had the wrong keys on-chain).

While preparing the solution, we left the relay pass 500+ blocks of finality lag, which "forces" the validator to finalize the blocks that are 500+ behind. This helped to recover the network.

However, the other nodes (RPC nodes) are now refusing the finality:

2024-03-01 09:32:25.357 ERROR tokio-runtime-worker grandpa: GRANDPA voter error: safety invariant has been violated: Invalid operation in the pending changes tree: Finalized descendent of Tree node without finalizing its ancestor(s) first    
2024-03-01 09:26:29.306 ERROR tokio-runtime-worker sc_service::task_manager: Essential task `grandpa-voter` failed. Shutting down service.

(Also) On another network, with same conditions, we fixed it by using the only valid validator and note_stalled. The RPC nodes are also refusing finality:

2024-03-01 07:51:21.022 DEBUG tokio-runtime-worker sync::import-queue: Justification import failed for hash = 0xf27b397021806e4bccfe6e0ab3f19e768f3f0495b71187ba8aab3d63bd7dcf28 with number = 12977265 coming from node = PeerId("12D3KooWQRVQqakRuPxwLoaRHTpEaTDQSX9MqAatJ2Jgqd5aucvf") with error: Import failed: bad justification for header: invalid signature for precommit in grandpa justification
2024-03-01 07:51:21.022  INFO tokio-runtime-worker sync: 💔 Invalid justification provided by 12D3KooWQRVQqakRuPxwLoaRHTpEaTDQSX9MqAatJ2Jgqd5aucvf for #0xf27b…cf28

How can we force nodes to accept the justification or to simply consider those blocks as final ?

Improve this question
0

1 Answer 1

Reset to default
0

The first error on the RPC nodes is really weird, because it means that some later validator set change was finalized without finalizing a previous one. Is this only happening on the RPC nodes, i.e. the validators are working fine? Also what happens if you try to sync from scratch, does it lead to the same error?

What happened to the session keys in the migration (if you can dump the value of session.queuedKeys from pjs-ui)? I think that from GRANDPA POV probably all the keys (except one) ended up resolving to the same value, which leads to all kinds of invariants being violated.

Improve this answer
8
  • You can see the details of the incident there: moonsonglabs.notion.site/… What happened was the same as on Westend (and Rococo maybe?), where the migration concatenated the beefy key to the session keys of each validator. The issue was that this migration was included in multiple release without a versioning system to protect multiple runs
    – Crystalin
    Commented Mar 11 at 10:03
  • Also, we are currently mostly interested in finding a solution for nodes to catch-up with finalization, as nodes cannot sync it (even from scratch) anymore
    – Crystalin
    Commented Mar 11 at 10:03
  • In the incident writeup there are only 2 keys shown, so it doesn't allow me to figure out if all of the other validators have the same (invalid) GRANDPA key. But since they were able to finalize, I assume that's probably the case. I am not really sure how to recover from this since all grandpa invariants have been broken, it finalized something with an invalid set of keys. This would require some manual hacks to revert finality, in which case all nodes on the network would have to resync anyway.
    – André
    Commented Mar 11 at 14:44
  • I didn't write all the keys but all of them except the first one were broken. The finality went on on one of the network because of the 500 limit of finality lag, and on the other one because of "forced" finalization using grandpa.noteStalled and para force approval. You are suggesting a manual hack, do you know how ?
    – Crystalin
    Commented Mar 19 at 6:31
  • Resyncing from scratch is not solving the issue as the node will refuse the finalization and get stuck. I suppose it requires modifying the code to force the node to bypass it but I was hoping there would already be a solution for that
    – Crystalin
    Commented Mar 20 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.