2

I have been running a testing network recently and they have encountered a very strange situation:

  1. the finalized has stopped at # 107379 and cannot be further confirmed.
  2. The logs of each consensus node are being printed:

Error importing block 0xb33e262e387c536589214e5c36f3aae1b69ad883ab79ec6408b428414249488f: block has an unknown parent

Error importing block 0xaa6dedf30fb9a23b46cc0c470ae002d29b2e378bd07c2c2270b1f1958c7fb3c6: block has an unknown parent

But the block hash is different

  1. And printing this error:

Verification failed for block 0xf53adc0a702940fae69ea237c9946be1e6549f66ceded06be0b8651464e82444 received from peer: 12D3KooWMV4EUiWq74RNaWk3rLhbWZTf9sDGk3htgZmVSdKgTLaD, "UnknownBlock: State already discarded for 0x80663f0ffcafa935fa605fe420385df0f7518de6b110b9cb164f26327aa8250b"

Here are the logs of my three consensus nodes, They seem to have been unable to synchronize blocks before, and each one is out of blocks, and the best is also different:

Consensus 1:

2023-09-08 10:08:15 Idle (40 peers), best: #107591 (0xbba3…3bc2), finalized #107379 (0x8121…4c3b), 901.7kiB/s 2.4MiB/s
2023-09-08 10:08:20 Idle (40 peers), best: #107591 (0xbba3…3bc2), finalized #107379 (0x8121…4c3b), 937.7kiB/s 2.7MiB/s
2023-09-08 10:08:24 Backing off claiming new slot for block authorship: finality is lagging.

Consensus 2:

2023-09-08 10:08:06 Pre-sealed block for proposal at 107592. Hash now 0x258807ab842841ff712860059fe2713b6d41880d38a8b9ea6d345dd3b176ce12, previously 0x005b096eb79e283b94d796a684aebce3354082bb7a5208308ebd053c6a97596e.
2023-09-08 10:08:06 Imported #107592 (0x2588…ce12)
2023-09-08 10:08:06 build valid transaction
2023-09-08 10:08:09 Idle (40 peers), best: #107592 (0x2588…ce12), finalized #107379 (0x8121…4c3b), 1.3MiB/s 2.5MiB/s
2023-09-08 10:08:14 Idle (40 peers), best: #107592 (0x2588…ce12), finalized #107379 (0x8121…4c3b), 1.4MiB/s 2.3MiB/s
2023-09-08 10:08:18 Backing off claiming new slot for block authorship: finality is lagging.
2023-09-08 10:08:19 Idle (40 peers), best: #107592 (0x2588…ce12), finalized #107379 (0x8121…4c3b), 1.0MiB/s 2.6MiB/s
2023-09-08 10:08:24 Idle (40 peers), best: #107592 (0x2588…ce12), finalized #107379 (0x8121…4c3b), 991.6kiB/s 2.5MiB/s

Consensus 3:

2023-09-08 10:08:15 Idle (40 peers), best: #107593 (0x1f79…b058), finalized #107379 (0x8121…4c3b), 1.8MiB/s 2.4MiB/s
2023-09-08 10:08:20 Idle (36 peers), best: #107593 (0x1f79…b058), finalized #107379 (0x8121…4c3b), 2.2MiB/s 2.2MiB/s
2023-09-08 10:08:23 Error importing block 0xb33e262e387c536589214e5c36f3aae1b69ad883ab79ec6408b428414249488f: block has an unknown parent
2023-09-08 10:08:25 Idle (22 peers), best: #107593 (0x1f79…b058), finalized #107379 (0x8121…4c3b), 2.4MiB/s 2.0MiB/s
2023-09-08 10:08:30 Idle (31 peers), best: #107593 (0x1f79…b058), finalized #107379 (0x8121…4c3b), 996.0kiB/s 1.9MiB/s

How did this happen? Can I recover?

This is the relevant log before and after # 107379:

2023-09-07 14:07:00 🎁 Prepared block for proposing at 107379 (198 ms) [hash: 0x27a3f1d99dbd10b741ac4d5794cf19e57a7f8c64537ae7172006968ec2802682; parent_hash: 0xb1db…fc77; extrinsics (19): [0xbf9e…c467, 0xe112…348c, 0x5e6f…c211, 0x85ce…966e, 0x2e92…574f, 0xffc6…f73f, 0xb740…f336, 0x2731…0553, 0xc4d5…1ee7, 0x5d58…53a2, 0x4d23…70af, 0x7eba…84fb, 0x9b78…c59b, 0xab14…a512, 0x2446…a436, 0x08fa…8b36, 0x79a3…ffb2, 0x46d4…8f56, 0x1d5f…bebc]]
2023-09-07 14:07:00 🔖 Pre-sealed block for proposal at 107379. Hash now 0x81216220258db8abb927bd2949ec743f0b7b80158fbed167d94e734c7bee4c3b, previously 0x27a3f1d99dbd10b741ac4d5794cf19e57a7f8c64537ae7172006968ec2802682.
2023-09-07 14:07:00 ✨ Imported #107379 (0x8121…4c3b)
2023-09-07 14:07:00 build valid transaction
2023-09-07 14:07:00 build valid transaction
2023-09-07 14:07:00 build valid transaction
2023-09-07 14:07:00 build valid transaction
2023-09-07 14:07:00 build valid transaction
2023-09-07 14:07:00 build valid transaction
2023-09-07 14:07:00 build valid transaction
2023-09-07 14:07:03 💤 Idle (61 peers), best: #107379 (0x8121…4c3b), finalized #107376 (0xd4ea…535d), ⬇ 7.9MiB/s ⬆ 21.8MiB/s
2023-09-07 14:07:06 🙌 Starting consensus session on top of parent 0x81216220258db8abb927bd2949ec743f0b7b80158fbed167d94e734c7bee4c3b
2023-09-07 14:07:06 build valid transaction
2023-09-07 14:07:06 🎁 Prepared block for proposing at 107380 (228 ms) [hash: 0x7af37f2e97c45ab3c9aa1f36512439e0913552c2df35e662162c31f9b76ec324; parent_hash: 0x8121…4c3b; extrinsics (16): [0x3a1d…32f4, 0xe9c7…4127, 0x7a22…c106, 0x5f63…a009, 0x7d4a…1409, 0x9dee…0750, 0xc40b…ed1e, 0x4fc6…ed4b, 0xeed0…273a, 0x6047…2dc3, 0x9548…76d2, 0x4429…e54a, 0xe6ee…66f3, 0x5ba0…5679, 0x9795…bb18, 0x11b5…b76d]]
2023-09-07 14:07:06 🔖 Pre-sealed block for proposal at 107380. Hash now 0x129ce5d5b7c84c2477e9ba2ed838a7082748f5742e6a623dc24b2244d56d30f9, previously 0x7af37f2e97c45ab3c9aa1f36512439e0913552c2df35e662162c31f9b76ec324.
2023-09-07 14:07:06 ✨ Imported #107380 (0x129c…30f9)
2023-09-07 14:07:06 build valid transaction
2023-09-07 14:07:06 build valid transaction
2023-09-07 14:07:06 build valid transaction
2023-09-07 14:07:06 build valid transaction
2023-09-07 14:07:06 build valid transaction
2023-09-07 14:07:06 build valid transaction
2023-09-07 14:07:07 build valid transaction
2023-09-07 14:07:07 ♻️ Reorg on #107380,0x129c…30f9 to #107378,0xdaed…f1e7, common ancestor #107377,0x517f…e155
2023-09-07 14:07:07 ✨ Imported #107378 (0xdaed…f1e7)
2023-09-07 14:07:07 build valid transaction
2023-09-07 14:07:07 build valid transaction
2023-09-07 14:07:07 build valid transaction
2023-09-07 14:07:07 build valid transaction
2023-09-07 14:07:08 build valid transaction
2023-09-07 14:07:08 build valid transaction
2023-09-07 14:07:08 build valid transaction
2023-09-07 14:07:08 build valid transaction
2023-09-07 14:07:08 build valid transaction
2023-09-07 14:07:08 build valid transaction
2023-09-07 14:07:08 build valid transaction
2023-09-07 14:07:08 build valid transaction
2023-09-07 14:07:08 💤 Idle (96 peers), best: #107378 (0xdaed…f1e7), finalized #107376 (0xd4ea…535d), ⬇ 10.8MiB/s ⬆ 24.3MiB/s
2023-09-07 14:07:12 🙌 Starting consensus session on top of parent 0xdaed352d7995e08e805aeaa9ddb7a98a19389a9a01f8ac4bb59e0c317924f1e7
2023-09-07 14:07:12 build valid transaction
2023-09-07 14:07:12 🎁 Prepared block for proposing at 107379 (110 ms) [hash: 0x1f2e488df8571486f8118e257f2bfe22d34736bed8e1c4ac20ca455c1cc111f6; parent_hash: 0xdaed…f1e7; extrinsics (6): [0xe9f9…728b, 0x6536…1594, 0xe112…348c, 0xcf1b…949e, 0xf0fb…96e9, 0x00e7…f2e1]]
2023-09-07 14:07:12 🔖 Pre-sealed block for proposal at 107379. Hash now 0xe84b2646b228dcec25ac46de1394ec32087452c1a8bd7e594b4b11321d604ee5, previously 0x1f2e488df8571486f8118e257f2bfe22d34736bed8e1c4ac20ca455c1cc111f6.
2023-09-07 14:07:12 ✨ Imported #107379 (0xe84b…4ee5)
2023-09-07 14:07:12 build valid transaction
2023-09-07 14:07:12 build valid transaction
2023-09-07 14:07:12 build valid transaction
2023-09-07 14:07:12 build valid transaction
2023-09-07 14:07:12 build valid transaction
2023-09-07 14:07:12 build valid transaction
2023-09-07 14:07:12 build valid transaction
2023-09-07 14:07:13 💤 Idle (143 peers), best: #107379 (0xe84b…4ee5), finalized #107377 (0x517f…e155), ⬇ 8.4MiB/s ⬆ 26.6MiB/s
2023-09-07 14:07:18 💤 Idle (150 peers), best: #107379 (0xe84b…4ee5), finalized #107377 (0x517f…e155), ⬇ 6.7MiB/s ⬆ 22.1MiB/s
2023-09-07 14:07:19 build valid transaction
2023-09-07 14:07:19 build valid transaction
2023-09-07 14:07:19 build valid transaction
2023-09-07 14:07:19 build valid transaction
2023-09-07 14:07:19 build valid transaction
2023-09-07 14:07:19 build valid transaction
2023-09-07 14:07:19 build valid transaction
2023-09-07 14:07:19 build valid transaction
2023-09-07 14:07:19 build valid transaction
2023-09-07 14:07:19 build valid transaction
2023-09-07 14:07:21 build valid transaction
2023-09-07 14:07:22 Potential safety failure: reverting finalized block (107378, 0xb1db73150718792c9319ed02084e768af6698aa46c54c69865f338c15d3ffc77)
2023-09-07 14:07:22 Block import error: Potential long-range attack: block not in finalized chain.
2023-09-07 14:07:22 💔 Error importing block 0x57038d95c1e0ee34c7029fc9f2a0cf215d73a775860888b3d7f535e386176cd6: consensus error: Import failed: Import failed: Potential long-range attack: block not in finalized chain.
2023-09-07 14:07:23 💤 Idle (150 peers), best: #107378 (0xb1db…fc77), finalized #107378 (0xb1db…fc77), ⬇ 8.2MiB/s ⬆ 26.1MiB/s
2023-09-07 14:07:25 build valid transaction
2023-09-07 14:07:25 Potential safety failure: reverting finalized block (107378, 0xb1db73150718792c9319ed02084e768af6698aa46c54c69865f338c15d3ffc77)
2023-09-07 14:07:25 Block import error: Potential long-range attack: block not in finalized chain.
2023-09-07 14:07:25 💔 Error importing block 0x57038d95c1e0ee34c7029fc9f2a0cf215d73a775860888b3d7f535e386176cd6: consensus error: Import failed: Import failed: Potential long-range attack: block not in finalized chain.
2023-09-07 14:07:28 💤 Idle (150 peers), best: #107378 (0xb1db…fc77), finalized #107378 (0xb1db…fc77), ⬇ 7.6MiB/s ⬆ 17.2MiB/s
2023-09-07 14:07:33 💤 Idle (150 peers), best: #107378 (0xb1db…fc77), finalized #107378 (0xb1db…fc77), ⬇ 5.0MiB/s ⬆ 14.1MiB/s
2023-09-07 14:07:36 🙌 Starting consensus session on top of parent 0x129ce5d5b7c84c2477e9ba2ed838a7082748f5742e6a623dc24b2244d56d30f9
2023-09-07 14:07:37 Timeout fired waiting for transaction pool at block #107380. Proceeding with production.
2023-09-07 14:07:37 build valid transaction
2023-09-07 14:07:37 🎁 Prepared block for proposing at 107381 (1543 ms) [hash: 0x9b1a83678ecc37dcf2bd71e33f99dd143d520160379a78a74254d3c3d4f83e7f; parent_hash: 0x129c…30f9; extrinsics (29): [0x6791…6414, 0xb1d3…2353, 0xe112…348c, 0xcf1b…949e, 0x27b9…d8ee, 0x96f7…dbc2, 0xd7c5…e841, 0x776d…1dca, 0x7cd0…1da3, 0xa044…4cba, 0x4689…023f, 0xab25…1ebc, 0xf0fb…96e9, 0xc56f…754a, 0xc705…c70d, 0x522e…0fce, 0x938d…4378, 0xbf0d…c4c3, 0xba24…fad7, 0x35c5…bc48, 0x4c21…b98a, 0x96d1…c4d2, 0x00e7…f2e1, 0x5df1…e352, 0x3f30…df0a, 0xc562…826d, 0xc9b9…cd40, 0x0ba1…f07a, 0xdcc7…9d82]]
2023-09-07 14:07:37 🔖 Pre-sealed block for proposal at 107381. Hash now 0x983216a66895ac6743cdbb794532260e9ff69a914da4164cafc69586d5c81eaf, previously 0x9b1a83678ecc37dcf2bd71e33f99dd143d520160379a78a74254d3c3d4f83e7f.
2023-09-07 14:07:37 ♻️ Reorg on #107379,0xe84b…4ee5 to #107381,0x9832…1eaf, common ancestor #107377,0x517f…e155
2023-09-07 14:07:37 ✨ Imported #107381 (0x9832…1eaf)
2023-09-07 14:07:37 build valid transaction
2023-09-07 14:07:37 build valid transaction
2023-09-07 14:07:37 build valid transaction
2023-09-07 14:07:37 build valid transaction
2023-09-07 14:07:37 build valid transaction
2023-09-07 14:07:37 build valid transaction
2023-09-07 14:07:38 💤 Idle (49 peers), best: #107381 (0x9832…1eaf), finalized #107378 (0xb1db…fc77), ⬇ 10.1MiB/s ⬆ 15.7MiB/s
2023-09-07 14:07:38 build valid transaction
2023-09-07 14:07:39 ✨ Imported #107381 (0x3e97…cb8e)
2023-09-07 14:07:43 💤 Idle (69 peers), best: #107381 (0x9832…1eaf), finalized #107378 (0xb1db…fc77), ⬇ 10.7MiB/s ⬆ 22.2MiB/s
2023-09-07 14:07:48 💤 Idle (77 peers), best: #107381 (0x9832…1eaf), finalized #107379 (0x8121…4c3b), ⬇ 5.6MiB/s ⬆ 21.1MiB/s
2023-09-07 14:07:51 build valid transaction
2023-09-07 14:07:51 ✨ Imported #107381 (0x3561…f012)
2023-09-07 14:07:53 💤 Idle (82 peers), best: #107381 (0x9832…1eaf), finalized #107379 (0x8121…4c3b), ⬇ 3.9MiB/s ⬆ 18.6MiB/s
2023-09-07 14:07:54 🙌 Starting consensus session on top of parent 0x983216a66895ac6743cdbb794532260e9ff69a914da4164cafc69586d5c81eaf
2023-09-07 14:07:54 build valid transaction
2023-09-07 14:07:54 🎁 Prepared block for proposing at 107382 (309 ms) [hash: 0x91b0246bca2e4f99d1f985ddb9a3254ae2765b6f9f7f6301e60f66dcde95ff03; parent_hash: 0x9832…1eaf; extrinsics (35): [0xd190…fe9e, 0x06f6…c607, 0x9235…b121, 0x698c…38d9, 0xf5c0…d13e, 0x8895…3051, 0x655b…5c57, 0x41d5…ad7c, 0x68a3…9f80, 0x4d9a…376e, 0xb551…a9c9, 0x67bc…9b03, 0x359b…54e1, 0xbf73…637e, 0xd72c…1c57, 0xacca…bdb5, 0x023f…5ce9, 0x3d35…d2b8, 0x9a68…e8e4, 0x8309…fc26, 0x2ed0…d02a, 0xce62…0dfa, 0x447c…93e8, 0x0ee5…bc9b, 0xa15c…e37d, 0x80a5…55f8, 0x4fbd…10d8, 0xfffc…62e6, 0xb5a8…8775, 0xb69e…3bf1, 0x4d21…b1b4, 0x6495…a8a2, 0xf40e…bd38, 0xc29a…3d0d, 0x7ee5…6b4b]]
2023-09-07 14:07:54 🔖 Pre-sealed block for proposal at 107382. Hash now 0x61a4bfa18ed9534ccd8811a0282be2724f16a792bec14eb4a240cc1723c4c2c8, previously 0x91b0246bca2e4f99d1f985ddb9a3254ae2765b6f9f7f6301e60f66dcde95ff03.
2023-09-07 14:07:54 ✨ Imported #107382 (0x61a4…c2c8)
2023-09-07 14:07:54 build valid transaction
2023-09-07 14:07:54 build valid transaction

Improve this question
4
  • What version are you running? and any weird logs before the block #107379 ?
    – Alex Bean
    Commented Sep 8, 2023 at 11:05
  • 1
    Also take a look into this previous one: substrate.stackexchange.com/questions/6337/… your node might ran out of disk space
    – Alex Bean
    Commented Sep 8, 2023 at 11:05
  • The version I am using is polkadot-v0.9.36. Then I checked my disk space and they are currently sufficient. A strange situation now is that I have a total of four validators, all of whom are online but do not confirm the blocks, or rather the confirmation is very slow
    – Yang Elden
    Commented Sep 9, 2023 at 1:40
  • I have re added the logs before and after # 107379. In addition to some error messages, I also sent a detail that the network upload and download rates used at that time were very high
    – Yang Elden
    Commented Sep 9, 2023 at 2:06

2 Answers 2

Reset to default
2

How did this happen?

From what I know there are two ways by which this kind of situation occurs:

  • When more than 1/3rd of the nodes are down due to which the system cannot come to any consensus and hence results in the stalling block finalization. In your case where there are only 3 nodes if any one node goes down it can lead to this problem.
  • Other situation which I know is when there are multiple validators from a single node can also cause this problem as to not to give more power to a single node.

Can I recover?

Yes you can recover from this.

  • In first case you need just need to start the node that is down and you are good to go.
  • In second case first you need to remove the faulty validators from the system, and go to extrinsics -> grandpa and in grandpa run note_stalled and the block finalization should start.

NOTE: In second case if the block production time has increased that of epoch_time, than you need to restart your chain from scratch. In case you don't know how to use note_stalled refer to How to use noteStalled to recover from a finality lagging chain?

Improve this answer
3
  • I checked my validators and found that they were all running normally. I don't know why they didn't confirm
    – Yang Elden
    Commented Sep 8, 2023 at 8:24
  • 1
    Hello @YangElden have you made the call note_stalled with the last finalised block as parameter ?
    – Alejandro - R 0 G U E
    Commented Sep 11, 2023 at 8:48
  • Hello @Alejandro Martínez. Yes, I have already called 'note'_ stalled ', parameter is 10, 107379. Unfortunately, it did not work.
    – Yang Elden
    Commented Sep 11, 2023 at 9:08
2

Check this previous StackExchange question Backing off claiming new slot for block authorship: finality is lagging, it has a great explanation of what might can happen.

Do you change something with the keys of a node? it might be something related to some node don't have the necessary keys to vote.

For investigate that query the grandpa_roundState RPC it returns the state of the current best round state as well as the ongoing background rounds, this should allow identifying who are the validators that have not voted yet. If something with the keys you might be able to recover it very easy.

Read this brilliant StackExchange answer from Andre for more detailled information: Recovering from stalled finality (babe/grandpa).

The last step to recover is to use as Alejandro suggested in the comments the extrinsic note_stalled, This will trigger a forced authority set change at the beginning of the next session, to be enacted delay blocks after that.

Be sure you pick the proper parameters for the extrinsic, take a look in this previous SE answer: How to use noteStalled to recover from a finality lagging chain?

Improve this answer
1
  • Thank you for your response. I have checked the missing validators through the grandpa_roundState RPC, and then I looked into my corresponding node. I found that it was working fine, and the corresponding keys were present in the keystore. However, they couldn't finality, which puzzled me. Later, I called note_stalled with a delay of 10, but I didn't realize that I needed to wait for the next session. So, I only waited for 10 blocks and gave up on the recovery when I saw it didn't work.
    – Yang Elden
    Commented Sep 13, 2023 at 2:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.