7

Upon an extrinsic call one of our collators stopped to produce blocks and here's the error log snippet:

2022-03-21 14:48:50 [Parachain] panicked at 'Transaction will be valid in the future', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/22d40c7/frame/executive/src/lib.rs:388:17
2022-03-21 14:48:50 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] 💔 Error importing block 0xe3d7fea7538be50a0174e65ae6cf5be56821a3f1a6a09e8029c45c1736f099ba: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed

Since then it was out of sync and couldn't connect to the parachain network anymore:

2022-03-21 14:48:50 [Parachain] 💤 Idle (0 peers), best: #117412 (0x6db7…430e), finalized #117411 (0x4f1e…9e09), ⬇ 111.7kiB/s ⬆ 5.6kiB/s

The other collators are working fine. Restarting this collator didn't seem to help. The collators were built based on polkadot-v0.9.17 branch.

Why did it happen? Is there a way to recover from this? Delete the whole local parachain DB and restart it?

Here's the full log, thanks for any help!

2022-03-21 14:48:00 [Parachain] 💤 Idle (5 peers), best: #117409 (0xd230…1c6a), finalized #117408 (0xd2b2…85a8), ⬇ 4.1kiB/s ⬆ 0.3kiB/s
2022-03-21 14:48:00 [Relaychain] ✨ Imported #11906559 (0xa6d4…39f0)
2022-03-21 14:48:01 [Relaychain] ♻️  Reorg on #11906559,0xa6d4…39f0 to #11906559,0xf481…863c, common ancestor #11906558,0xc316…cb4c
2022-03-21 14:48:01 [Relaychain] ✨ Imported #11906559 (0xf481…863c)
2022-03-21 14:48:03 [Relaychain] 💤 Idle (46 peers), best: #11906559 (0xf481…863c), finalized #11906554 (0x6aa2…2c9a), ⬇ 357.9kiB/s ⬆ 387.9kiB/s
2022-03-21 14:48:05 [Parachain] 💤 Idle (5 peers), best: #117409 (0xd230…1c6a), finalized #117408 (0xd2b2…85a8), ⬇ 0.3kiB/s ⬆ 0.3kiB/s
2022-03-21 14:48:06 [Relaychain] ✨ Imported #11906560 (0xac51…57fc)
2022-03-21 14:48:06 [Parachain] Starting collation. relay_parent=0xac51c45933f7f20cd091043a409a0ac7eb85d28cc2cb932bce3f6bfd129257fc at=0x1d3031df9370bc788e04dd83b6ae06539cc33729a5041501e7f5973064c6e380
2022-03-21 14:48:06 [Parachain] 🙌 Starting consensus session on top of parent 0x1d3031df9370bc788e04dd83b6ae06539cc33729a5041501e7f5973064c6e380
2022-03-21 14:48:06 [Parachain] 🎁 Prepared block for proposing at 117411 (1 ms) [hash: 0x4644f38f1e294fdfa9548654eb517340c7948f7d77d20ba7293bbaaff574cddf; parent_hash: 0x1d30…e380; extrinsics (2): [0x20c3…76d4, 0x1a4e…62f0]]
2022-03-21 14:48:06 [Parachain] 🔖 Pre-sealed block for proposal at 117411. Hash now 0x4f1e697ca979e8f00015ba04f544cb87a31aaf22867a4dd0c36c4a2a69349e09, previously 0x4644f38f1e294fdfa9548654eb517340c7948f7d77d20ba7293bbaaff574cddf.
2022-03-21 14:48:06 [Parachain] ✨ Imported #117411 (0x4f1e…9e09)
2022-03-21 14:48:06 [Parachain] PoV size { header: 0.181640625kb, extrinsics: 3.525390625kb, storage_proof: 4.6123046875kb }
2022-03-21 14:48:06 [Parachain] Compressed PoV size: 7.3828125kb
2022-03-21 14:48:06 [Parachain] Produced proof-of-validity candidate. block_hash=0x4f1e697ca979e8f00015ba04f544cb87a31aaf22867a4dd0c36c4a2a69349e09
2022-03-21 14:48:06 [Relaychain] ✨ Imported #11906560 (0x963c…1165)
2022-03-21 14:48:06 [Parachain] Starting collation. relay_parent=0x963c3603694a50f889bd32ed8ca5681359647468b533b03839f98ac0e2601165 at=0x1d3031df9370bc788e04dd83b6ae06539cc33729a5041501e7f5973064c6e380
2022-03-21 14:48:06 [Parachain] 🙌 Starting consensus session on top of parent 0x1d3031df9370bc788e04dd83b6ae06539cc33729a5041501e7f5973064c6e380
2022-03-21 14:48:06 [Parachain] 🎁 Prepared block for proposing at 117411 (1 ms) [hash: 0x2088d388da88bfbf7f00a8b06d4c4a9e56326fd37502a76a3fbf10789f174e57; parent_hash: 0x1d30…e380; extrinsics (2): [0x00ae…e5fd, 0x9a44…b9cd]]
2022-03-21 14:48:06 [Parachain] 🔖 Pre-sealed block for proposal at 117411. Hash now 0x55662869e5f85b6e28906c30c8876ac79a08763cd9bf14c2798fba4d4334dab8, previously 0x2088d388da88bfbf7f00a8b06d4c4a9e56326fd37502a76a3fbf10789f174e57.
2022-03-21 14:48:06 [Parachain] PoV size { header: 0.181640625kb, extrinsics: 3.525390625kb, storage_proof: 4.6123046875kb }
2022-03-21 14:48:06 [Parachain] Compressed PoV size: 7.3955078125kb
2022-03-21 14:48:06 [Parachain] Produced proof-of-validity candidate. block_hash=0x55662869e5f85b6e28906c30c8876ac79a08763cd9bf14c2798fba4d4334dab8
2022-03-21 14:48:06 [Parachain] ✨ Imported #117411 (0x5566…dab8)
2022-03-21 14:48:08 [Relaychain] 💤 Idle (47 peers), best: #11906560 (0xac51…57fc), finalized #11906556 (0x126d…c413), ⬇ 311.1kiB/s ⬆ 482.3kiB/s
2022-03-21 14:48:10 [Parachain] 💤 Idle (5 peers), best: #117410 (0x1d30…e380), finalized #117408 (0xd2b2…85a8), ⬇ 0.4kiB/s ⬆ 16.0kiB/s
2022-03-21 14:48:12 [Relaychain] ✨ Imported #11906561 (0xa3e8…6e5d)
2022-03-21 14:48:13 [Relaychain] 💤 Idle (46 peers), best: #11906561 (0xa3e8…6e5d), finalized #11906557 (0x35bc…e52b), ⬇ 435.7kiB/s ⬆ 1.1MiB/s
2022-03-21 14:48:15 [Parachain] 💤 Idle (5 peers), best: #117410 (0x1d30…e380), finalized #117408 (0xd2b2…85a8), ⬇ 32 B/s ⬆ 0.1kiB/s
2022-03-21 14:48:18 [Relaychain] ✨ Imported #11906562 (0xdaa6…9f28)
2022-03-21 14:48:18 [Relaychain] ✨ Imported #11906562 (0x3638…c47e)
2022-03-21 14:48:18 [Relaychain] 💤 Idle (47 peers), best: #11906562 (0xdaa6…9f28), finalized #11906557 (0x35bc…e52b), ⬇ 316.9kiB/s ⬆ 981.9kiB/s
2022-03-21 14:48:20 [Parachain] 💤 Idle (5 peers), best: #117410 (0x1d30…e380), finalized #117409 (0xd230…1c6a), ⬇ 0.1kiB/s ⬆ 0.2kiB/s
2022-03-21 14:48:23 [Relaychain] 💤 Idle (48 peers), best: #11906562 (0xdaa6…9f28), finalized #11906559 (0xf481…863c), ⬇ 477.9kiB/s ⬆ 2.9MiB/s
2022-03-21 14:48:24 [Relaychain] ✨ Imported #11906563 (0x13f9…b3b5)
2022-03-21 14:48:24 [Parachain] Starting collation. relay_parent=0x13f9f26f921929e6c532795adabc763137ae71de26b72d02061a4fc764bcb3b5 at=0x4f1e697ca979e8f00015ba04f544cb87a31aaf22867a4dd0c36c4a2a69349e09
2022-03-21 14:48:24 [Relaychain] ✨ Imported #11906563 (0x9b1a…9732)
2022-03-21 14:48:24 [Parachain] Starting collation. relay_parent=0x9b1abde446f897d8600780120542fd5dc91a9efbf1e18df8423d45328dfa9732 at=0x4f1e697ca979e8f00015ba04f544cb87a31aaf22867a4dd0c36c4a2a69349e09
2022-03-21 14:48:25 [Parachain] 💤 Idle (5 peers), best: #117411 (0x4f1e…9e09), finalized #117410 (0x1d30…e380), ⬇ 90 B/s ⬆ 0.2kiB/s
2022-03-21 14:48:27 [Parachain] ✨ Imported #117412 (0x6db7…430e)
2022-03-21 14:48:27 [Parachain] ✨ Imported #117412 (0xc45f…ea14)
2022-03-21 14:48:28 [Relaychain] 💤 Idle (47 peers), best: #11906563 (0x13f9…b3b5), finalized #11906560 (0xac51…57fc), ⬇ 705.3kiB/s ⬆ 1.3MiB/s
2022-03-21 14:48:30 [Relaychain] ✨ Imported #11906564 (0x3d5b…2895)
2022-03-21 14:48:30 [Parachain] 💤 Idle (5 peers), best: #117411 (0x4f1e…9e09), finalized #117410 (0x1d30…e380), ⬇ 2.5kiB/s ⬆ 0.8kiB/s
2022-03-21 14:48:33 [Relaychain] 💤 Idle (48 peers), best: #11906564 (0x3d5b…2895), finalized #11906561 (0xa3e8…6e5d), ⬇ 526.9kiB/s ⬆ 998.2kiB/s
2022-03-21 14:48:35 [Parachain] 💤 Idle (5 peers), best: #117411 (0x4f1e…9e09), finalized #117410 (0x1d30…e380), ⬇ 99 B/s ⬆ 99 B/s
2022-03-21 14:48:36 [Relaychain] ✨ Imported #11906565 (0x31c4…6348)
2022-03-21 14:48:36 [Relaychain] ✨ Imported #11906565 (0x91d7…0b0c)
2022-03-21 14:48:38 [Relaychain] 💤 Idle (50 peers), best: #11906565 (0x31c4…6348), finalized #11906561 (0xa3e8…6e5d), ⬇ 618.7kiB/s ⬆ 1.7MiB/s
2022-03-21 14:48:40 [Parachain] 💤 Idle (5 peers), best: #117411 (0x4f1e…9e09), finalized #117410 (0x1d30…e380), ⬇ 24 B/s ⬆ 24 B/s
2022-03-21 14:48:42 [Relaychain] ✨ Imported #11906566 (0xb368…7b9d)
2022-03-21 14:48:42 [Relaychain] ♻️  Reorg on #11906566,0xb368…7b9d to #11906566,0x5e83…4a28, common ancestor #11906565,0x31c4…6348
2022-03-21 14:48:42 [Relaychain] ✨ Imported #11906566 (0x5e83…4a28)
2022-03-21 14:48:42 [Parachain] Starting collation. relay_parent=0x5e8371280c4ed92ffaf3df2218d6ef111eebbf78ce4d41a751e9fffc107f4a28 at=0x6db76adc2fb91abf78bf4610fa8e65f7b93dde99271c013b32499c525617430e
2022-03-21 14:48:42 [Parachain] 🙌 Starting consensus session on top of parent 0x6db76adc2fb91abf78bf4610fa8e65f7b93dde99271c013b32499c525617430e
2022-03-21 14:48:42 [Parachain] 🎁 Prepared block for proposing at 117413 (1 ms) [hash: 0x30e63829d63f1962ad28aac14426b8e7d917ddbc0d411288ed72a15b7e5bfb5b; parent_hash: 0x6db7…430e; extrinsics (2): [0x9956…4ce5, 0xe068…cc45]]
2022-03-21 14:48:42 [Parachain] 🔖 Pre-sealed block for proposal at 117413. Hash now 0x843ab65bbb5bd29c01a1411b29f39ffd975eb7f2fcb8e996532c7109b9ea2402, previously 0x30e63829d63f1962ad28aac14426b8e7d917ddbc0d411288ed72a15b7e5bfb5b.
2022-03-21 14:48:42 [Parachain] ✨ Imported #117413 (0x843a…2402)
2022-03-21 14:48:42 [Parachain] PoV size { header: 0.181640625kb, extrinsics: 3.525390625kb, storage_proof: 4.5537109375kb }
2022-03-21 14:48:42 [Parachain] Compressed PoV size: 7.32421875kb
2022-03-21 14:48:42 [Parachain] Produced proof-of-validity candidate. block_hash=0x843ab65bbb5bd29c01a1411b29f39ffd975eb7f2fcb8e996532c7109b9ea2402
2022-03-21 14:48:43 [Relaychain] 💤 Idle (50 peers), best: #11906566 (0x5e83…4a28), finalized #11906563 (0x13f9…b3b5), ⬇ 511.5kiB/s ⬆ 1022.4kiB/s
2022-03-21 14:48:45 [Parachain] 💤 Idle (5 peers), best: #117412 (0x6db7…430e), finalized #117411 (0x4f1e…9e09), ⬇ 51.0kiB/s ⬆ 18 B/s
2022-03-21 14:48:48 [Relaychain] ✨ Imported #11906567 (0x797d…4dcd)
2022-03-21 14:48:48 [Parachain] Starting collation. relay_parent=0x797de76cbc186beaf702321557fc6243f6ac751dd46cc5c247c15c530cb64dcd at=0x6db76adc2fb91abf78bf4610fa8e65f7b93dde99271c013b32499c525617430e
2022-03-21 14:48:48 [Relaychain] 💤 Idle (50 peers), best: #11906567 (0x797d…4dcd), finalized #11906563 (0x13f9…b3b5), ⬇ 423.2kiB/s ⬆ 706.8kiB/s
2022-03-21 14:48:50 [Parachain] panicked at 'Transaction will be valid in the future', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/22d40c7/frame/executive/src/lib.rs:388:17
2022-03-21 14:48:50 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] 💔 Error importing block 0xe3d7fea7538be50a0174e65ae6cf5be56821a3f1a6a09e8029c45c1736f099ba: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] panicked at 'Transaction will be valid in the future', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/22d40c7/frame/executive/src/lib.rs:388:17
2022-03-21 14:48:50 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] 💔 Error importing block 0xe3d7fea7538be50a0174e65ae6cf5be56821a3f1a6a09e8029c45c1736f099ba: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] panicked at 'Transaction will be valid in the future', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/22d40c7/frame/executive/src/lib.rs:388:17
2022-03-21 14:48:50 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] 💔 Error importing block 0xe3d7fea7538be50a0174e65ae6cf5be56821a3f1a6a09e8029c45c1736f099ba: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] panicked at 'Transaction will be valid in the future', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/22d40c7/frame/executive/src/lib.rs:388:17
2022-03-21 14:48:50 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] 💔 Error importing block 0xe3d7fea7538be50a0174e65ae6cf5be56821a3f1a6a09e8029c45c1736f099ba: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] panicked at 'Transaction will be valid in the future', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/22d40c7/frame/executive/src/lib.rs:388:17
2022-03-21 14:48:50 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] 💔 Error importing block 0xe3d7fea7538be50a0174e65ae6cf5be56821a3f1a6a09e8029c45c1736f099ba: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] panicked at 'Transaction will be valid in the future', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/22d40c7/frame/executive/src/lib.rs:388:17
2022-03-21 14:48:50 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] 💔 Error importing block 0xe3d7fea7538be50a0174e65ae6cf5be56821a3f1a6a09e8029c45c1736f099ba: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] panicked at 'Transaction will be valid in the future', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/22d40c7/frame/executive/src/lib.rs:388:17
2022-03-21 14:48:50 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] 💔 Error importing block 0xe3d7fea7538be50a0174e65ae6cf5be56821a3f1a6a09e8029c45c1736f099ba: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
2022-03-21 14:48:50 [Parachain] 💤 Idle (0 peers), best: #117412 (0x6db7…430e), finalized #117411 (0x4f1e…9e09), ⬇ 111.7kiB/s ⬆ 5.6kiB/s
2022-03-21 14:48:53 [Relaychain] 💤 Idle (50 peers), best: #11906567 (0x797d…4dcd), finalized #11906564 (0x3d5b…2895), ⬇ 875.0kiB/s ⬆ 1.2MiB/s
2022-03-21 14:48:54 [Relaychain] ✨ Imported #11906568 (0xa8f0…ba95)
2022-03-21 14:48:55 [Parachain] 💤 Idle (0 peers), best: #117412 (0x6db7…430e), finalized #117411 (0x4f1e…9e09), ⬇ 0.2kiB/s ⬆ 0.3kiB/s
2022-03-21 14:48:58 [Relaychain] 💤 Idle (49 peers), best: #11906568 (0xa8f0…ba95), finalized #11906565 (0x31c4…6348), ⬇ 672.7kiB/s ⬆ 745.4kiB/s
3
  • 2
    It seems after some time, it recovered from that by re-syncing the blocks again, I think it can be closed
    – Kailai
    Commented Mar 25, 2022 at 1:59
  • 3
    I believe this problem is due to the inconsistent cache in Cumulus. Beyond the "Transaction will be valid in the future" error, you may also encounter another error "State root mismatch that calculated". It's a very old problem that Parity has been trying to fix it for months. Usually a restart can help. In some rare case you will need to resync the parachain db. To mitigate, you can set --state-cache 0 when starting the collator, but bear in mind it will hurt the block production performance.
    – h4x3rotab
    Commented Mar 25, 2022 at 18:03
  • Thank you @h4x3rotab !
    – Kailai
    Commented Apr 24, 2022 at 9:40

1 Answer 1

4

This problem should be resolved now that the state cache has been removed in favor for a trie cache:

See: https://github.com/paritytech/substrate/pull/11407

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.