2

We have a few collators for the Kintsugi parachain actively syncing and producing blocks when they suddenly fall-over due to the following error:

Apr 02 21:06:00 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: 2022-04-02 21:06:00 [Relaychain] ✨ Imported #12082344 (0x9ef8…3509)
Apr 02 21:06:01 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: 2022-04-02 21:06:01 [Parachain] 💤 Idle (27 peers), best: #665718 (0x7648…e130), finalized #665718 (0x7648…e130), ⬇ 15.9kiB/s ⬆ 16.8kiB/s
Apr 02 21:06:03 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: 2022-04-02 21:06:03 [Relaychain] 💤 Idle (50 peers), best: #12082344 (0x9ef8…3509), finalized #12082341 (0xbdf5…24bb), ⬇ 312.3kiB/s ⬆ 549.4kiB/s
Apr 02 21:06:06 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: 2022-04-02 21:06:06 [Parachain] 💤 Idle (27 peers), best: #665718 (0x7648…e130), finalized #665718 (0x7648…e130), ⬇ 7.6kiB/s ⬆ 8.3kiB/s
Apr 02 21:06:06 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: 2022-04-02 21:06:06 [Relaychain] ✨ Imported #12082345 (0xa6b2…9cb1)
Apr 02 21:06:06 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: 2022-04-02 21:06:06 Off-chain worker started on block 665719
Apr 02 21:06:06 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: 2022-04-02 21:06:06 [Parachain] Starting collation. relay_parent=0xa6b2df698486715d55479010149e18cd14e11196553ad920752e9e181fd69cb1 at=0x2ba467927cfdf92d41997301b1b9aa38f09cef2c365ed9676fafa71a4f5eb669
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: 2022-04-02 21:06:07 [Parachain] panicked at 'Storage root must match that calculated.', /builds/interlay/interbtc/.cargo/git/checkouts/substrate-f4423eebfa0046a3/22d40c7/frame/executive/src/lib.rs:488:9
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: 2022-04-02 21:06:07 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: WASM backtrace:
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]:     0: 0x32b297 - <unknown>!rust_begin_unwind
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]:     1: 0x1173a - <unknown>!core::panicking::panic_fmt::hce09957bf3e88e08
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]:     2: 0xc0f4f - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h2b20ae17664d24fe
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]:     3: 0x14ff47 - <unknown>!Core_execute_block
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]:     
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: 2022-04-02 21:06:07 [Parachain] 💔 Error importing block 0x0520227ed2b74353dc2517aaf3a59e66e874fbb391a849faf6a1640812ff2f29: consensus error: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]: WASM backtrace:
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]:     0: 0x32b297 - <unknown>!rust_begin_unwind
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]:     1: 0x1173a - <unknown>!core::panicking::panic_fmt::hce09957bf3e88e08
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]:     2: 0xc0f4f - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h2b20ae17664d24fe
Apr 02 21:06:07 ubuntu-s-2vcpu-4gb-amd-fra1-01 kintsugi[309550]:     3: 0x14ff47 - <unknown>!Core_execute_block

No runtime upgrades were performed and the collators all have --execution=wasm set. This appears to be unrecoverable until we manually restart the node.

1 Answer 1

3

This is probably related to the broken state cache. See here for more information. For now you can run your nodes with --state-cache-size 0 to prevent this bug.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.