3

We're in the process of updating the Polimec runtime and node from the Polkadot SDK 1.0 to 1.6. We're doing the final tests before the merge and we are encountering an issue while syncing the currently live on-chain runtime with the new version of the node. We're starting the new node with the following parameters:

./target/release/polimec-parachain-node --name="NODE_NAME" \
        --chain=polimec-polkadot \
        --tmp \
        --pruning=archive \
        --database=paritydb \
        --rpc-port=9944 \
        --rpc-methods=safe \
        --rpc-cors=all \
        --rpc-max-connections=10000 \
        -- \
        --database=paritydb \
        --sync=warp

Everything works correctly until it gets to synchronizing Polimec block number #788846. At that specific block, the latest runtime came into effect after an upgrade. When the node reaches this block, it performs migrations that did not occur yesterday. Pasted image 20240320151218 1

(Textual version of the error message)

2024-03-20 15:13:08 [Parachain] πŸ₯ New pallet "Utility" detected in the runtime. The pallet has no defined storage version, so the on-chain version is being initialized to StorageVersion(0).
2024-03-20 15:13:08 [Parachain] πŸ₯ New pallet "Proxy" detected in the runtime. The pallet has no defined storage version, so the on-chain version is being initialized to StorageVersion(0).
2024-03-20 15:13:08 [Parachain] πŸ₯ New pallet "MessageQueue" detected in the runtime. The pallet has no defined storage version, so the on-chain version is being initialized to StorageVersion(0).
2024-03-20 15:13:08 [Parachain] πŸ₯ New pallet "Oracle" detected in the runtime. The pallet has no defined storage version, so the on-chain version is being initialized to StorageVersion(0).
2024-03-20 15:13:08 [Parachain] πŸ₯ New pallet "OracleOffchainWorker" detected in the runtime. The pallet has no defined storage version, so the on-chain version is being initialized to StorageVersion(0).
2024-03-20 15:13:08 [Parachain] 🚚 Pallet "XcmpQueue" VersionedMigration migrating storage version from 3 to 4.
2024-03-20 15:13:08 [Parachain] 🚚 Pallet "Identity" VersionedMigration migrating storage version from 0 to 1.
2024-03-20 15:13:08 [Parachain] running storage migration from version 0 to version 1.
2024-03-20 15:13:08 [Parachain] all 8 identities migrated
2024-03-20 15:13:08 [Parachain] Corrupted state at `0x45323df7cc47150b3930e2666b0aa313c522231880238a0c56021b8744a00743`: Error { cause: Some(Error { cause: Some(Error { cause: None, desc: "Not enough data to fill buffer" }), desc: "Could not decode `AsyncBackingParams::max_candidate_depth`" }), desc: "Could not decode `AbridgedHostConfiguration::async_backing_params`" }

And It arrives at a state that the collators have never sent to the relay, panicing with the following error:

====================

Version: 0.5.1-67b1de47b69

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic_handler::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::maybe_drop_included_ancestors
   7: frame_support::storage::transactional::with_transaction
   8: <cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}
   9: environmental::using_once
  10: <polimec_base_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter
  11: <polimec_base_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch
  12: <sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply
  13: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic
  14: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block
  15: polimec_base_runtime::api::dispatch
  16: environmental::using
  17: sc_executor::executor::WasmExecutor<H>::with_instance::{{closure}}
  18: <sc_executor::executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  19: sp_state_machine::execution::StateMachine<B,H,Exec>::execute
  20: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  21: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  22: <polimec_base_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_consensus_aura::AuraApi<__SrApiBlock__,sp_consensus_aura::sr25519::app_sr25519::Public>>::__runtime_api_internal_call_api_at
  23: sp_api::Core::execute_block
  24: <&sc_service::client::client::Client<B,E,Block,RA> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  25: <alloc::sync::Arc<T> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
  26: <cumulus_client_consensus_common::ParachainBlockImport<Block,BI,BE> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  27: <alloc::boxed::Box<dyn sc_consensus::block_import::BlockImport<B>+Error = sp_consensus::error::Error+core::marker::Send+core::marker::Sync> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
  28: <core::pin::Pin<P> as core::future::future::Future>::poll
  29: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}
  30: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  31: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  32: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  33: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  34: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
  35: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  36: tokio::runtime::task::core::Core<T,S>::poll
  37: tokio::runtime::task::harness::Harness<T,S>::poll
  38: tokio::runtime::blocking::pool::Inner::run
  39: std::sys_common::backtrace::__rust_begin_short_backtrace
  40: core::ops::function::FnOnce::call_once{{vtable.shim}}
  41: std::sys::unix::thread::Thread::new::thread_start
  42: <unknown>
  43: <unknown>


Thread 'tokio-runtime-worker' panicked at 'included head not present in relay storage proof', /home/lrazovic/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-parachain-system-0.7.0/src/lib.rs:1297

This is a bug. Please report it at:

    https://github.com/Polimec/polimec-node/issues/new

2024-03-20 15:13:08 [Parachain] Evicting failed runtime instance error=Runtime panicked: included head not present in relay storage proof
2024-03-20 15:13:08 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-03-20 15:13:08 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-03-20 15:13:08 [Parachain] Block prepare storage changes error: Error at calling runtime api: Execution failed: Runtime panicked: included head not present in relay storage proof
2024-03-20 15:13:08 [Parachain] πŸ’” Error importing block 0x50c819439db83195bbdf4df1f20ace0aa9b96f04fa1c0b86cf1a7b627ab302e2: consensus error: Import failed: Error at calling runtime api: Execution failed: Runtime panicked: included head not present in relay storage proof

The pallets in the list of "πŸ₯ New pallet '<PALLET_NAME>' detected in the runtime." currently have a STORAGE_VERSION set to 0 (which should be the default if not specified in the pallet, correct me if I'm wrong), in the SDK 1.0 they don't have any STORAGE_VERSION (Example provided for the Identity Pallet 1.1 and 1.6). Furthermore, the node tries to migrate the MessageQueue pallet, but this is currently not present in the live runtime, but it is present in the new runtime.

  1. Why does this problem only occur when trying to synchronise the block where the last upgrade took place and not during all previous upgrades?
  2. Why does the node perform these migrations/initialisations?
  3. How can this be solved?

1 Answer 1

5

Your problem is that you did not bump the spec_version. So, when the node reaches this block it assumes that the native runtime is compatible and tries to use it. However, it isn't compatible and it executes different code leading to the issues you are seeing.

Bump the spec_version and it will be solved.

Here you could also change NativeElseWasmExecutor to WasmExecutor. Then it will not try anymore to call into the native runtime and will always use the wasm runtime.

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.