0

I am trying to upgrade the Substrate network from Substrate commit 88c64e06471cc12aa9b25290f24d5566bcb5dd82 from Sept 2021 to a recent version in Substrate branch polkadot-v0.9.17 from Feb 2022.

When the updated node tries to sync from the network, it shows the following logs and panics.

Hash: given=81900b0feb7b205fbae0b1becbb65ffecd2cb0ee04c05b64c423c105c96a1724, expected=8ba92748e4c435cde701c1e426205b9a25cb5509300e2d9af30c841a271a8cba    

====================

Version: 3.0.0-d63622c0-aarch64-macos

   0: backtrace::backtrace::trace
   1: backtrace::capture::Backtrace::new
   2: sp_panic_handler::set::{{closure}}
   3: std::panicking::rust_panic_with_hook
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:610:17
   4: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:500:13
   5: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:139:18
   6: rust_begin_unwind
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:498:5
   7: core::panicking::panic_fmt
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:107:14
   8: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::final_checks
   9: tracing::span::Span::in_scope
  10: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block
  11: <node_polkadex_runtime::Runtime as sp_api::runtime_decl_for_Core::Core<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,u32>,node_polkadex_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<node_polkadex_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<node_polkadex_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<node_polkadex_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<node_polkadex_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<node_polkadex_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<node_polkadex_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<node_polkadex_runtime::Runtime>)>>>>::execute_block
  12: sp_api::runtime_decl_for_Core::execute_block_native_call_generator::{{closure}}
  13: std::thread::local::LocalKey<T>::with
  14: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
  15: sc_executor::wasm_runtime::RuntimeCache::with_instance
  16: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
  17: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
  18: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  19: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  20: sp_api::runtime_decl_for_Core::execute_block_call_api_at
  21: <node_polkadex_runtime::RuntimeApiImpl<__SR_API_BLOCK__,RuntimeApiImplCall> as sp_api::Core<__SR_API_BLOCK__>>::Core_execute_block_runtime_api_impl
  22: sp_api::Core::execute_block_with_context
  23: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  24: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  25: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  26: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  27: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  28: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  29: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  30: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  31: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  32: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  33: tokio::park::thread::CachedParkThread::block_on
  34: tokio::runtime::handle::Handle::block_on
  35: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  36: tokio::runtime::task::harness::Harness<T,S>::poll
  37: tokio::runtime::blocking::pool::Inner::run
  38: std::sys_common::backtrace::__rust_begin_short_backtrace
  39: core::ops::function::FnOnce::call_once{{vtable.shim}}
  40: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/boxed.rs:1694:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/boxed.rs:1694:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys/unix/thread.rs:106:17
  41: __pthread_deallocate


Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /Users/gautham/.cargo/git/checkouts/substrate-7e08433d4c370a21/22d40c7/frame/executive/src/lib.rs:488

This is a bug. Please report it at:

I am not able to understand what's causing the error.

What is the proper procedure to upgrade the Substrate dependencies of a production network?

2 Answers 2

4

The error is saying that at some point your node computed an incorrect state-root compared to the one that was written in the block header by a validator who authored the block. The cause can be multiple mistakes, but I recommend the following:

  1. try and purge-chain and re-sync from scratch.
  2. make sure you have bumped your spec_version after upgrading.
  3. if in doubt, use --execution Wasm to not use the native runtime. The validators always use Wasm execution method for authoring, and you can probably sync with them if you also execute in Wasm.

Having read my suggestions, I think 2 is the most plausible in your case. Also, not that if you managed to sync the chain with method 3 again, you are certainly doing something wrong regarding your runtime spec_version as well, and that needs fixing.

1
  • I did increment the spec_version. As soon as I added --execution wasm, it worked. :)
    – Gautham J
    Commented Mar 21, 2022 at 2:24
2

When you say "updated node" what exactly do you mean?

In substrate networks all nodes are updated at once - as soon as the network upgrade transaction is executed. This is because every node will run the WASM blob from that transaction. Your underlying native code maybe out of date however.

To upgrade your native code, it is never recommended to attempt to upgrade the Cargo manifest yourself, instead you should compile the correct branch/tag from Github.

With regards to the error this should not happen, but it may indicate that the database has become corrupted or you have compiled your node incorrectly and have effectively created a fork of the existing network.

If you have inadvertently created a fork, it is best to follow the steps set here to get the correct binary and purge the chain.

Another way to avoid such issues is to use the official Docker image to run your node.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.