I require some insight into how to troubleshoot the following runtime error which only occurs when I try to peer with my running node.

Using substrate v3.0.0 monthly-10-2021

I should also add that if I'm running a single node as a validator, I get no block finalization if that has anything to do with the following. Peering with a node results in the following error for the node trying to start and peer:

Version: 1.0.0-fce706a-x86_64-linux-gnu

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:610:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:502:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/sys_common/backtrace.rs:139:18
   4: rust_begin_unwind
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:498:5
   5: core::panicking::panic_fmt
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:107:14
   6: <pallet_staking::pallet::pallet::GenesisConfig<T> as frame_support::traits::hooks::GenesisBuild<T>>::build
   7: std::thread::local::LocalKey<T>::with
   8: sp_state_machine::basic::BasicExternalities::execute_with_storage
   9: <node_runtime::GenesisConfig as sp_runtime::BuildStorage>::assimilate_storage
  10: sp_runtime::BuildStorage::build_storage
  11: <sc_chain_spec::chain_spec::ChainSpec<G,E> as sp_runtime::BuildStorage>::build_storage
  12: sc_service::client::client::Client<B,E,Block,RA>::new
  13: sc_service::builder::new_full_parts
  14: node::service::new_partial
  15: node::service::new_full
  16: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  17: tokio::park::thread::CachedParkThread::block_on
  18: tokio::runtime::thread_pool::ThreadPool::block_on
  19: tokio::runtime::Runtime::block_on
  20: sc_cli::runner::Runner<C>::run_node_until_exit
  21: node::command::run
  22: node::main
  23: std::sys_common::backtrace::__rust_begin_short_backtrace
  24: std::rt::lang_start::{{closure}}
  25: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ops/function.rs:259:13
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:406:40
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:370:19
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panic.rs:133:14
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/rt.rs:128:48
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:406:40
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:370:19
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panic.rs:133:14
             at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/rt.rs:128:20
  26: main
  27: __libc_start_main
  28: _start

Thread 'main' panicked at 'Expected Ok(_). Got Err(
    Module {
        index: 16,
        error: 4,
        message: Some(
)', /home/turtle/.cargo/git/checkouts/substrate-7e08433d4c370a21/bf9683e/frame/staking/src/pallet/mod.rs:518

This is a bug. Please report it at:


Some information: I am running a dev chain and my testnet_genesis looks something like this:

fn testnet_genesis(
    wasm_binary: &[u8],
    initial_authorities: Vec<(
        AccountId, // Stash
        AccountId, // Controller
    root_key: AccountId,
    initial_nominators: Vec<AccountId>,
) -> GenesisConfig {
    let mut rng = rand::thread_rng();
    let endowed_accounts = vec![
    let stakers = initial_authorities
        .map(|x| (x.0.clone(), x.1.clone(), STASH, StakerStatus::Validator))
        .chain(initial_nominators.iter().map(|x| {
            use rand::{seq::SliceRandom, Rng};
            let limit = (MAX_NOMINATIONS as usize).min(initial_authorities.len());
            let count = rng.gen::<usize>() % limit;
            let nominations = initial_authorities
                .choose_multiple(&mut rng, count)
                .map(|choice| choice.0.clone())
            (x.clone(), x.clone(), STASH, StakerStatus::Nominator(nominations))

    let num_endowed_accounts = endowed_accounts.len();
    const ENDOWMENT: Balance = 10_000_000 * DOLLARS;
    const STASH: Balance = ENDOWMENT / 1000;

    GenesisConfig {
        system: SystemConfig {
            // Add Wasm runtime to storage.
            code: wasm_binary.to_vec(),
            changes_trie_config: Default::default(),
        authority_discovery: AuthorityDiscoveryConfig { keys: vec![] },
        balances: BalancesConfig {
            balances: endowed_accounts.iter().cloned().map(|k| (k, ENDOWMENT)).collect(),
        babe: BabeConfig {
            authorities: vec![],
            epoch_config: Some(BabeEpochConfiguration {
                c: (1, 4),
                allowed_slots: sp_consensus_babe::AllowedSlots::PrimaryAndSecondaryPlainSlots,
        grandpa: GrandpaConfig { authorities: vec![] },
        im_online: ImOnlineConfig { keys: vec![] },
        session: SessionConfig {
            keys: initial_authorities
                .map(|x| {
                        session_keys(x.3.clone(), x.2.clone(), x.4.clone(), x.5.clone()),
        sudo: SudoConfig {
            // Assign network admin rights.
            key: root_key,
        staking: StakingConfig {
            validator_count: 255,
            minimum_validator_count: initial_authorities.len() as u32,
            invulnerables: initial_authorities.iter().map(|x| x.0.clone()).collect(),
            slash_reward_fraction: Perbill::from_percent(10),
        treasury: Default::default(),
        council: CouncilConfig::default(),
        indices: IndicesConfig { indices: vec![] },

My intuition tells me that I am doing something grossly wrong in this chain spec which is causing the runtime to panic. Running a single node works fine. I understand that troubleshooting these errors aren't very realistic online, but I just need a hint on where to look thanks :)

EDIT #1 : N.B: If I try to run this node again, it starts without this exact error, but gives a different in the chain logs:

 Bootnode with peer id `12D3KooWBmAwcd4PJNJvfV89HwE48nwkRmAgo8Vy3uQEyNNHBox2` is on a different chain (our genesis: 0x6cab…1f0c theirs: 0x7196…f402)    

1 Answer 1


The issue might be that initial_authorities accounts don't have any funds, and therefore they don't exist. The error you see comes from the staking module, presumably because it didn't get any valid validators / nominators.

Edit: regarding the issue of different genesis hashes, it is likely the case that these nodes are being run with slightly different binaries. E.g. imagine that you had node A already running, then you tweaked the chainspec a bit, compiled and started running node B. Since you changed the chainspec, this will affect the genesis hash and therefore node A and B won't be able to connect with each other (from their point-of-view they are on different blockchains).

  • Thanks for the comment Andre; please would you care to glance at the EDIT #1 I've made to my question? Also I've made sure that vec![authority_keys_from_seed("Alice"), authority_keys_from_seed("Bob"), authority_keys_from_seed("Charlie"),], get's the endowment which is 10M tokens. I'm passing this vec from development_config to my testnet_genesis, which then gives the funds. Commented Feb 25, 2022 at 15:07
  • Added a comment regarding that.
    – André
    Commented Feb 25, 2022 at 15:53
  • Also, making sure that Alice, Bob, etc. are funded is not enough if they are not validators. If you look at the example spec for substrate node, you can see that validator accounts are also funded to make sure they exist: github.com/paritytech/substrate/blob/master/bin/node/cli/src/…
    – André
    Commented Feb 25, 2022 at 15:56
  • They are running the same binary, with the exact same dev chainspec. Also in my testnet_genesis you can see that I give all endowed accounts with some balance. And endowed_accounts is defined as a vec over all test accounts + their stash accounts. So I don't see the logic of writing the code you linked in that comment. Commented Feb 25, 2022 at 16:10
  • Have you tried starting both nodes from a clean db? I mean there must be something different otherwise they wouldn't have distinct genesis hashes AFAIK. Regarding endowed_accounts funding, my point was about authorities and nominators (i.e. initial_authorities and initial_nominators. From your code I could not tell whether these accounts are the same as the endowed accounts. Judging from your comment it seems that you're setting Alice, Bob and Charlie as authorities (which are funded), if that's the case then I don't know what is the cause of the error.
    – André
    Commented Feb 25, 2022 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.