Grandpa stuck at "Polling"

Question

I am trying to spin up a small BABE PoS network of four nodes and not able to get GRANDPA to start.

It's hard to know what exact pieces of information to provide, here are some notes.

A few notes:

• I am inserting the session keys immediately with the --alice, --bob, etc flags for testing, for now, and I can confirm the expected keys are set via RPC.
• Blocks produce, authored by any validator running,but don't finalize. All the nodes are peered correctly.
• In a previous configuration, I noted at least Alice could get finalization to start GRANDPA not finalizing with 2 of 3 nodes on network. Then I switched to four nodes due to that answer.
• I note every node spits out the following grandpa log message endlessly

DEBUG tokio-runtime-worker grandpa: Casting prevote for round 1    
DEBUG tokio-runtime-worker grandpa: Announcing block 0xb0d7…cad6 to peers which we voted on in round 1 in set 0    
TRACE tokio-runtime-worker grandpa: Round 1: Got incoming message    
TRACE tokio-runtime-worker grandpa: Polling round 1, state = State { prevote_ghost: None, finalized: None, estimate: None, completable: false }, step = Some(Prevoted)    
TRACE tokio-runtime-worker grandpa: Polling round 1, state = State { prevote_ghost: None, finalized: None, estimate: None, completable: false }, step = Some(Prevoted)    
INFO tokio-runtime-worker libp2p_mdns::behaviour: discovered: 12D3KooWBWC4U9DsmbrDHqDBvUbk9o6xbxnSZaDyXpzpPsJm6GVe /ip4/172.18.0.1/tcp/30334    
INFO tokio-runtime-worker libp2p_mdns::behaviour: discovered: 12D3KooWBWC4U9DsmbrDHqDBvUbk9o6xbxnSZaDyXpzpPsJm6GVe /ip4/10.0.0.143/tcp/30334    
INFO tokio-runtime-worker libp2p_mdns::behaviour: discovered: 12D3KooWBWC4U9DsmbrDHqDBvUbk9o6xbxnSZaDyXpzpPsJm6GVe /ip4/172.17.0.1/tcp/30334    
INFO tokio-runtime-worker libp2p_mdns::behaviour: discovered: 12D3KooWBWC4U9DsmbrDHqDBvUbk9o6xbxnSZaDyXpzpPsJm6GVe /ip4/172.23.0.1/tcp/30334    
TRACE tokio-runtime-worker grandpa: Polling round 1, state = State { prevote_ghost: None, finalized: None, estimate: None, completable: false }, step = Some(Prevoted)    
ERROR tokio-runtime-worker sub-libp2p: add_set_reserved_peer with unknown protocol: /b0d742dbbf156179748027452dcd7c57c4eec99183ddab43f8def5c2b8bfcad6/grandpa/1    
TRACE tokio-runtime-worker grandpa: Polling round 1, state = State { prevote_ghost: None, finalized: None, estimate: None, completable: false }, step = Some(Prevoted)    
INFO tokio-runtime-worker libp2p_mdns::behaviour: discovered: 12D3KooWCtSAfr4yMvLZDXvid8vLPHKWyG5JLYi6MUFPj93nXbd2 /ip4/10.0.0.143/tcp/30335    
INFO tokio-runtime-worker libp2p_mdns::behaviour: discovered: 12D3KooWCtSAfr4yMvLZDXvid8vLPHKWyG5JLYi6MUFPj93nXbd2 /ip4/172.23.0.1/tcp/30335    
INFO tokio-runtime-worker libp2p_mdns::behaviour: discovered: 12D3KooWCtSAfr4yMvLZDXvid8vLPHKWyG5JLYi6MUFPj93nXbd2 /ip4/172.18.0.1/tcp/30335    
INFO tokio-runtime-worker libp2p_mdns::behaviour: discovered: 12D3KooWCtSAfr4yMvLZDXvid8vLPHKWyG5JLYi6MUFPj93nXbd2 /ip4/172.17.0.1/tcp/30335    
TRACE tokio-runtime-worker grandpa: Polling round 1, state = State { prevote_ghost: None, finalized: None, estimate: None, completable: false }, step = Some(Prevoted)    
ERROR tokio-runtime-worker sub-libp2p: add_set_reserved_peer with unknown protocol: /b0d742dbbf156179748027452dcd7c57c4eec99183ddab43f8def5c2b8bfcad6/grandpa/1    
TRACE tokio-runtime-worker grandpa: Polling round 1, state = State { prevote_ghost: None, finalized: None, estimate: None, completable: false }, step = Some(Prevoted)    
TRACE tokio-runtime-worker grandpa: Polling round 1, state = State { prevote_ghost: None, finalized: None, estimate: None, completable: false }, step = Some(Prevoted)

Here is an example of the response from the roundState RPC. Any of the nodes report three keys in missing prevotes, and four in missing precommits, though I don't understand what to do with that information.

{
  setId: 0
  best: {
    round: 1
    totalWeight: 4
    thresholdWeight: 3
    prevotes: {
      currentWeight: 1
      missing: [
        0x3544624b6A684E4c707158337A715a644e426339
        0x35454354777636635a356e4a51506b3674576661
        0x354641396e51445667323637444564386d315A79
      ]
    }
    precommits: {
      currentWeight: 0
      missing: [
        0x3544624b6A684E4c707158337A715a644e426339
        0x35454354777636635a356e4a51506b3674576661
        0x354641396e51445667323637444564386d315A79
        0x35476F4e6b66365764627843466E5064416E5959
      ]
    }
  }
  background: []
}

You can see the code here I am starting the nodes like:

 ./target/release/my-node --chain ./dev.json --node-key="3a9a7f43c5e6415106d09334fe698a33eda6c204667524a7127b475630e3953b" --alice --in-peers 256 --validator --ws-external --rpc-external --rpc-cors all --rpc-methods=unsafe --base-path="./data/node1"

 ./target/release/my-node --chain ./dev.json --bob --in-peers 256 --validator --ws-external --rpc-external --rpc-cors all --rpc-methods=unsafe --base-path="./data/node2"

and so on...

Answer 1

The issue was the error: "ERROR tokio-runtime-worker sub-libp2p: add_set_reserved_peer with unknown protocol: /b0d742dbbf156179748027452dcd7c57c4eec99183ddab43f8def5c2b8bfcad6/grandpa/1 " This is resolved by registering the grandpa protocol correctly.

Answer 2

Please try the following:

• It seems like you've tried to created a custom chain definition to synchronise and validate with different authorities. But you've only created a custom chain specification customSpec.json based on code changes in a "dev" chain id that should be specified in src/chain_spec.rs. But you don't have a chain id "dev" since it appears you renamed that chain id to "tread_dev" here, so create your custom chain specification with the correct chain id that exists:

./target/release/transmit-node build-spec --chain tread_dev --disable-default-bootnode > customSpec.json

Also after trying to generate a chain specification you've tried to then build a "raw" chain definition for the new chain from the custom chain specification, but you haven't used --raw, so try using that flag as shown below:

./target/release/transmit-node build-spec --chain customSpec.json --disable-default-bootnode --raw > dev.json

• Have you tried changing the --node-key to 88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee (without the 0x prefix)?
• Why don't you explicitly specify a different --rpc-port for Bob (i.e. 9934) and Charlie (i.e. 9935) and Dave (i.e. 9936) other than the RPC port 9933 that Alice is using?
• Why are you running two Charlie validators (both using --charlie) instead of just one Charlie and one Dave --dave?
• In the code for your chain specification here you mention four authorities Alice, Bob, Charlie, and Dave. You're also using Babe, ImOnline, and Grandpa session keys here. So you need to generate season keys for each of those four authorities like in the example shown below (which only generates them for Alice, so repeat for Bob, and Charlie, and Dave by changing it from Alice to Bob, etc):

subkey inspect --scheme ed25519 "//Alice"
subkey inspect --scheme sr25519 "//Alice"//babe
subkey inspect --scheme sr25519 "//Alice"//imonline
subkey inspect --scheme ed25519 "//Alice"//grandpa

Then after starting all the validator nodes insert each of their session keys (only Alice shown below). Repeat below for Bob, Charlie, and Dave using their respective secret key and unique Public Key (hex) that was shown in the output from the previous step. Also change the respective RPC port of each cURL request to match the explicit different RPC port that you started each authority with so Alice (i.e. 9933), Bob (i.e. 9934), Charlie (i.e. 9935) and Dave (i.e. 9936)

curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["babe", "//Alice//babe", "0x46ffa3a808850b2ad55732e958e781146ed1e6436ffb83290e0cb810aacf5070"],"id":1 }' 127.0.0.1:9933
curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["imon", "//Alice//imonline", "0xee725cf87fa2d6f264f26d7d8b84b1054d2182cdcce51fdea95ec868be9d1e17"],"id":1 }' 127.0.0.1:9933
curl -vH 'Content-Type: application/json' --data '{ "jsonrpc":"2.0", "method":"author_insertKey", "params":["gran", "//Alice//grandpa", "0x6e2de2e5087b56ed2370359574f479d7e5da1973e17ca1b55882c4773f154d2f"],"id":1 }' 127.0.0.1:9933

Then restart all the authorities and verify they're all finalising blocks.

• Have you tried using --unsafe-ws-external and --unsafe-rpc-external too?
• Does unsafe need to use a capital letter like --rpc-methods=Unsafe?
• Why do you need --in-peers 256?
• Have you tried using --execution=native -lruntime=debug?