Why are finalized blocks zero?

Question

I've created a local live staging network with 1 bootnode and 3 validators. They all run in separate VMs.

After block #52 I saw Backing off claiming new slot for block authorship: finality is lagging. and block times got higher and higher.

2022-06-05 22:28:48 🙌 Starting consensus session on top of parent 0x64ac48b3695c3c02f2d55619e4411e1eaa4f9b5a930e989859fd309052d5b64c    
2022-06-05 22:28:48 🎁 Prepared block for proposing at 49 (0 ms) [hash: 0xa82832ce9f009ec9e4c35651bf2da7b2b9de0ff70c3096abf78356f5f76f781b; parent_hash: 0x64ac…b64c; extrinsics (1): [0xf93c…d576]]    
2022-06-05 22:28:48 🔖 Pre-sealed block for proposal at 49. Hash now 0x2847d747ad23a609aaf0215c8256e4b120618d587fcd542b98258e8b121f5e7c, previously 0xa82832ce9f009ec9e4c35651bf2da7b2b9de0ff70c3096abf78356f5f76f781b.    
2022-06-05 22:28:48 ✨ Imported #49 (0x2847…5e7c)    
2022-06-05 22:28:52 💤 Idle (2 peers), best: #49 (0x2847…5e7c), finalized #0 (0x69ab…327a), ⬇ 80 B/s ⬆ 0.3kiB/s    
2022-06-05 22:28:54 🙌 Starting consensus session on top of parent 0x2847d747ad23a609aaf0215c8256e4b120618d587fcd542b98258e8b121f5e7c    
2022-06-05 22:28:54 🎁 Prepared block for proposing at 50 (1 ms) [hash: 0x9c2c820ac61ec988f009d5bd6d6142db07b4ec7f331ced941d7339df1e96e445; parent_hash: 0x2847…5e7c; extrinsics (1): [0x3fbc…f968]]    
2022-06-05 22:28:54 🔖 Pre-sealed block for proposal at 50. Hash now 0xa9318da838d6709c21102680b92b672ef3631157fc9c2e25dcef8f727c835307, previously 0x9c2c820ac61ec988f009d5bd6d6142db07b4ec7f331ced941d7339df1e96e445.    
2022-06-05 22:28:54 ✨ Imported #50 (0xa931…5307)    
2022-06-05 22:28:57 💤 Idle (2 peers), best: #50 (0xa931…5307), finalized #0 (0x69ab…327a), ⬇ 80 B/s ⬆ 0.3kiB/s    
2022-06-05 22:29:00 ✨ Imported #51 (0x2ced…0ed1)    
2022-06-05 22:29:02 💤 Idle (2 peers), best: #51 (0x2ced…0ed1), finalized #0 (0x69ab…327a), ⬇ 0.3kiB/s ⬆ 0.2kiB/s    
2022-06-05 22:29:06 ✨ Imported #52 (0xf9b2…43e6)    
2022-06-05 22:29:07 💤 Idle (2 peers), best: #52 (0xf9b2…43e6), finalized #0 (0x69ab…327a), ⬇ 0.3kiB/s ⬆ 86 B/s    
2022-06-05 22:29:12 Backing off claiming new slot for block authorship: finality is lagging.  

Steps I've followed

1. Start the bootnode

./target/release/realm3 \
--node-key 6c7285131dbf7367e10e30d960c57177ca43844cefe7a49d80a63916bebce9d7 \
--base-path /tmp/bootnode1 \
--chain node/specs/customSpecRaw.json \
--name bootnode1

2. Start the validators

./target/release/realm3 --base-path /tmp/validator1 \
--chain node/specs/customSpecRaw.json \
--ws-port 9922 \
--port 30333 \
--rpc-port 9934 \
--validator \
--rpc-methods Unsafe \
--ws-external \
--rpc-external \
--rpc-cors all \
--name "Realm3 Validator 1" \
--bootnodes /ip4/10.0.0.22/tcp/30333/p2p/12D3KooWCurm4ieJvkiVPPMzVNXzSMkceRfbTVJQUvhzJHkFTF2t

3. Insert the keys

./target/release/realm3 key insert --base-path /tmp/validator1 \
--chain node/specs/customSpecRaw.json \
--scheme Sr25519 \
--suri <suri> \
--password-interactive \
--key-type babe

./target/release/realm3 key insert --base-path /tmp/validator1 \
--chain node/specs/customSpecRaw.json \
--scheme Ed25519 \
--suri <suri> \
--password-interactive \
--key-type gran

./target/release/realm3 key insert --base-path /tmp/validator1 \
--chain node/specs/customSpecRaw.json \
--scheme Sr25519 \
--suri <suri> \
--password-interactive \
--key-type imon

I've followed steps #2 and #3 for each validator. I inserted the keys after the node is started.

chainSpec.json

PS: I'm running the 2 nodes with different DNS than the other 2. The bootnode and validator node that has the same DNS with it have 3 peers. However, the other 2 nodes have 2 peers.

Answer 1

For grandpa the keys need to exist before starting grandpa. As you have inserted the keys after you have started the node, grandpa isn't picking them up. So, restarting the node will make finality work.