Parachain blocks stopped finalizing after adding another collator

Question

I had setup the relay chain and parachain (collator-1) in a local testnet environment on different servers and blocks was finalising for both. But after 1 month I tried adding another collator(collator-2) on same relaychain but its not finalising blocks and also the previous collator(collator-1) stopped finalising the blocks and showing an error "Collation wasn't advertised to any validator"

collator-1 error logs-

 2022-06-01 17:18:36 [Parachain] PoV size { header: 0.220703125kb, extrinsics: 2.5576171875kb, storage_proof: 9.318359375kb }
 2022-06-01 17:18:36 [Parachain] Compressed PoV size: 11.1484375kb
 2022-06-01 17:18:36 [Parachain] Produced proof-of-validity candidate. block_hash=0x11aec83abc1246d85d64a05733f478fa26999beb971a2b6675a57c414c8945fe
 2022-06-01 17:18:36 [Relaychain] 💤 Idle (4 peers), best: #580555 (0x4903…0aa0), finalized #580552 (0x6b19…196d), ⬇ 2.2kiB/s ⬆ 2.2kiB/s
 2022-06-01 17:18:36 [Parachain] 💤 Idle (0 peers), best: #274796 (0x494f…61e7), finalized #274794 (0xd062…e40c), ⬇ 65 B/s ⬆ 86 B/s
 2022-06-01 17:18:37 Accepted a new tcp connection from 127.0.0.1:45496.
 2022-06-01 17:18:37 Accepted a new tcp connection from 127.0.0.1:45506.
 2022-06-01 17:18:41 Accepted a new tcp connection from 127.0.0.1:47060.
 2022-06-01 17:18:41 [Relaychain] 💤 Idle (4 peers), best: #580555 (0x4903…0aa0), finalized #580553 (0xac92…2a37), ⬇ 1.4kiB/s ⬆ 1.3kiB/s
 2022-06-01 17:18:41 [Parachain] 💤 Idle (0 peers), best: #274796 (0x494f…61e7), finalized #274795 (0x2544…788e), ⬇ 0 ⬆ 0
 2022-06-01 17:18:42 [Relaychain] ✨ Imported #580556 (0xb739…d8ff)
 2022-06-01 17:18:42 [Parachain] Starting collation. relay_parent=0xb739e12c0e91e6516b90a73929c765f2c66e67aed239c420cfdffe7013b8d8ff at=0x494f4f6b232e8221e3aa56a0f44592eba04a5249ff058f5ae85872010b8861e7
 2022-06-01 17:18:42 [Relaychain] ♻️  Reorg on #580556,0xb739…d8ff to #580556,0x8fba…d379, common ancestor #580555,0x4903…0aa0
 2022-06-01 17:18:42 [Relaychain] ✨ Imported #580556 (0x8fba…d379)
 2022-06-01 17:18:42 [Parachain] Starting collation. relay_parent=0x8fbae1f1bb8694d22492b9fa2b14c9d2b698800152d1aa556dc2e63ef499d379 at=0x494f4f6b232e8221e3aa56a0f44592eba04a5249ff058f5ae85872010b8861e7
 2022-06-01 17:18:46 [Relaychain] 💤 Idle (4 peers), best: #580556 (0x8fba…d379), finalized #580554 (0xe7ed…2446), ⬇ 2.7kiB/s ⬆ 2.6kiB/s
 2022-06-01 17:18:46 [Parachain] 💤 Idle (0 peers), best: #274796 (0x494f…61e7), finalized #274795 (0x2544…788e), ⬇ 0 ⬆ 0
 2022-06-01 17:18:48 [Relaychain] ✨ Imported #580557 (0x28a7…4722)
 2022-06-01 17:18:48 [Parachain] Starting collation. relay_parent=0x28a75a8ec01b2d1c0c3c5217565490a476ff9b4f890e5f62c163cb153e3a4722 at=0x494f4f6b232e8221e3aa56a0f44592eba04a5249ff058f5ae85872010b8861e7
 2022-06-01 17:18:48 [Parachain] 🙌 Starting consensus session on top of parent 0x494f4f6b232e8221e3aa56a0f44592eba04a5249ff058f5ae85872010b8861e7
 2022-06-01 17:18:48 [Parachain] on_timestamp_set called
 2022-06-01 17:18:48 [Parachain] 🎁 Prepared block for proposing at 274797 (1 ms) [hash: 0x0947a20046b254b258429c165b7d216d0b33bb0294e3b7f52634384a25c681a9; parent_hash: 0x494f…61e7; extrinsics (2): [0x2636…0777, 0x430c…2fc5]]
 2022-06-01 17:18:48 [Parachain] 🔖 Pre-sealed block for proposal at 274797. Hash now 0xc41c88a611b28e289b5b816e32040c5324a3e8072a4537d8b173674655aa57cc, previously 0x0947a20046b254b258429c165b7d216d0b33bb0294e3b7f52634384a25c681a9.
 2022-06-01 17:18:48 [Parachain] ✨ Imported #274797 (0xc41c…57cc)
 2022-06-01 17:18:48 [Parachain] PoV size { header: 0.220703125kb, extrinsics: 2.5576171875kb, storage_proof: 9.318359375kb }
 2022-06-01 17:18:48 [Parachain] Compressed PoV size: 11.1484375kb
 2022-06-01 17:18:48 [Parachain] Produced proof-of-validity candidate. block_hash=0xc41c88a611b28e289b5b816e32040c5324a3e8072a4537d8b173674655aa57cc
 2022-06-01 17:18:51 [Relaychain] 💤 Idle (4 peers), best: #580557 (0x28a7…4722), finalized #580554 (0xe7ed…2446), ⬇ 1.3kiB/s ⬆ 1.5kiB/s
 2022-06-01 17:18:51 [Parachain] 💤 Idle (0 peers), best: #274796 (0x494f…61e7), finalized #274795 (0x2544…788e), ⬇ 0 ⬆ 0
 2022-06-01 17:18:52 [Relaychain] 👴 Applying authority set change scheduled at block #580555
 2022-06-01 17:18:52 [Relaychain] 👴 Applying GRANDPA set change to new set [(Public(88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee (5FA9nQDV...)), 1), (Public(d17c2d7823ebf260fd138f2d7e27d114c0145d968b5ff5006125f2414fadae69 (5GoNkf6W...)), 1)]
 2022-06-01 17:18:54 [Relaychain] ✨ Imported #580558 (0xd49f…8314)
 2022-06-01 17:18:54 [Relaychain] Collation wasn't advertised to any validator. candidate_hash=0x5edc61513bdcfb7b82d2e891252362bd16b54d6ee55e5e88c7404da0000a736c pov_hash=0xa0e6b151f64d8ef00997c806c5bffedb0773ef5be0e576bd0601193ebfca6425

Steps I had done to add both the collators-

Collator-1 Steps-

1.Generated plain spec and updated ParaID

./node build-spec --disable-default-bootnode --chain test > test-parachain-plain.json

2.Generated raw chain spec

./node build-spec --chain test-parachain-plain.json --raw --disable-default-bootnode > test-parachain-2000-raw.json

3.Obtained Wasm runtime

./node export-genesis-wasm --chain test-parachain-2000-raw.json > para-2000-wasm

4.Generated genesis state

./node export-genesis-state --chain test-parachain-2000-raw.json > para-2000-genesis

5.Added keys to the keystore

./node key insert --base-path /data/parachain/test --chain test-parachain-2000-raw.json --scheme Sr25519 --suri <1st-account-secret-seed-key> --key-type aura

6.Started the collator node

./node --collator --force-authoring --chain test-parachain-2000-raw.json --base-path /data/parachain/test --name "Test-Collator-1" --pruning archive --port 40343  --ws-external --ws-port 9955 --rpc-external --rpc-port 9943 --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" --rpc-methods=unsafe --rpc-cors all -- --execution wasm --chain rococo-local.json --port 30343 --ws-port 9977 --bootnodes /ip4/<relaychain-IP>/tcp/30333/p2p/<local-node-identity>

Collator-2 Steps-

1.Copied raw spec file from collator-1 node to collator-2 node

2.Added keys to the keystore

./node key insert --base-path /data/parachain/test --chain test-parachain-2000-raw.json --scheme Sr25519 --suri <2nd-account-secret-seed-key> --key-type aura

3.Started the collator node

./node --collator --force-authoring --chain test-parachain-2000-raw.json --base-path /data/parachain/test --name "Test-Collator-2" --pruning archive --port 40344  --ws-external --ws-port 9956 --rpc-external --rpc-port 9944 --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" --rpc-methods=unsafe --rpc-cors all -- --execution wasm --chain rococo-local.json --port 30344 --ws-port 9978 --bootnodes /ip4/<relaychain-IP>/tcp/30333/p2p/<local-node-identity>

Does somebody have any idea how to resolve this issue? Is collator-1 will start finalising blocks from where it stopped? If yes how can I restore back please

Answer 1

To add new collators to an already running chain you will need to use something like collator-selection pallet. This will help you when adding collators with keys not present on the chainspec of your chain.

Answer 2

You have to set the session keys using the RPC call author.insertKey so the collator can start producing blocks, see this step-by-step explanation: How to properly set session keys for Relay and Parachain with custom keys.