We are seeing some strange behaviour with some (not all) of our validator nodes. We are using the standard BABE / GRANDPA pallets.
When it is their turn to produce a block, they produce the block, but immediately after their peer count drops and it looks like none of the validator peers ever receives the block they've produced.
The next block producer then produces a block at the same height and the original validator / block producer has their block reorg'ed out of the chain.
An example log file from one of the validators is below. The other validators never receive block 0x8cb6. Any ideas or suggestions on how to debug or what might be causing the error would be much appreciated!
Apr 25 07:16:50 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:16:50 💤 Idle (37 peers), best: #2556897 (0x04de…82bb), finalized #2556895 (0xcf6d…6a77), ⬇ 123.2kiB/s ⬆ 151.5kiB/s
Apr 25 07:16:54 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:16:54 🙌 Starting consensus session on top of parent 0x04de4479769fb895cdaac629985ba1161ca1e3f326798973009ebd11579082bb
Apr 25 07:16:54 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:16:54 🎁 Prepared block for proposing at 2556898 [hash: 0x20769becac0a66900a5414ba276f82495f1c096a2af3037c5e5d86ef11ddfd5e; parent_hash: 0x04de…82bb; extrinsics (1): [0x3098…7607]]
Apr 25 07:16:54 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:16:54 🔖 Pre-sealed block for proposal at 2556898. Hash now 0x8cb6aab5906ce62841dc8c18bd3fd1ceb856eb967524987937a37cdc520590a8, previously 0x20769becac0a66900a5414ba276f82495f1c096a2af3037c5e5d86ef11ddfd5e.
Apr 25 07:16:54 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:16:54 ✨ Imported #2556898 (0x8cb6…90a8)
Apr 25 07:16:55 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:16:55 💤 Idle (37 peers), best: #2556898 (0x8cb6…90a8), finalized #2556895 (0xcf6d…6a77), ⬇ 100.5kiB/s ⬆ 132.0kiB/s
Apr 25 07:17:00 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:17:00 💤 Idle (37 peers), best: #2556898 (0x8cb6…90a8), finalized #2556895 (0xcf6d…6a77), ⬇ 20.1kiB/s ⬆ 64.7kiB/s
Apr 25 07:17:00 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:17:00 ✨ Imported #2556898 (0xc087…2639)
Apr 25 07:17:05 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:17:05 💤 Idle (9 peers), best: #2556898 (0x8cb6…90a8), finalized #2556895 (0xcf6d…6a77), ⬇ 10.9kiB/s ⬆ 57.6kiB/s
Apr 25 07:17:06 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:17:06 ♻️ Reorg on #2556898,0x8cb6…90a8 to #2556899,0xf011…2102, common ancestor #2556897,0x04de…82bb
Apr 25 07:17:06 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:17:06 ✨ Imported #2556899 (0xf011…2102)
Apr 25 07:17:10 ip-10-20-1-156.ap-southeast-2.compute.internal polymesh-4.1.1-linux-amd64[12043]: 2022-04-25 07:17:10 💤 Idle (12 peers), best: #2556899 (0xf011…2102), finalized #2556897 (0x04de…82bb), ⬇ 14.0kiB/s ⬆ 54.2kiB/s
