We are currently running a 5 validator standalone network, after about ~1000 blocks the peers disconnect and block production stalls, all the nodes continue to produce blocks but does not gossip to each other. We are able to reproduce the issue after multiple retries.
Non validator node logs:
2022-10-11 21:13:15.020 INFO tokio-runtime-worker substrate: 💤 Idle (4 peers), best: #1193 (0x7cde…4646), finalized #1190 (0xe8ad…b51d), ⬇ 0 ⬆ 0
2022-10-11 21:13:20.020 INFO tokio-runtime-worker substrate: 💤 Idle (2 peers), best: #1193 (0x7cde…4646), finalized #1190 (0xe8ad…b51d), ⬇ 0 ⬆ 0
2022-10-11 21:13:20.130 DEBUG tokio-runtime-worker dkg_gadget::gossip_engine::network: Peer 12D3KooWDhesPWmWcwyL5WsPG1eC3isnPqbHJRoBZCHybmMX9eKo disconnected from gossip protocol
2022-10-11 21:13:20.130 DEBUG tokio-runtime-worker dkg_gadget::gossip_engine::network: Peer 12D3KooWDhesPWmWcwyL5WsPG1eC3isnPqbHJRoBZCHybmMX9eKo disconnected from gossip protocol
2022-10-11 21:13:22.124 DEBUG tokio-runtime-worker dkg_gadget::gossip_engine::network: Peer 12D3KooWHXiHVg1YX9PHsa8NrS5ZWEoaKBydqZvLoegw2bcTiCtf disconnected from gossip protocol
2022-10-11 21:13:22.125 DEBUG tokio-runtime-worker dkg_gadget::gossip_engine::network: Peer 12D3KooWHXiHVg1YX9PHsa8NrS5ZWEoaKBydqZvLoegw2bcTiCtf disconnected from gossip protocol
2022-10-11 21:13:25.021 INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #1193 (0x7cde…4646), finalized #1190 (0xe8ad…b51d), ⬇ 0 ⬆ 12 B/s
2022-10-11 21:13:30.022 INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #1193 (0x7cde…4646), finalized #1190 (0xe8ad…b51d), ⬇ 0 ⬆ 37 B/s
2022-10-11 21:13:35.023 INFO tokio-runtime-worker substrate: 💤 Idle (0 peers), best: #1193 (0x7cde…4646), finalized #1190
Validator node logs :
2022-10-11 21:12:31.508 DEBUG tokio-runtime-worker dkg_gadget::gossip_engine::network: Sending message to all peers
2022-10-11 21:12:42.001 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🙌 Starting consensus session on top of parent 0xd6343bee26f09a8ab36b7de40e3c7e16f60e6061b4b3c0bf430c3ae4d0051af6
2022-10-11 21:12:42.011 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 1193 (9 ms) [hash: 0x7789595157ee0d3d31559b6fe90154c471b97eb5d76466824e4f5600801774c3; parent_hash: 0xd634…1af6; extrinsics (11): [0x0ed8…2350, 0xf99e…8844, 0x9da9…5be4, 0x8a26…9e1e, 0x6a0d…3092, 0xa7b1…be54, 0xf20c…3d50, 0xd284…da13, 0x27ac…f326, 0xfd0c…8260, 0x381a…b461]]
2022-10-11 21:12:42.013 INFO tokio-runtime-worker aura: 🔖 Pre-sealed block for proposal at 1193. Hash now 0x43236d6a42ad6ecf7d667aa444e74912825270062947dd4c3c7013d9e9796d67, previously 0x7789595157ee0d3d31559b6fe90154c471b97eb5d76466824e4f5600801774c3.
2022-10-11 21:13:12.001 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🙌 Starting consensus session on top of parent 0x43236d6a42ad6ecf7d667aa444e74912825270062947dd4c3c7013d9e9796d67
2022-10-11 21:13:13.635 WARN tokio-runtime-worker sc_basic_authorship::basic_authorship: Timeout fired waiting for transaction pool at block #1193. Proceeding with production.
2022-10-11 21:13:13.637 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 1194 (1635 ms) [hash: 0x7535a79d22e45e07b89e712382fac195466c2fbe430eb9adcac2c0366a24b29d; parent_hash: 0x4323…6d67; extrinsics (1): [0xea7b…480a]]
2022-10-11 21:13:13.638 INFO tokio-runtime-worker aura: 🔖 Pre-sealed block for proposal at 1194. Hash now 0x1ab409a29c55c41932fc708ecea433cf8ee98b97fabac9aa0cf24566a34825ed, previously 0x7535a79d22e45e07b89e712382fac195466c2fbe430eb9adcac2c0366a24b29d.
2022-10-11 21:13:42.000 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🙌 Starting consensus session on top of parent 0x1ab409a29c55c41932fc708ecea433cf8ee98b97fabac9aa0cf24566a34825ed
2022-10-11 21:13:42.009 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 1195 (1 ms) [hash: 0xc5cbc8d26a78762d5326cb9c200ad8354f5629053de24c1407d5625a6b5d7973; parent_hash: 0x1ab4…25ed; extrinsics (1): [0x0019…7ffb]]
2022-10-11 21:13:42.011 INFO tokio-runtime-worker aura: 🔖 Pre-sealed block for proposal at 1195. Hash now 0x9116b721793b7ab809cfa65c00516e6142511183b70085309d63ac722dbbba85, previously 0xc5cbc8d26a78762d5326cb9c200ad8354f5629053de24c1407d5625a6b5d7973.
2022-10-11 21:14:12.001 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🙌 Starting consensus session on top of parent 0x9116b721793b7ab809cfa65c00516e6142511183b70085309d63ac722dbbba85
2022-10-11 21:14:12.002 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 1196 (0 ms) [hash: 0xd8d86809bf02ceeb8575110817323dfab2b5099a976615b1cb7421fc72ab0dd5; parent_hash: 0x9116…ba85; extrinsics (1): [0x98bf…b349]]
2022-10-11 21:14:12.003 INFO tokio-runtime-worker aura: 🔖 Pre-sealed block for proposal at 1196. Hash now 0xae78d95b47b8e1c9b01187334ef62683c297d5c776d343f40adcabed1eda75f2, previously 0xd8d86809bf02ceeb8575110817323dfab2b5099a976615b1cb7421fc72ab0dd5.
2022-10-11 21:14:42.000 INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🙌 Starting consensus session on top of parent 0xae78d95b47b8e1c9b01187334ef62683c297d5c776d343f40adcabed1eda75f2
What would be the possible reason for this stall? Could a runtime panic cause this issue? using polkadot-v0.9.29
