On February 20, 2023 around 22:08 UTC, Shiden block production has been hurt and almost stopped during 40 minutes.A lot of collators were missing blocks for no apparent reason.
Examples of blocks that have been missed by several collators:
Location and hardware specs of the collators are not the source of the error, we checked the logs from different collators with correct specs / location.
Generally on the relay side, a lot of these messages are issued.
[Relaychain] Missing block weight for new head. Skipping chain. hash=0x942d17da9ac8ce3634542da34757d4a66cfed6d27e79aed7d0857828f433caf0
The parachain was build with a previous version of relay, we have now upgraded to latest and monitoring for the next time it may happen.
Timeline of errors
22:09 UTC - On 1st round of blocks missed by our collators 1 and 2
Collator 1
Noticeable messages
2023-02-20 22:09:28 [Parachain] Failed to create inherent data providers. error="Failed to create parachain inherent"
2023-02-20 22:09:28 [Parachain] Cannot obtain the hrmp ingress channel. relay_parent=0x4c7d1661f220b49c3c7015810cc01ce2d336ee555c9191f74e58a234092768f1 error=BlockchainError(UnknownBlock("State already discarded for 0x4c7d1661f220b49c3c7015810cc01ce2d336ee555c9191f74e58a234092768f1"))
Collator 2 (next one in Aura selection) got no message and didn't even try to enter block production process, but at the exact time of if supposed to produce the block:
2023-02-20 22:09:36 [Relaychain] Missing block weight for new head. Skipping chain. hash=0x942d17da9ac8ce3634542da34757d4a66cfed6d27e79aed7d0857828f433caf0
22:20 UTC - another round of blocks missed
Collator 1
Collator 2
