Our testnet parachain is stuck exactly at the block in which the runtime upgrade is authored. It was authorized and enacted via parachainSystem as recommended. The upgrade neither includes any migrations nor larger code changes, the previous upgrade was just as large (~3.6mb) and included many migrations without any issue.
Since we have full control over the relay chain, we tried unstucking it by forcing new parachain code on the relay via sudo.paras.forceSetCurrentCode. Unfortunately, that did not have any effect.
I would really appreciate any insights about this issue as I was under the impression that a relaychain can ALWAYS unblock a parachain. So there needs to be a solution, I could not find it though unfortunately. Maybe we can manually force discarding the block for collators?
Of course, we have checked this answer regarding a stuck parachain block production:
- Possible issue #1 (trouble sending the block): The PoV size is quite large with ~3.6mb. However, validators and collators are physically close and the network connection between these should not be an issue AFAICT.
- Possible issue #2 (trouble verifying the block in time): This should not be an issue as block proposal preparation is 10 to 20ms
- Possible issue #3 (Invalid parachain code in the relay): Since both above ones should not apply, we tried forcing parachain code via relay. As stated above, this did not yield any effects.
The two log lines which stand out
WARN tokio-runtime-worker parachain::collator-protocol: [Relaychain] distribute collation message parent is outside of our view relay_parent=0x5edb8867f6ac41d67e3e545926379b8db05d9d388f9c08ce4c52e341b645d7a3
INFO tokio-runtime-worker aura: [Parachain] ⌛️ Discarding proposal for slot 141223015; block production took too long
Based on this response about the "block production took too long", we stopped all parachain nodes and then brought back a single collator hoping the transaction would leave the tx pool due to hitting mortality limit. That did not happen unfortunately. First we kept the nodes offline for ~10 minutes, then ~30 minutes, then hours. The issue persists after 12 hours.
Testnet setup
- 2 Relay nodes with full control
- Relay spec based on
rococo-local - 3 Collators with full control
