We upgraded our testnet Shibuya yesterday, for which we have our own validator network, and block production stalled for almost a whole day. The block on which it was stuck was the one where the upgrade was applied (ValidationFunctionApplied). Collators were constantly proposing the same block, but it was never validated.
The block was finally produced after we deployed an additional validator. The idea was to have both a collator & validator on the single machine to help eliminate network issues. Funny enough, this helped but another collator's proposed block was actually included and backed, not the one on the same machine.
We'd like to understand why exactly this happened, and how can we better understand when block production is stalled due to network issues. I've dug into the code but couldn't find what trace would notify us about network issue?
We got this on the validator side, which might be indicative of the error but isn't verbose:
fetch_pov_job err=FetchPoV(NetworkError(Network(Timeout))) para_id=Id(1000) pov_hash=0x2f13ff0d6cc8eb806b95f19b80c915e4d2342c731b424a61bece5d00356fdf4f authority_id=Public(aaf593ee1fc807bcb6f6c3b87f043ae538c17426c2c99ab98951ea6bff561d21 (14sA6MN1...))
On the collator side, I never saw this trace:
debug!(target: logging_target, "Starting authorship at slot: {slot}");
but saw traces preceding that one. Not sure if that helps.
Few more pieces of info about the upgrade:
- the upgrade block was heavy, but even when we reduced tx pool limit to 0 for our collators, the problem persisted, so it could be just
PoVsize related - the same runtime blob was applied to our parachain on Rococo and upgrade went just fine
