We upgraded our testnet Shibuya yesterday, for which we have our own validator network, and block production stalled for almost a whole day. The block on which it was stuck was the one where the upgrade was applied (ValidationFunctionApplied). Collators were constantly proposing the same block, but it was never validated.

The block was finally produced after we deployed an additional validator. The idea was to have both a collator & validator on the single machine to help eliminate network issues. Funny enough, this helped but another collator's proposed block was actually included and backed, not the one on the same machine.

We'd like to understand why exactly this happened, and how can we better understand when block production is stalled due to network issues. I've dug into the code but couldn't find what trace would notify us about network issue?

We got this on the validator side, which might be indicative of the error but isn't verbose:

fetch_pov_job err=FetchPoV(NetworkError(Network(Timeout))) para_id=Id(1000) pov_hash=0x2f13ff0d6cc8eb806b95f19b80c915e4d2342c731b424a61bece5d00356fdf4f authority_id=Public(aaf593ee1fc807bcb6f6c3b87f043ae538c17426c2c99ab98951ea6bff561d21 (14sA6MN1...))

On the collator side, I never saw this trace:

debug!(target: logging_target, "Starting authorship at slot: {slot}");

but saw traces preceding that one. Not sure if that helps.

Few more pieces of info about the upgrade:

  • the upgrade block was heavy, but even when we reduced tx pool limit to 0 for our collators, the problem persisted, so it could be just PoV size related
  • the same runtime blob was applied to our parachain on Rococo and upgrade went just fine
  • 1
    Just adding a detail to the Rococo upgrade: block has been produced by a collator hosted on GCP, as we know Rococo validators are also hosted on GCP. Leveraging GCP private network is useful for high connectivity but kills decentralization possibilities. It seems surprising to be in this situation for a PoV size of 1.7 Mb only.
    – bLd
    Commented Jul 31, 2023 at 11:22
  • will this tutorial helps to setup rococo testnet and produce blocks?
    – go11li
    Commented Nov 21, 2023 at 5:13


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.