Parachain cannot build blocks after winning a slot

Question

We have a slight issue: we won a parachain slot on Saturday and are still waiting to produce blocks this Monday. https://polkadot.subscan.io/block/14512138?tab=event

We should be producing blocks by now since it has been at least an era since we won our slot. Our parathread was not producing blocks with imported #0 whilst we were bidding.

Here are some points to help with some questions:

Genesis state

We have confirmed that the head in the relay chain is the same as the export-genesis-state from the chain spec and that we built our chainspecs using build-spec: https://github.com/t3rn/t3rn/blob/master/specs/t3rn.raw.json and here's the clear version: https://github.com/t3rn/t3rn/blob/master/specs/t3rn.json

head: 0x00000000000000000000000000000000000000000000000000000000000000000061f523881c58116b34662cc35d2e921dd63fbd1132f66c3b4c54aba09c0a9e8d03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c11131400 which seems to be the same length as others.

Greyed out in parachain info

We noticed we are also greyed out on the parachains tab. Is this relevant?

Do invulnerable collators need balance?

We set our invulnerable collators with zero balance, do they need a balance to author blocks?

Minimum session candidates

We have also set out minimum session candidates to two, which seems sane since we have two collators.

Block hash at genesis 0

Our block hash for genesis is set to 0x4545454545454545454545454545454545454545454545454545454545454545, which we assume will be enriched after the block is imported.

There doesn't seem to be any logs regarding this at trace level.

Useful links

Here's our node endpoint: https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fws.t3rn.io#/chainstate

And our runtime is open source, here: https://github.com/t3rn/t3rn/tree/master/runtime/t3rn-parachain/src

Do you have any idea what issue we may be facing? We are currently stumped since it doesn't seem like we are doing anything wrong. This configuration is also roughly the same config as our t0rn deployment on rococo.

Answer 1

So we found the issue, thanks to the checklist here - Thanks a bunch @bruno.

We had previously checked whether author.hasKey worked, and we overlooked it since it didn't return true for us on t0rn.

Actually, it turned out to be a slight mismatch. Once we had rubber ducked through the list, we were inserting via RPC but not inserting the key into the node via key insert, and we were also calling author.rotateKeys in our deployment script, which added to the mix. Once we issued key insert we started building blocks.