Para header not found after moving to another relay chain

Question

In one of our testnets, we moved the parachain (:= P) to another relay chain (:= R). Unfortunately, we have not been able to produce blocks after moving over. I think to have overseen/misinterpreted a minor detail and thus hope to find help here.

Error

The collators keep throwing Failed to collect collation info. error=Application(UnknownBlock("Header was not found in the database: 0x7ccd4c501e544478053ad1e7e0ff3e9e13f4637e3f8913ac29657a3871ff4c02")) for changing blocks, e.g.

2022-12-20 16:25:20 [Parachain] 💤 Idle (1 peers), best: #2140029 (0xfa9a…8baa), finalized #2140029 (0xfa9a…8baa), ⬇ 0 ⬆ 0    
2022-12-20 16:25:24 [Relaychain] ✨ Imported #16120 (0xb03c…c873)    
2022-12-20 16:25:24 [Parachain] Starting collation. relay_parent=0xb03c653b53212450429f0b52e87a325e8b3d5f4f4c03395e6d3c644bfc1ec873 at=0xfa9a2a1a3f88770aeda78c1364fbddd329ec8d0afb973d8db2205e1d7c638baa
2022-12-20 16:25:24 [Parachain] 🙌 Starting consensus session on top of parent 0xfa9a2a1a3f88770aeda78c1364fbddd329ec8d0afb973d8db2205e1d7c638baa    
2022-12-20 16:25:24 [Parachain] 🎁 Prepared block for proposing at 2140030 (0 ms) [hash: 0x8833dee16a124b43efb7d8663ba0eee9b46b5a898c59e13cbff82b84098b522c; parent_hash: 0xfa9a…8baa; extrinsics (2): [0x3d40…323c, 0x0db2…c6fd]]    
2022-12-20 16:25:24 [Parachain] 🔖 Pre-sealed block for proposal at 2140030. Hash now 0x7ccd4c501e544478053ad1e7e0ff3e9e13f4637e3f8913ac29657a3871ff4c02, previously 0x8833dee16a124b43efb7d8663ba0eee9b46b5a898c59e13cbff82b84098b522c.    
2022-12-20 16:25:24 [Parachain] Block import error: State Database error: Too many sibling blocks inserted    
2022-12-20 16:25:24 [Parachain] Error with block built on 0xfa9a2a1a3f88770aeda78c1364fbddd329ec8d0afb973d8db2205e1d7c638baa: Import failed: State Database error: Too many sibling blocks inserted    
2022-12-20 16:25:24 [Parachain] PoV size { header: 0.181640625kb, extrinsics: 2.8251953125kb, storage_proof: 14.001953125kb }
2022-12-20 16:25:24 [Parachain] Compressed PoV size: 14.734375kb
2022-12-20 16:25:24 [Parachain] Failed to collect collation info. error=Application(UnknownBlock("Header was not found in the database: 0x7ccd4c501e544478053ad1e7e0ff3e9e13f4637e3f8913ac29657a3871ff4c02"))

How to reproduce/steps taken on our side

1. Stop P while running on old relay.
2. On R, register P with header of last finalized block (= 2140029) of P and current WASM.
3. Restart Collators of P pointing to new relay chain.
4. Collators fail to build blocks, since R has a lower block number than old relay.
5. In runtime of P: Replace RelayNumberStrictlyIncreases with AnyRelayNumber (commit).
6. In chainspec of P: Use codeSubstitutes: {"2140029": "..."} for last finalized block of P to overwrite WASM with the one resulting from Step 5.
7. On R: Force new para WASM via sudo.para.forceSetCurrentCode.
8. Unfortunately, Collators of P still fail to produce a block with the above error message.

Questions

1. Is it correct to substitute the WASM for the last finalized block or should I pick an earlier one or the next one to be finalized?
2. I did not bump the spec version for the WASM used in codeSubstitutes due to this comment. Is this truly correct or does it not matter at all?

I would be super grateful about any hint where to dig. Thanks in advance!

Versions

• Relay: Runtime westend/9330, client Polkadot v0.9.33
• Para: Runtime and Client based on Substrate v0.9.29

Answer 1

Is it correct to substitute the WASM for the last finalized block or should I pick an earlier one or the next one to be finalized?

Using the last finalized block is correct. You could not take a block that is before the last finalized block, because otherwise the node would reject to import blocks that are before/on the same height as the last finalized block.

I did not bump the spec version for the WASM used in codeSubstitutes due to this comment. Is this truly correct or does it not matter at all?

Yes this is also correct.

Now to your problem ;)

Block import error: State Database error: Too many sibling blocks inserted

This here being your actual problem. The collator had build to many blocks on the same height (more than 32), making it fail to import its newly build block. So, it could still build the block, but not import it. When it then tried to collect the collation information from this block, it failed to find the block (as it failed to be imported).

In the future this "too many siblings blocks" error should not happen anymore, as we implemented a way to prevent this from happening.

Answer 2

You want to migrate your data data from one chain to another, there are two ways according to me for this task.

1. You can migrate your state as centrifuge has done when they migrated from a solochain to parachain you can refer to https://github.com/centrifuge/centrifuge-cli/tree/main/packages/plugins/migration
2. Another way to do this is by populating your state from one chain to another for this you can refer How to selectively populate state from a snapshot of one chain onto another?