3

We are observing the following panic for a parachain node that is warp synced fully on both sides when we try to restart the node:

   Thread 'main' panicked at 'route from finalized to leaf should be available; qed: UnknownBlock("Header was not found in the database: 0x1e70664aec1b2dcbec1ac26210ddb6d7388700e33604edbfa3a55141b22c3837")', /root/.cargo/git/checkouts/cumulus-59522f43471fa161/f603a61/client/consensus/common/src/level_monitor.rs:123

This is pointing to this line in the cumulus project. My own reading from the code is that a recent header is missing simply because it wasn't yet flushed to the permanent db. But that to me is an expected situation rather than a panic. For two reasons:

  1. It happened at least two times consistently after we restarted a node which is sort of not accidental.
  2. I don't know of a safe way to restart a node but we do need nodes to be resilient to restarts and so on.

My questions are:

  1. Is it expected?
  2. Is it related to warp sync flag?
  3. How can we avoid it?

1 Answer 1

1

No this wasn't expected. This was/is a bug.

The problem here is that when a node is warp syncing, it imports a header around the tip of the chain. However, in the case of the parachain this header is not imported as finalized block. The level monitor will then see this header as a leaf and will try to go back down to the last finalized block (genesis block). However, the node hasn't yet synced all the blocks between genesis and the leaf and thus, it will fail to fetch this block from the db.

Here is a fix.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.