After restarting a chain I get a ClientImport("Unexpect epoch change") error. What does this mean in English?

Question

So while testing I have about 4 nodes (local) running and if I were to stop them all, and then come back a few minutes later and restart them, with the exact same --base-path, I'd get this error :

WARN tokio-runtime-worker babe: Error with block built on 0xe8e8..a5ff26f911: ClientImport("Unexpected epoch change")    

My question is what does this error really indicate, and when does this occur? Is this error common in babe based chains (or is my chain weird)? Does it imply that if all nodes in a substrate blockchain were to go offline, it couldn't be restarted?

Answer 1

This error occurs when you use BABE block production, and stop running the network for an extended period of time.

Basically, BABE expects that a new block is formed at least once per epoch, which is measured using the computer clock.

When you turn off the whole network for let's say an hour, and then you start the network again, the next block producer will try to produce a block with the current computer clock time, but this will be far in the future for when the next block is expected by BABE.

This is not something that is currently recoverable, and thus the best solution is to restart your network, and ensure that this kind of offline-ness does not occur.

If you do expect to go offline a lot, you should use AURA.

Answer 2

what does this error really indicate

It means that an unexpected epoch change transition occurred that BABE found and rejected when it was validating the epoch change signal transitions of a block it was trying to import, and indicates that your blockchain is bricked and unable to finalize further blocks.

"One invariant of BABE is that you have to have at least one block every epoch_duration slots" - Robert Habermeier

when does this occur?

It happens when you are using the BABE consensus mechanism and your validator set goes offline for more than one epoch. For example in this may occur if many validators ran out of server storage space at the same time.

It is caused in the import_block block-import handler function of the BABE consensus section of the Substrate source code here and documented here

Is this error common in BABE based chains (or is my chain weird)?

Yes, this error is common in chains that are using BABE. Here are a few examples:

• https://github.com/paritytech/substrate/issues/4464
• https://github.com/paritytech/subport/issues/218
• https://github.com/DataHighway-DHX/node/issues/249

Does it imply that if all nodes in a substrate blockchain were to go offline, it couldn't be restarted?

It may be restarting using the "hard spoon", "time warp", or "purge chain" approach, as documented here and answered here.

If you choose the "hard spoon" approach then it may be advisable to switch to using Aura for block production instead of Babe since it does not experience this same issue of becoming bricked when the validator set goes offline for more than one epoch, and it also does not cause delays in block production. Different ways to get a snapshot of the on-chain storage state of an existing blockchain, such as all account balances that are to be included in the genesis of the new blockchain is described here.