Is it possible to use the chain specification file or some other mechanism to configure a Substrate network with injected block history? In this case, the motivation is CI/CD/testing, but I can see how this could also be useful for network migrations.
How to configure a Substrate Network with Injected Block History from a snapshot or database backup?
-
Are you using the Substrate or Polkadot CLI?– Oliver Tale-YazdiFeb 23, 2022 at 17:13
-
This question is about Substrate.– Dan ForbesFeb 23, 2022 at 17:14
Add a comment
|
4 Answers
You should be able to do this either by persisting the whole database and always using that as a starting point, or using the export / import functionality supported by Substrate. In order to export the historical block data you could do:
substrate export-blocks --chain my-chain -d my-chain-db blocks.bin
You could then persist this blob somewhere and share it with your test environment. You can then re-import the block data (into a blank database) as the first step before starting your tests:
substrate import-blocks --chain my-chain -d my-chain-db blocks.bin
When using BABE this isn't possible / pratical due to the liveness requirements of the protocol (i.e. if the chain is down for more than one epoch then it gets bricked and you get the dreaded "unexpected epoch change" error).
-
Amazing, thank you! So, to clarify, the above is not possible for a network that is configured to use BABE? Feb 23, 2022 at 17:28
-
Short answer is no, it would not be usable with BABE. That said, I plan to add a flag
--unsafe-babe-epoch-changes(or something similar) that will allow overcoming the "unexpected epoch change" issue (there's some caveats but I think it would be usable in testing environments). If the main thing being tested is the runtime I would still suggest to use Aura as it will always be easier to manage than BABE (and for the purpose of testing the runtime the block production protocol shouldn't make a difference).– AndréFeb 23, 2022 at 17:38 -
-
Am I right in thinking you can run
import-blockswhile the node is running? I have substrate running in a docker container so it would be nice if I could simply issue this command to the container and have it continue to run, rather than restarting the container.– forgetsoMar 22, 2022 at 16:26 -
1No, you can't do that, any process needs to have exclusive access to the database.– AndréMar 22, 2022 at 16:28
You can build up a local chain with the --dev flag and then import it in your CI.
There are several ways to restore snapshots, depending on your needs.
- For Polkadot/Kusama you can download a chain snapshot from Polkachu. This would be a real chain snapshot, so probably to heavy for testing.
- Import some blocks with the
import-blockscommand that were exported with theexport-blockscommand before. This works fine for real chain specs, but--devcreate a new genesis and therfore the import fails. - Use
export-stateand pass that the resulting file to--chain. This will put all the chain state into the genesis block. The new chain will have the same state as the old one, but start at block zero. - Just copy paste the
~/.local/share/polkadot/chains/directory from some backup. Works for real specs. For--devspecs you need to copy the directory you specify with--base-path.
answered Feb 23, 2022 at 17:30
-
1If you go with the
export-stateapproach, you will probably want to patch the genesis config to remove some storage (e.g. change the root account, remove the session history, etc). You can do that with a simple js tool we have created: github.com/Phala-Network/phala-blockchain/blob/master/scripts/… Mar 23, 2022 at 6:36
You can also use fork-off-substrate script to generate chainspec containing storage of any running chain - even your live network.
Once great resource and reference on generation of snapshots is the polkadot-snapshot-generator tool, that is hosted on https://polkashots.io/ for Polkadot and Kusama (and some other networks/parachains may as well.
When running a node, --base-path <my/base/path> will generate the following structure with the db directory containing the
<my/base/path>/chains ==> <the chains set from chainspec> ==> db
|
==> keystore
|
==> network
You want to copy only the db section here, assuming that you do not want to share the keystore with your priv keys (potentially sensitive!) and network with your network libp2p info and peering priv keys.