10

Is it possible to use the chain specification file or some other mechanism to configure a Substrate network with injected block history? In this case, the motivation is CI/CD/testing, but I can see how this could also be useful for network migrations.

Improve this question
2
  • Are you using the Substrate or Polkadot CLI?
    – Oliver Tale-Yazdi
    Feb 23, 2022 at 17:13
  • This question is about Substrate.
    – Dan Forbes
    Feb 23, 2022 at 17:14

4 Answers 4

Reset to default
10

You should be able to do this either by persisting the whole database and always using that as a starting point, or using the export / import functionality supported by Substrate. In order to export the historical block data you could do:

substrate export-blocks --chain my-chain -d my-chain-db blocks.bin

You could then persist this blob somewhere and share it with your test environment. You can then re-import the block data (into a blank database) as the first step before starting your tests:

substrate import-blocks --chain my-chain -d my-chain-db blocks.bin

When using BABE this isn't possible / pratical due to the liveness requirements of the protocol (i.e. if the chain is down for more than one epoch then it gets bricked and you get the dreaded "unexpected epoch change" error).

Improve this answer
6
  • Amazing, thank you! So, to clarify, the above is not possible for a network that is configured to use BABE?
    – Dan Forbes
    Feb 23, 2022 at 17:28
  • Short answer is no, it would not be usable with BABE. That said, I plan to add a flag --unsafe-babe-epoch-changes (or something similar) that will allow overcoming the "unexpected epoch change" issue (there's some caveats but I think it would be usable in testing environments). If the main thing being tested is the runtime I would still suggest to use Aura as it will always be easier to manage than BABE (and for the purpose of testing the runtime the block production protocol shouldn't make a difference).
    – André
    Feb 23, 2022 at 17:38
  • I am testing the client ;-)
    – Dan Forbes
    Feb 24, 2022 at 14:22
  • Am I right in thinking you can run import-blocks while the node is running? I have substrate running in a docker container so it would be nice if I could simply issue this command to the container and have it continue to run, rather than restarting the container.
    – forgetso
    Mar 22, 2022 at 16:26
  • 1
    No, you can't do that, any process needs to have exclusive access to the database.
    – André
    Mar 22, 2022 at 16:28
8

You can build up a local chain with the --dev flag and then import it in your CI.
There are several ways to restore snapshots, depending on your needs.

  1. For Polkadot/Kusama you can download a chain snapshot from Polkachu. This would be a real chain snapshot, so probably to heavy for testing.
  2. Import some blocks with the import-blocks command that were exported with the export-blocks command before. This works fine for real chain specs, but --dev create a new genesis and therfore the import fails.
  3. Use export-state and pass that the resulting file to --chain. This will put all the chain state into the genesis block. The new chain will have the same state as the old one, but start at block zero.
  4. Just copy paste the ~/.local/share/polkadot/chains/ directory from some backup. Works for real specs. For --dev specs you need to copy the directory you specify with --base-path.
Improve this answer
1
1

You can also use fork-off-substrate script to generate chainspec containing storage of any running chain - even your live network.

Improve this answer
0

Once great resource and reference on generation of snapshots is the polkadot-snapshot-generator tool, that is hosted on https://polkashots.io/ for Polkadot and Kusama (and some other networks/parachains may as well.

When running a node, --base-path <my/base/path> will generate the following structure with the db directory containing the

<my/base/path>/chains ==> <the chains set from chainspec> ==> db
                                                          |
                                                          ==> keystore
                                                          |
                                                          ==> network

You want to copy only the db section here, assuming that you do not want to share the keystore with your priv keys (potentially sensitive!) and network with your network libp2p info and peering priv keys.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.