How to use try-runtime when upgrading a chain/parachain to latest substrate release?

Question

What is the best practice regarding try-runtime, what errors and warnings should one look out for in the output.

When upgrading a chain it is good practice to try to execute storage migration before upgrade Substrate docs describes how to enable try-runtime and this answer describes how it works

So now we have a try-runtime step in CI, but what output from try-runtime requires actions from the engineer preparing the upgrade? Is the only potential outcome that one of the upstream dependencies is about to run a costly migration on chain upgrade? Are there other problems which this could reveal?

Answer 1

Bruno gave a good overview of the practical part; I will focus a bit on the theoretical part of your questions.

So now we have a try-runtime step in CI, but what output from try-runtime requires actions from the engineer preparing the upgrade?

The try-runtime CLI is designed to error out when there is a problem. Errors like "Undecodable storage" can come from migrations themselves, but are fine, as long as it does not abort.

Is the only potential outcome that one of the upstream dependencies is about to run a costly migration on chain upgrade?

The outcome of running try-runtime can be: There is a migration missing, wrong or in the wrong order. Additionally with the try-state checks it also runs some consistency checks in all pallets that support it. Ideally when all pallets would expose this, then it should be possible to really ensure that nothing is missing.

Are there other problems which this could reveal?

There are some additional commands being worked on like follow-chain or fast-forward, that will be able to ensure that not only the migrations worked, but the chain is also in a usable state afterwards. Some problems don't manifest immediately but take some blocks, which is currently not detected by the CLI.

Answer 2

How to use try runtime in when upgrading a chain/parachin to latest substrate release?

If there are any upstream migrations, you should use try-runtime on-runtime-upgrade subcommand to test that the storage migration is successful before you apply the changes in a production environment.

What is the best practice regarding try-runtime, what errors and warnings should one look out for in the output.

In regards to on-runtime-upgrade you want to make sure that the storage migration was applied correctly on your storage by using pre- and post-upgrade checks. When running try-runtime on-runtime-upgrade you also want to confirm that the runtime spec version was bumped. In regards to a pallet storage migration you may need to bump the storage_version for your pallet.

When running try-runtime on-runtime-upgrade, ideally you have an RPC node dedicated for this with flags set to enable large request/response size so that you can test off of live chain state:

./node-try-runtime \                                                                    
    try-runtime \
    --runtime runtime-try-runtime.wasm \
    -lruntime=debug \
    on-runtime-upgrade \
    live --uri ws://try-runtime-prod-chain-rpc-node:9921

All your migrations should pass with green. Look at the output of the try-runtime logs, it will tell you. You may also want to look at the consumed weight for your migrations, wildly off consumed weight may indicate a flaw in your storage migration's logic. Never take the upstream migrations blindly, always review the code and see if it fits your chain.

A good pipeline is to test with try-runtime-upgrade, then test on a live test network for some time by enacting the upgrade there, then if all is good --> production.

Ideally, you have a try-runtime CI workflow set with your repo for testing releases as well as try-runtime bot that you can kick off manually.

The on-runtime-upgrade subcommand also has an optional try-state hook which can be used to check the pallet's state (sanity check)

Of course, this is only the icing on the cake, try-runtime include a few other subcommands which can be useful for testing such as:

  execute-block       Executes the given block against some state
  offchain-worker     Executes *the offchain worker hooks* of a given block against some state
  follow-chain        Follow the given chain's finalized blocks and apply all of its extrinsics

For example, say you're testing a complex storage migration that your team wrote and it touches several aspects of your pallets storage, you may want to on-runtime-upgrade to ensure your storage migration executed as expected and for added peace-of-mind you may also want to test that the chain continues to produce blocks as expected using follow-chain.

try-runtime can also be used for testing other scenarios, not only storage migrations. For example, perhaps your team discovered a bug in your code that happened in production at block X. Your team then writes code to fix the bug and they could use the try-runtime execute-block subcommand to execute that code against block X to make sure the bug has been fixed before rolling out the fix to production.

Resource

• https://paritytech.github.io/substrate/master/try_runtime_cli

Related

• Do I need to write migration script if upstream pallet's storage has been changed?