3

This is my first time migrating data, and I'd like to receive general guidance about what steps and command lines I need to use. I've been working with the node-template before to implement it in my relay chain.

I followed this tutorial Basic Storage Migration from Substrate Docs. But I noticed a better way to perform a storage migration using theframe-executive module with the try-runtime-cli.

Regarding the try-runtime-cli implementation, I struggled a bit following the tutorial from Substrate docs, because the tutorial is based on the polkadot-v0.9.28 release, following those steps, I couldn't compile the node with the try-runtime feature. After some research, I updated it to be compatible with the polkadot-v0.9.37, there was a significant change from the polkadot-v0.9.28 to the polkadot-v0.9.29 release.

I defined a new type to allocate migrations:

pub type Migrations = (
    // pallet_nicks::migration::v1::MigrateToV1<Runtime>,
    // pallet_nicks::migration::v2::MigrateToV2<Runtime>,
);

https://github.com/didiermis/test-storage-migration/blob/ba6af5da8025c4157ee7ef758a0590d88c8c89f6/runtime/src/lib.rs#L351

Then pass it to the executive type:

pub type Executive = frame_executive::Executive<
    Runtime,
    Block,
    frame_system::ChainContext<Runtime>,
    Runtime,
    AllPalletsWithSystem,
    Migrations,
>;

https://github.com/didiermis/test-storage-migration/blob/ba6af5da8025c4157ee7ef758a0590d88c8c89f6/runtime/src/lib.rs#L357

After finishing the Basic Storage Migration tutorial, I tried to add an account status for the Nicks structure:

    #[derive(CloneNoBound, Encode, Decode, Default, TypeInfo, MaxEncodedLen, PartialEqNoBound, RuntimeDebug)]
    #[scale_info(skip_type_params(T))]
    #[codec(mel_bound())]
    pub struct Nickname<T: Config> {
        pub first: BoundedVec<u8, T::MaxLength>,
        pub last: Option<BoundedVec<u8, T::MaxLength>>,
        pub third: AccountStatus
    }

    #[derive(Clone, Encode, Decode, TypeInfo, MaxEncodedLen, PartialEqNoBound, RuntimeDebug)]
    #[scale_info(skip_type_params(T))]
    #[codec(mel_bound())]
    pub enum AccountStatus {
        Active,
        Inactive,
    }

    impl Default for AccountStatus {
        fn default() -> Self {
            AccountStatus::Active
        }
    }

Steps to reproduce the issue:

  • git clone [email protected]:didiermis/test-storage-migration.git
  • cd test-storage-migration
  • git checkout main
  • cargo build --release
  • ./target/release/node-template --dev
  • execute the bash script ./pallets/nicks/src/migrations_old/execute_tx_v2.sh to create some initial accounts (4) (requires polkadot-CLI)

Migration part:

  • Rename the lib.rs file to lib.v1.rs
  • Rename the lib.v2.rs file to lib.rs (lib.v2.rs file contains the changes related to the new account status)
  • Rename the migration.v2.rs file to migration.rs
  • Uncomment the following line in the runtime/src/lib.rs:
pub type Migrations = (
    // pallet_nicks::migration::v1::MigrateToV1<Runtime>,
    pallet_nicks::migration::v2::MigrateToV2<Runtime>,
);

https://github.com/didiermis/test-storage-migration/blob/ba6af5da8025c4157ee7ef758a0590d88c8c89f6/runtime/src/lib.rs#L353

  • Update the spec version (runtime/src/lib.rs):
`spec_version: 101,`

https://github.com/didiermis/test-storage-migration/blob/ba6af5da8025c4157ee7ef758a0590d88c8c89f6/runtime/src/lib.rs#L108

  • cargo build --release --features=try-runtime
  • RUST_LOG=runtime=trace target/release/node-template try-runtime --detailed-log-output --runtime target/release/wbuild/node-template-runtime/node_template_runtime.compact.compressed.wasm on-runtime-upgrade --checks live --uri ws://localhost:9944

Then I got the following encoded error (detailed error provided in the gist):

2023-03-24 14:21:22.519 ERROR main frame_support::storage: (key, value) failed to decode at [101, 250, 43, 123, 42, 225, 198, 168, 220, 201, 193, 123, 220, 138, 232, 42, 217, 172, 213, 218, 186, 94, 26, 239, 28, 164, 85, 53, 14, 203, 193, 46, 14, 91, 224, 15, 188, 46, 21, 181, 254, 101, 113, 125, 173, 4, 71, 215, 21, 246, 96, 160, 165, 132, 17, 222, 80, 155, 66, 230, 239, 184, 55, 95, 86, 47, 88, 165, 84, 213, 134, 14]: Error    

https://gist.github.com/didiermis/92d2ae0bfea89b1692645317d34b8973

If I comment pre & post upgrade functions, I get the same encoded error but notice the last statement: https://gist.github.com/didiermis/7049e846197f7926abcd576bdc180c8d

2023-03-24 14:32:42.441  INFO main try-runtime::cli: TryRuntime_on_runtime_upgrade executed without errors. Consumed weight = (625000000 ps, 0 byte), total weight = (2000000000000 ps, 18446744073709551615 byte) (0.03 %, 0.00 %). 

Something noticeable is if I run a runtime-upgrade as:

  • update spec_version: 101
  • cargo build --release -p node-template-runtime
  • upload the new binary (target/release/wbuild/node-template-runtime/node_template_runtime.compact.compressed.wasm) using the Polkadot-UI

Migration works as expected!

Before runtime-upgrade: enter image description here

After runtime-upgrade enter image description here

Let me know if detailed steps are required. I really appreciate any help you can provide.

Update:

I added the following validation for the pre-upgrade function:

            let names = NameOf::<T>::iter_keys().count() as u32;            
            let decodable_names = NameOf::<T>::iter_values().count() as u32;
            log::info!(
                target: LOG_TARGET,
                "pre_upgrade: {:?} names, {:?} decodable names",
                names,
                decodable_names,
            );
            ensure!(
                names == decodable_names,
                "Not all values are decodable."
            );

I got this log:

2023-03-24 17:53:28.198  INFO main nicks: pre_upgrade: 4 names, 0 decodable names  
2023-03-24 17:53:28.198  INFO main node_template_runtime: try-runtime::on_runtime_upgrade failed with: "Not all values are decodable." 

I've created 4 initial accounts with the script, so the number of keys is correct. It seems to me that there's an issue decoding stored values.

5
  • 1
    Works fine when I do it locally, no decoding error. Maybe try to delete your DB snapshot, sometimes it does not clear. Your "failed to decode" probably comes from the translate call, I would avoid that at all cost since it very error prone. There is a post here about the try-runtime CLI but it looks like you got it right. You can also try to test it first. Mar 25 at 13:05
  • Would you mind explaining how to delete the DB? I'm using the --dev flag. I tried with the following command ./target/release/node-template purge-chain --dev, but nothing changed, I still get the same encoded error.
    – Didier M
    Mar 28 at 16:33
  • A friend tried to reproduce the issue, but he got no error. We thought maybe the error was related to the rust version, so I updated it to the latest version, but then I faced this issue reported here: substrate.stackexchange.com/questions/7714/… I downgraded it to version 1.68.0 and I was able to run a local node again, but no luck at this point, same encoded error.
    – Didier M
    Mar 28 at 16:35
  • 2
    @OliverTale-Yazdi Sorry, I uploaded the wrong lib.rs file. There was no difference between the two lib.rs I provided above, that's why you didn't get any error. I've already updated the repository as well as the steps to reproduce the issue. Thank you so much for your attention and participation.
    – Didier M
    Mar 29 at 0:13
  • 1
    Great way to ask a question, very clear and easy to replicate the problem locally.
    – Alex Bean
    Apr 3 at 10:36

1 Answer 1

3

In the pre_upgrade() and in the on_runtime_upgrade() functions there is an issue with the:

    let prev_count = NameOf::<T>::iter().count();

Because you are reading the data from the new storage with the new structure:

    #[pallet::storage]
    pub(super) type NameOf<T: Config> =
        StorageMap<_, Twox64Concat, T::AccountId, (Nickname<T>, BalanceOf<T>)>;

What you have to do is use the previous storage, so in your migration.rs code add:

    mod v1 {
      use super::*;

      #[derive(Decode, Encode)]
      pub struct OldNickname<T: Config> {
        pub first: BoundedVec<u8, T::MaxLength>,
        pub last: Option<BoundedVec<u8, T::MaxLength>>,
      }

      #[cfg(feature = "try-runtime")]
      #[storage_alias]
      pub(super) type NameOf<T: Config> = StorageMap<Pallet<T>, Twox64Concat, <T as frame_system::Config>::AccountId, (OldNickname<T>, BalanceOf<T>)>;
}

And then is your pre_upgrade() and in the on_runtime_upgrade() functions use:

    let prev_count = v1::NameOf::<T>::iter().count();

You can check the migration file in the Democracy FRAME pallet as an example.

2
  • 1
    Thanks Alex, your guidance helped me to solve the encoded error. Now I understand better how to use the [storage_alias] macro. If I want to test a storage migration for a relay chain, Should I only change the -- uri?, for example, wss://test.cat.systems -> RUST_LOG=runtime=trace target/release/node- template try-runtime --detailed-log-output --runtime target/release/wbuild/node-template-runtime/node_template_runtime.compact.compressed.wasm on-runtime-upgrade --checks live --uri wss://test.cat.systems
    – Didier M
    Apr 4 at 2:27
  • 1
    Yes! And include the flag —checks=pre-and-post to perform pre- and post-upgrade checks.
    – Alex Bean
    Apr 4 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.