Encoded error while performing a storage migration

Question

This is my first time migrating data, and I'd like to receive general guidance about what steps and command lines I need to use. I've been working with the node-template before to implement it in my relay chain.

I followed this tutorial Basic Storage Migration from Substrate Docs. But I noticed a better way to perform a storage migration using theframe-executive module with the try-runtime-cli.

Regarding the try-runtime-cli implementation, I struggled a bit following the tutorial from Substrate docs, because the tutorial is based on the polkadot-v0.9.28 release, following those steps, I couldn't compile the node with the try-runtime feature. After some research, I updated it to be compatible with the polkadot-v0.9.37, there was a significant change from the polkadot-v0.9.28 to the polkadot-v0.9.29 release.

I defined a new type to allocate migrations:

pub type Migrations = (
    // pallet_nicks::migration::v1::MigrateToV1<Runtime>,
    // pallet_nicks::migration::v2::MigrateToV2<Runtime>,
);

https://github.com/didiermis/test-storage-migration/blob/ba6af5da8025c4157ee7ef758a0590d88c8c89f6/runtime/src/lib.rs#L351

Then pass it to the executive type:

pub type Executive = frame_executive::Executive<
    Runtime,
    Block,
    frame_system::ChainContext<Runtime>,
    Runtime,
    AllPalletsWithSystem,
    Migrations,
>;

https://github.com/didiermis/test-storage-migration/blob/ba6af5da8025c4157ee7ef758a0590d88c8c89f6/runtime/src/lib.rs#L357

After finishing the Basic Storage Migration tutorial, I tried to add an account status for the Nicks structure:

    #[derive(CloneNoBound, Encode, Decode, Default, TypeInfo, MaxEncodedLen, PartialEqNoBound, RuntimeDebug)]
    #[scale_info(skip_type_params(T))]
    #[codec(mel_bound())]
    pub struct Nickname<T: Config> {
        pub first: BoundedVec<u8, T::MaxLength>,
        pub last: Option<BoundedVec<u8, T::MaxLength>>,
        pub third: AccountStatus
    }

    #[derive(Clone, Encode, Decode, TypeInfo, MaxEncodedLen, PartialEqNoBound, RuntimeDebug)]
    #[scale_info(skip_type_params(T))]
    #[codec(mel_bound())]
    pub enum AccountStatus {
        Active,
        Inactive,
    }

    impl Default for AccountStatus {
        fn default() -> Self {
            AccountStatus::Active
        }
    }

Steps to reproduce the issue:

• git clone [email protected]:didiermis/test-storage-migration.git
• cd test-storage-migration
• git checkout main
• cargo build --release
• ./target/release/node-template --dev
• execute the bash script ./pallets/nicks/src/migrations_old/execute_tx_v2.sh to create some initial accounts (4) (requires polkadot-CLI)

---

Migration part:

• Rename the lib.rs file to lib.v1.rs
• Rename the lib.v2.rs file to lib.rs (lib.v2.rs file contains the changes related to the new account status)
• Rename the migration.v2.rs file to migration.rs
• Uncomment the following line in the runtime/src/lib.rs:

pub type Migrations = (
    // pallet_nicks::migration::v1::MigrateToV1<Runtime>,
    pallet_nicks::migration::v2::MigrateToV2<Runtime>,
);

https://github.com/didiermis/test-storage-migration/blob/ba6af5da8025c4157ee7ef758a0590d88c8c89f6/runtime/src/lib.rs#L353

• Update the spec version (runtime/src/lib.rs):

`spec_version: 101,`

https://github.com/didiermis/test-storage-migration/blob/ba6af5da8025c4157ee7ef758a0590d88c8c89f6/runtime/src/lib.rs#L108

• cargo build --release --features=try-runtime
• RUST_LOG=runtime=trace target/release/node-template try-runtime --detailed-log-output --runtime target/release/wbuild/node-template-runtime/node_template_runtime.compact.compressed.wasm on-runtime-upgrade --checks live --uri ws://localhost:9944

Then I got the following encoded error (detailed error provided in the gist):

2023-03-24 14:21:22.519 ERROR main frame_support::storage: (key, value) failed to decode at [101, 250, 43, 123, 42, 225, 198, 168, 220, 201, 193, 123, 220, 138, 232, 42, 217, 172, 213, 218, 186, 94, 26, 239, 28, 164, 85, 53, 14, 203, 193, 46, 14, 91, 224, 15, 188, 46, 21, 181, 254, 101, 113, 125, 173, 4, 71, 215, 21, 246, 96, 160, 165, 132, 17, 222, 80, 155, 66, 230, 239, 184, 55, 95, 86, 47, 88, 165, 84, 213, 134, 14]: Error    

https://gist.github.com/didiermis/92d2ae0bfea89b1692645317d34b8973

If I comment pre & post upgrade functions, I get the same encoded error but notice the last statement: https://gist.github.com/didiermis/7049e846197f7926abcd576bdc180c8d

2023-03-24 14:32:42.441  INFO main try-runtime::cli: TryRuntime_on_runtime_upgrade executed without errors. Consumed weight = (625000000 ps, 0 byte), total weight = (2000000000000 ps, 18446744073709551615 byte) (0.03 %, 0.00 %). 

Something noticeable is if I run a runtime-upgrade as:

• update spec_version: 101
• cargo build --release -p node-template-runtime
• upload the new binary (target/release/wbuild/node-template-runtime/node_template_runtime.compact.compressed.wasm) using the Polkadot-UI

Migration works as expected!

Before runtime-upgrade:

After runtime-upgrade

Let me know if detailed steps are required. I really appreciate any help you can provide.

Update:

I added the following validation for the pre-upgrade function:

            let names = NameOf::<T>::iter_keys().count() as u32;            
            let decodable_names = NameOf::<T>::iter_values().count() as u32;
            log::info!(
                target: LOG_TARGET,
                "pre_upgrade: {:?} names, {:?} decodable names",
                names,
                decodable_names,
            );
            ensure!(
                names == decodable_names,
                "Not all values are decodable."
            );

I got this log:

2023-03-24 17:53:28.198  INFO main nicks: pre_upgrade: 4 names, 0 decodable names  
2023-03-24 17:53:28.198  INFO main node_template_runtime: try-runtime::on_runtime_upgrade failed with: "Not all values are decodable." 

I've created 4 initial accounts with the script, so the number of keys is correct. It seems to me that there's an issue decoding stored values.

Answer 1

In the pre_upgrade() and in the on_runtime_upgrade() functions there is an issue with the:

    let prev_count = NameOf::<T>::iter().count();

Because you are reading the data from the new storage with the new structure:

    #[pallet::storage]
    pub(super) type NameOf<T: Config> =
        StorageMap<_, Twox64Concat, T::AccountId, (Nickname<T>, BalanceOf<T>)>;

What you have to do is use the previous storage, so in your migration.rs code add:

    mod v1 {
      use super::*;

      #[derive(Decode, Encode)]
      pub struct OldNickname<T: Config> {
        pub first: BoundedVec<u8, T::MaxLength>,
        pub last: Option<BoundedVec<u8, T::MaxLength>>,
      }

      #[cfg(feature = "try-runtime")]
      #[storage_alias]
      pub(super) type NameOf<T: Config> = StorageMap<Pallet<T>, Twox64Concat, <T as frame_system::Config>::AccountId, (OldNickname<T>, BalanceOf<T>)>;
}

And then is your pre_upgrade() and in the on_runtime_upgrade() functions use:

    let prev_count = v1::NameOf::<T>::iter().count();

You can check the migration file in the Democracy FRAME pallet as an example.