5

Assume we have a StorageMap MyMap of which we want to extend the type of the decoded keys from KeyTypeOld to an enum including the old type as well as a new one.

Before

pub type MyMap = StorageMap<_, Blake2_128Concat, KeyTypeOld, V>;

After

pub enum MyEnum<A, B> {
    Old(A),
    New(B),
}
pub type MyMap = StorageMap<_, Blake2_128Concat, MyEnum<KeyTypeOld, KeyTypeAddition>, V>;

  1. Is it possible to migrate from the old map to the new one without keeping the definition of the old one in the WASM blob, e.g., only in-memory in the migration? This would keep the API of this storage entry consistent.

Unfortunately, my attempt (see below) fails to decode the keys when iterating over the in-memory created MyMapOld, at least when testing with the try-runtime feature.

mod migration {
    use crate::MyMap as MyMapNew;

    #[storage_alias]
    type MyMap = StorageMap<_, Blake2_128Concat, KeyTypeOld, V>;
    #[storage_alias]
    type MyMapTmp = StorageMap<_, Blake2_128Concat, MyEnum<KeyTypeOld, KeyTypeAddition>, V>;

    pub struct MyMigration(PhantomData<P>);
    impl<P: crate::pallet::Config> OnRuntimeUpgrade for MyMigration<P>{
        fn on_runtime_upgrade() -> frame_support::weights::Weight {
            // Should not write to same storage item during drain 
            // Thus, write to temporary storage and move at the end
            MyMap::<P>::drain().for_each(|(key_old, value)| {
                let key_new = MyEnum::<KeyTypeOld, KeyTypeAddition>::Old(key_old);
                MyMapTmp::<R>::insert(key_new, value);
            });

            // Move items from MyMapTmp to MyMapNew similar to 
            // frame_support::storage::migration::move_storage_from_pallet
            ...
    }

}

The only solution I have found so far is to keep the old definition of MyMap and have a co-existing MyMapNew. However, that would break the API for services which I would like to avoid.

// inside pallet macro
#[pallet::storage]
pub type MyMap = StorageMap<_, Blake2_128Concat, KeyTypeOld, V>;
#[pallet::storage]
pub type MyMapNew = StorageMap<_, Blake2_128Concat, MyEnum<KeyTypeOld, KeyTypeAddition>, V>;

  1. Moreover, is my assumption correct that we can we expect services, which periodically connect to my chain and query MyMap storage via Polkadot-JS, to be able to automatically understand the correct decoding by the provided post-upgrade metadata?
3
  • What do you need MyMapTmp for? Why not directly insert into crate::MyMap? Otherwise you insert into the same map that you drain. Jun 14, 2022 at 12:44
  • Ah sorry, MyMapTmp is supposed to have the same keytype as MyMapNew. I thought the temporary map is required because else I would write into the same map I am draining: I thought that the in-memory old map does have the same hex prefix as the new one defined in the pallet since the only diff between both is the keytype. Is that assumption wrong? Jun 14, 2022 at 13:17
  • I updated the keytype of MyMapTmp to mirror MyMapNew. Jun 14, 2022 at 13:18

1 Answer 1

7

First of all, there's a misunderstanding here. Defining something like

#[pallet::storage]
pub type MyMap = StorageMap<_, Blake2_128Concat, KeyTypeOld, V>;
#[pallet::storage]
pub type MyMapNew = StorageMap<_, Blake2_128Concat, MyEnum<KeyTypeOld, KeyTypeAddition>, V>;

if not used, has no overhead. A [#pallet::storage] generates a some alias types for you to work with storage, but having the definition around does not actually write anything to any storage. To compare, it is almost the same as

#[storage_alias]
type MyMap = StorageMap<_, Blake2_128Concat, KeyTypeOld, V>;

which you define in mod migration.

As for your migration:

  1. As noted, you don't need MyMapTmp.
  2. Your usage of #[storage_alias] seems to be incorrect. You are basically telling it that _ (when hashed) is the prefix that it should find the map, which is certainly wrong. You either have to pass the correct one, like
#[storage_alias]
type MyMap = StorageMap<NameOfThePalletInYourRuntime, Blake2_128Concat, KeyTypeOld, V>;

Or make it generic over <P: Config> and let it be derived via PalletInfoAccess. See here for an example.


Moreover, is my assumption correct that we can we expect services, which periodically connect to my chain and query MyMap storage via Polkadot-JS, to be able to automatically understand the correct decoding by the provided post-upgrade metadata?

This entirely depends on that service, and how it is implemented. The existential purpose of having a dynamic metadata that changes per-rutnime-upgrade is to enable offchain services to do that. Whether they do it or not is beyond the scope of substrate.

4
  • 1
    Thanks a lot! Providing the pallet name via PalletInfoAccess instead of underscore _ solved my issue. Do you have an answer two my second question by any chance? Jun 16, 2022 at 7:16
  • On a note: The temporary storage is actually required because else we write into the same storage map from which we drain. In my case, the drain iteration accessed almost twice as many keys as existent. Writing to the temp storage and then moving this after the drain did not come with any weird side effects. Jun 16, 2022 at 8:30
  • answered the second question as well.
    – kianenigma
    Jun 16, 2022 at 11:02
  • 1
    As for the temporary storage, I understand your caution and don't want to tell you that I am 100% sure, but I am still almost sure that you won't need it. Once you are draining, if you write the new value to the same key that is being drained right now, then it should be fine, because the internals of that iteration of drain was the last time that the old value was needed. Nonetheless, I suggest verifying this first. And, if two storage items works, then you should do that for extra safety.
    – kianenigma
    Jun 16, 2022 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.