Migrate keytype of StorageMap without breakin API

Question

Assume we have a StorageMap MyMap of which we want to extend the type of the decoded keys from KeyTypeOld to an enum including the old type as well as a new one.

Before

pub type MyMap = StorageMap<_, Blake2_128Concat, KeyTypeOld, V>;

After

pub enum MyEnum<A, B> {
    Old(A),
    New(B),
}
pub type MyMap = StorageMap<_, Blake2_128Concat, MyEnum<KeyTypeOld, KeyTypeAddition>, V>;

1. Is it possible to migrate from the old map to the new one without keeping the definition of the old one in the WASM blob, e.g., only in-memory in the migration? This would keep the API of this storage entry consistent.

Unfortunately, my attempt (see below) fails to decode the keys when iterating over the in-memory created MyMapOld, at least when testing with the try-runtime feature.

mod migration {
    use crate::MyMap as MyMapNew;

    #[storage_alias]
    type MyMap = StorageMap<_, Blake2_128Concat, KeyTypeOld, V>;
    #[storage_alias]
    type MyMapTmp = StorageMap<_, Blake2_128Concat, MyEnum<KeyTypeOld, KeyTypeAddition>, V>;

    pub struct MyMigration(PhantomData<P>);
    impl<P: crate::pallet::Config> OnRuntimeUpgrade for MyMigration<P>{
        fn on_runtime_upgrade() -> frame_support::weights::Weight {
            // Should not write to same storage item during drain 
            // Thus, write to temporary storage and move at the end
            MyMap::<P>::drain().for_each(|(key_old, value)| {
                let key_new = MyEnum::<KeyTypeOld, KeyTypeAddition>::Old(key_old);
                MyMapTmp::<R>::insert(key_new, value);
            });

            // Move items from MyMapTmp to MyMapNew similar to 
            // frame_support::storage::migration::move_storage_from_pallet
            ...
    }

}

The only solution I have found so far is to keep the old definition of MyMap and have a co-existing MyMapNew. However, that would break the API for services which I would like to avoid.

// inside pallet macro
#[pallet::storage]
pub type MyMap = StorageMap<_, Blake2_128Concat, KeyTypeOld, V>;
#[pallet::storage]
pub type MyMapNew = StorageMap<_, Blake2_128Concat, MyEnum<KeyTypeOld, KeyTypeAddition>, V>;

2. Moreover, is my assumption correct that we can we expect services, which periodically connect to my chain and query MyMap storage via Polkadot-JS, to be able to automatically understand the correct decoding by the provided post-upgrade metadata?

Answer 1

First of all, there's a misunderstanding here. Defining something like

#[pallet::storage]
pub type MyMap = StorageMap<_, Blake2_128Concat, KeyTypeOld, V>;
#[pallet::storage]
pub type MyMapNew = StorageMap<_, Blake2_128Concat, MyEnum<KeyTypeOld, KeyTypeAddition>, V>;

if not used, has no overhead. A [#pallet::storage] generates a some alias types for you to work with storage, but having the definition around does not actually write anything to any storage. To compare, it is almost the same as

#[storage_alias]
type MyMap = StorageMap<_, Blake2_128Concat, KeyTypeOld, V>;

which you define in mod migration.

As for your migration:

1. As noted, you don't need MyMapTmp.
2. Your usage of #[storage_alias] seems to be incorrect. You are basically telling it that _ (when hashed) is the prefix that it should find the map, which is certainly wrong. You either have to pass the correct one, like

#[storage_alias]
type MyMap = StorageMap<NameOfThePalletInYourRuntime, Blake2_128Concat, KeyTypeOld, V>;

Or make it generic over <P: Config> and let it be derived via PalletInfoAccess. See here for an example.

---

Moreover, is my assumption correct that we can we expect services, which periodically connect to my chain and query MyMap storage via Polkadot-JS, to be able to automatically understand the correct decoding by the provided post-upgrade metadata?

This entirely depends on that service, and how it is implemented. The existential purpose of having a dynamic metadata that changes per-rutnime-upgrade is to enable offchain services to do that. Whether they do it or not is beyond the scope of substrate.