access storage map from another pallet without trait pallet Config

Question

I have one pallet with a storagemap that I want to use/call with another Pallet.

After doing some research I ran over Centrifuge fudge project(https://github.com/centrifuge/fudge/blob/376e4d6b89324b24d131e7f7c19c7c504871d5d3/core/src/builder/relay_chain.rs#L62-L70) that has a similar approach.

I tried to do this using a StorageInstance:

pub mod llmmod {
    use super::*;
    use frame_support::{
        storage::types::{StorageMap, StorageValue, ValueQuery},
        traits::StorageInstance,
        Blake2_128Concat, Twox64Concat,
    };
    // ParaLifecyclesPrefix, based on centrifuge
    pub struct LLMPoliticsCopy;

    impl StorageInstance for LLMPoliticsCopy {
        fn pallet_prefix() -> &'static str {
            "llm"
        }

        const STORAGE_PREFIX: &'static str = "LLMPolitics";
    }
    pub type LLMPolitics<T> = frame_support::storage::types::StorageMap<
        LLMPoliticsCopy,
        Blake2_128Concat,
        <T as frame_system::Config>::AccountId,
        u128,
        frame_support::pallet_prelude::ValueQuery,
    >;

    pub fn check_pooled_llm<T: frame_system::Config>(sender: T::AccountId) -> bool {
        LLMPolitics::<T>::contains_key::<T::AccountId>(sender)
    }

}


Then I check the storage map with a regular contains, to check if the account is in the storage map:

ensure!(!llmmod::check_pooled_llm::<T>(who.clone()), Error::<T>::NoPolLLM);

The code is compiling, but the storage map is not syncing, when I insert a new entry using the pallet, it does not sync.

Is there a way to Query a storage map in another pallet without adding new_pallet::Config to the config trait ? Is there a way to get around this?

I need the democracy pallet to query a storage map from another pallet but adding + LLM_Pallet::Config in the trait config for the pallet just introduce a lot of bugs.

Is there a way to query storage maps that other pallets have without the pallet's config? manually adding a trait for it?

from pallet:

    #[pallet::storage] // allocated in politics
    #[pallet::getter(fn get_politics_balance)]
    pub(super) type LLMPolitics<T: Config> =
        StorageMap<_, Blake2_128Concat, T::AccountId, T::Balance, ValueQuery>;

Link to the code repo:
https://github.com/liberland/liberland_substrate/tree/storage https://github.com/liberland/liberland_substrate/blob/storage/frame/democracy/src/lib.rs https://github.com/liberland/liberland_substrate/blob/storage/frame/llm/src/lib.rs

Answer 1

I wanted to provide some overview, of what I think are the three possible solutions (if one does NOT want to tight couple - i.e. trait Config: other_pallet::Config the pallets).

1. Create keys manually
2. Create StorageMap manually
3. Abstraction via trait

Generally, I would suggest always favoring tight coupling or soft coupling (solution 3.) over creating keys or maps manually. The manual solutions are error-prone and can lead to pallets accessing different storage locations and/or assuming different types in the storage, which in the end can lead to nasty bugs.

Setup

• pallet_provider
  • Provides:

    #[pallet::storage]
    pub type DataStorage<T: Config> =
       StorageMap<_, Blake2_128Concat, T::Index, T::Data>;
    

  • Stores T::Data under the key T::Index
• pallet_consumer
  • wants to access DataStorage
• lib.rs of our runtime

  construct_runtime! {
     pub enum Runtime where
         Block = Block,
         NodeBlock = Block,
         UncheckedExtrinsic = UncheckedExtrinsic
     {
         System: system::{Pallet},
         Provider: pallet_provider::{Pallet, Storage},
         Consumer: pallet_consumer::{Pallet}
     }
  }
  

Create keys manually

Substrate's storage can be seen as a key-value-DB and the current storage types (StorageValue, StorageMap and StorageDoubleMap) just are convenience abstractions over this key-value logic. But in the end, they are always creating a deterministic key for accessing a specific value.

There is a really good deep dive into how the storage actually works from Shawn Tabrizi here.

That being said one can access a pallet's storage elements by manually creating said keys. All pallet's storage elements are created after the following logic:

• StorageValue-Key: twox_128("PALLET_NAME") + twox_128("STORAGE_NAME")
• StorageMap-Key: twox_128("PALLET_NAME") + twox_128("STORAGE_NAME") + DEV_CHOOSEN_HASHER("KEY_1")
• StorageDoubleMap-Key: twox_128("PALLET_NAME") + twox_128("STORAGE_NAME") + DEV_CHOOSEN_HASHER("KEY_1") + DEV_CHOOSEN_HASHER("KEY_2")

NOTE: The PALLET_NAME is the name defined for the pallet in the construct_runtime!() macro.

For our above defined setup we could retrieve the T::Data from storage in the pallet_consumer as follows:

const PALLET_NAME: &'static str = "Provider";
const STORAGE_NAME: &'static str = "DataStorage";
type Index = u32;
type Data = u32;

fn get_storage_value(key: Index) -> Option<Data> {
    let pallet_hash = sp_io::hashing::twox_128(PALLET_NAME.as_bytes());
    let storage_hash = sp_io::hashing::twox_128(STORAGE_NAME.as_bytes());
    // Hashing the scale-encoded key
    let key_hashed = 
        frame_support::hash::Blake2_128Concat::hash(key.encode());

    let mut final_key = Vec::new();
    final_key.extend_from_slice(&pallet_hash);
    final_key.extend_from_slice(&storage_hash);
    final_key.extend_from_slice(&key_hashed);
    
    frame_support::storage::unhashed::get::<Data>(&final_key)
}

NOTE: We have no way of checking whether our defined Data and Index types here, match with the T::Data and T::Index of the pallet_provider storage.

Create StorageMaps manually

This solution uses the normal interface of the StorageMaps abstraction over storage. Meaning, one can use the created type just like any other normal storage element in substrate. BUT again, we manually define the names (pallet and storage) from which the final is key is generated.

Here, we would create a copy of the pub type Data<T: Config> in the pallet_consumer, but without binding it to the config. Hence, relying again on manually knowing the key type T::Index and the data type T::Data.

type Index = u32;
type Data = u32;

pub struct DataStorageCopyPrefix;
impl StorageInstance for DataStorageCopyPrefix {
    fn pallet_prefix() -> &'static str {
        "Provider"
    }

    const STORAGE_PREFIX: &'static str = "Data";
}

pub type DataStorageCopy<T> = frame_support::storage::types::StorageMap<
        DataStorageCopyPrefix,
        Blake2_128Concat,
        Index,
        Data,
    >;

NOTE: We have no way of checking whether our defined Data and Index types here, match with the T::Data and T::Index of the pallet_provider storage.

Abstraction via trait

This solution does not rely on knowledge of the underlying StorageMap in any way. Which, in my opinion is a bonus, as the pallet_provider is free to change the StorageMap without the need for the pallet_consumer to be changed.

• Define a trait that "abstracts" what the pallet_consumer wants

  trait DataProvider {
     type Key;
     type Data;
  
     fn query(key: Self::Key) -> Option<Self::Data>;
  

• Implement DataProvider for the pallet_provider::Pallet

  impl<T: Config> DataProvider for Pallet<T> {
     type Key = T::Index;
     type Data = T::Data;
  
     fn query(key: Self::Key) -> Option<Self::Data> {
        Data::<T>::get(&key)
     }
  }
  

• Create an associated type in the pallet_consumer::Config

  #[pallet::config]
  pub trait Config: frame_system::Config {
      ...
  
      type Provider: DataProvider<Key = u32, Data = u32>;
  
      ...
  }
  

NOTE: We are enforcing here, that the associated type that implements DataProvider is using the Key and Index we are expecting. Which gives us an error upon compile time if the used type for Provider in the runtime does not match this expectation.

Summary

• The first two solutions are error prone, as changes in naming or types in the pallet_provder might be unnoticed by the pallet_consumer. This could result in unexpected errors or even UB if the decoding still succeeds.
• Solution 1 should be a last resort, but is generally useful to understand how keys for storage elements are generated. Furthermore, it allows querying storage via rpc methods for easily retrieving a whole map.
• Solution 2 could be a solution when a pallet has a private storage type and changes to that privacy or trait abstraction are out of reach (e.g. pallet is maintained by another team).
• Solution 3 should be favored in my opinion (if not tight-coupling). It allows to test the pallets independently of each other, it uses the compiler to ensure type-safety and is friendly to refactoring in the future.

Answer 2

After some debugging and help from mustermeiszer I was able to spot the problem. In the problem was that I have the pallet defined as "LLM" and not "llm"(small cap) in the construct_runtime:

construct_runtime!(
    pub enum Runtime where
        Block = Block,
        NodeBlock = node_primitives::Block,
        UncheckedExtrinsic = UncheckedExtrinsic
    {
        System: frame_system,
        Utility: pallet_utility,
        Babe: pallet_babe,
        Timestamp: pallet_timestamp,
        // Authorship must be before session in order to note author in the correct session and era
        // for im-online and staking.
        Authorship: pallet_authorship,
        Indices: pallet_indices,
        Balances: pallet_balances,
        TransactionPayment: pallet_transaction_payment,
        AssetTxPayment: pallet_asset_tx_payment,
        ElectionProviderMultiPhase: pallet_election_provider_multi_phase,
        Staking: pallet_staking,
        Session: pallet_session,
        Democracy: pallet_democracy,
        Council: pallet_collective::<Instance1>,
        TechnicalCommittee: pallet_collective::<Instance2>,
        Elections: pallet_elections_phragmen,
        TechnicalMembership: pallet_membership::<Instance1>,
        Grandpa: pallet_grandpa,
        Treasury: pallet_treasury,
        Contracts: pallet_contracts,
        Sudo: pallet_sudo,
        ImOnline: pallet_im_online,
        AuthorityDiscovery: pallet_authority_discovery,
        Offences: pallet_offences,
        Historical: pallet_session_historical::{Pallet},
        RandomnessCollectiveFlip: pallet_randomness_collective_flip,
        Identity: pallet_identity,
        Society: pallet_society,
        Recovery: pallet_recovery,
        Vesting: pallet_vesting,
        Scheduler: pallet_scheduler,
        Preimage: pallet_preimage,
        Proxy: pallet_proxy,
        Multisig: pallet_multisig,
        Bounties: pallet_bounties,
        Tips: pallet_tips,
        Assets: pallet_assets,
        Mmr: pallet_mmr,
        Lottery: pallet_lottery,
        Gilt: pallet_gilt,
        Uniques: pallet_uniques,
        TransactionStorage: pallet_transaction_storage,
        BagsList: pallet_bags_list,
        StateTrieMigration: pallet_state_trie_migration,
        ChildBounties: pallet_child_bounties,
        Referenda: pallet_referenda,
        ConvictionVoting: pallet_conviction_voting,
        Whitelist: pallet_whitelist,
        LLM: pallet_llm, //{Pallet, Storage, Event<T>}, // LLM Pallet
    }
);

Based on centrifuge integration I created a mod for it:

pub mod llmmod {
    use super::*;
    use frame_support::{
        storage::types::{StorageMap, StorageValue, ValueQuery},
        traits::StorageInstance,
        Blake2_128Concat, Twox64Concat,
    };
    // ParaLifecyclesPrefix, based on centrifuge
    pub struct LLMPoliticsCopy;

    impl StorageInstance for LLMPoliticsCopy {
        fn pallet_prefix() -> &'static str {
            "LLM"
        }

        const STORAGE_PREFIX: &'static str = "LLMPolitics";
    }
    pub type LLMPolitics<T> = frame_support::storage::types::StorageMap<
        LLMPoliticsCopy,
        Blake2_128Concat,
        <T as frame_system::Config>::AccountId,
        u128,
        frame_support::pallet_prelude::ValueQuery,
    >;
    //type AccountId = frame_system::Config::AccountId;

    pub fn check_pooled_llm<T: frame_system::Config>(sender: T::AccountId) -> bool {
        LLMPolitics::<T>::contains_key::<T::AccountId>(sender)
    }

    

    
}

Then in the other pallet I do a simple contains check to check for the account:

ensure!(!llmmod::check_pooled_llm::<T>(who.clone()), Error::<T>::NoPolLLM);