Benchmarking Mock

Question

We’re trying to do the exact thing as referenced in How to mock trait in benchmarking, and we have implemented the struct + trait we need in our mocks, however, the benchmarks are calling the runtime config version and not the mock config version of the trait implementation. The mock pallet config uses T::AccountHandler, as defined inside mock.rs, for a defined AccountProvider trait. NB: this was converted from an answer to a comment in the linked question, but all the code I included was cut off and I can’t edit the comment.

mock.rs:

pub struct AccountHandler;
impl AccountProvider<AccountId, BlockNumber> for AccountHandler {
    fn get_msa_id(key: &AccountId) -> Option<MessageSenderId> {
      // etc.
    }
}
impl pallet_messages::Config for Test {
    type Event = Event;
    type AccountProvider = AccountHandler;
    type WeightInfo = ();
    type MaxMessagesPerBlock = MaxMessagesPerBlock;
    type MaxMessageSizeInBytes = MaxMessageSizeInBytes;
}

Inside the pallet, it uses AccountProvider like so:

    #[pallet::config]
    pub trait Config: frame_system::Config {
        
        type AccountProvider: AccountProvider<Self::AccountId, Self::BlockNumber>;
       /// etc.
    }
    // inside the extrinsic we want to benchmark, the trait function is called like so:
        #[pallet::weight(T::WeightInfo::add(message.len() as u32, 1_000))]
        pub fn add(
            origin: OriginFor<T>,
            message: Vec<u8>,
        ) -> DispatchResultWithPostInfo {
            let who = ensure_signed(origin)?;
            // etc.
            let msa_id = T::AccountProvider::get_msa_id(&who);
    }

We’re building the benchmarks with --runtime, and we are using substrate polkadot-v0.9.18 substrate.

why is our benchmark not using the mocked trait

Answer 1

Benchmarks will use the "mock" when running tests, since that is the runtime environment set up for tests.

However, for benchmarks to be accurate in production, when they are actually run, they are executed against the real runtime environment. You can imagine your mock and your real runtime can be configured completely differently, and this can cause differences in the performance and results of your benchmarks.

When writing benchmarks, you must make sure to write them generally, so that they would work across your mock and production environment. For example, if there is some logic which needs to transfer balance between two users, you should not say: transfer(100, alice, bob), since "100" is an arbitrary number, and may work in your mock, but may not work on a chain where the minimum balance is more like 1_000_000_000_000. So instead, doing something like: transfer(T::Currency::ExistentialDeposit::get(), alice, bob) is correct, since this will always read from your runtime configuration and execute the appropriate logic needed.

Answer 2

I think the real problem is that you should not only build the binary with --release, but also you should enable the runtime-benchmarks feature.

For eg.

cargo build --release --features=runtime-benchmarks

Pay attention to the "--features..." part.