Panic when Using Rayon with runtime_api in a custom RPC

Question

Works: Use normal iteration in a custom RPC with a runtime_api() call. Goal: Use rayon parallel iteration instead. Result: Panic

Working code Simplified Real Code in GitHub

let api = self.client.runtime_api();
let at = BlockId::hash(self.client.info().best_hash);

let provider = Provider(provider_msa_id);

Ok(delegator_msa_ids
    .iter()
    .map(|&id| {
        let delegator = Delegator(id);
        let has_delegation =
            api.has_delegation(&at, delegator, provider, block_number).unwrap();
        (id, has_delegation)
    }).collect())

I want to swap out .iter with par_iter from https://github.com/rayon-rs/rayon

It works if I only pass 1 delegator_msa_ids into the request.

If I do more, I get this panic:

   0: backtrace::capture::Backtrace::new
   1: sp_panic_handler::set::{{closure}}
   2: std::panicking::rust_panic_with_hook
             at /rustc/848090dcd18553b790461132ca9d2a020aeea9a2/library/std/src/panicking.rs:702:17
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/848090dcd18553b790461132ca9d2a020aeea9a2/library/std/src/panicking.rs:588:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/848090dcd18553b790461132ca9d2a020aeea9a2/library/std/src/sys_common/backtrace.rs:138:18
   5: rust_begin_unwind
             at /rustc/848090dcd18553b790461132ca9d2a020aeea9a2/library/std/src/panicking.rs:584:5
   6: core::panicking::panic_fmt
             at /rustc/848090dcd18553b790461132ca9d2a020aeea9a2/library/core/src/panicking.rs:142:14
   7: core::result::unwrap_failed
             at /rustc/848090dcd18553b790461132ca9d2a020aeea9a2/library/core/src/result.rs:1814:5
   8: <frequency_rococo_runtime::RuntimeApiImpl<__SR_API_BLOCK__,RuntimeApiImplCall> as sp_api::Core<__SR_API_BLOCK__>>::__runtime_api_internal_call_api_at
   9: pallet_msa_runtime_api::MsaApi::has_delegation
  10: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
  11: rayon::iter::plumbing::bridge_producer_consumer::helper
  12: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
  13: rayon_core::registry::WorkerThread::wait_until_cold
  14: rayon_core::registry::ThreadBuilder::run
  15: std::sys_common::backtrace::__rust_begin_short_backtrace
  16: core::ops::function::FnOnce::call_once{{vtable.shim}}
  17: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/848090dcd18553b790461132ca9d2a020aeea9a2/library/alloc/src/boxed.rs:1935:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/848090dcd18553b790461132ca9d2a020aeea9a2/library/alloc/src/boxed.rs:1935:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/848090dcd18553b790461132ca9d2a020aeea9a2/library/std/src/sys/unix/thread.rs:108:17
  18: __pthread_deallocate


Thread '<unnamed>' panicked at 'already borrowed: BorrowMutError', /Users/wilwade/dev/mrc/runtime/frequency-rococo/src/lib.rs:713

The lib.rs:713 is just the macro impl_runtime_apis! around the runtime api impl which does no mutation.

• Impl Link
• Pallet Logic Link

Current theories:

1. The runtime_api CAN do a mutation, and so something there is causing the borrow issue
2. I am accessing state wrong in the pallet logic and that is providing a mutation lock when it shouldn't
3. Bug in substrate?

Current Fix Ideas:

• See current theories
• A different way entirely to access the runtime state that is clear that I am not, and cannot mutate state.
• A different type that makes it clear I am not mutating.
• Create a new runtime_api inside of the par_iter (This works, but it warns that it runs out of wasm instances and also feels like a waste)

Answer 1

The impl_runtime_apis! macro added some invalid unsafe trait implementations of Sync and Send. These are going to be removed by this pr.

Creating the runtime_api would be the correct way, as you already have done this. The warning that you are running out of wasm instances would also be the result of sharing the runtime_api, because internally the wasm executor uses one instance per runtime call. The executor caches these instances, but when the upper number is reached you see this warning as it will create instances on the fly that are being destroyed afterwards. So, I would propose that you may change your has_delegation to take a Vec<Delegate> or maybe add a second function to do this. Then you could fetch all the required information in one call to the runtime.