7

We are running into an allocator memory error where the number of heap pages appears to be insufficient to run a transaction on our validator (substrate-node).

This is the error in question:

2022-04-25 13:10:40 Evicting failed runtime instance: Runtime panicked: Failed to allocate memory: "Allocator ran out of space"
2022-04-25 13:10:40 Essential task `transaction-pool-task-0` failed. Shutting down service.
^C^C2022-04-25 13:10:42 running out of space with current bumper 135395720, mem size 135397376

====================

Version: 4.0.0-dev-e9702fe-x86_64-linux-gnu

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:702:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:588:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:138:18
   4: rust_begin_unwind
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
   5: core::panicking::panic_fmt
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
   6: core::result::unwrap_failed
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/result.rs:1749:5
   7: tracing::span::Span::in_scope
   8: sp_io::allocator::ExtAllocatorMallocVersion1::call
   9: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
  18: <unknown>
  19: wasmtime_runtime::traphandlers::catch_traps::call_closure
  20: wasmtime_setjmp
  21: wasmtime_runtime::traphandlers::catch_traps
  22: wasmtime::func::invoke_wasm_and_catch_traps
  23: wasmtime::func::typed::TypedFunc<Params,Results>::call
  24: sc_executor_wasmtime::instance_wrapper::EntryPoint::call
  25: sc_executor_wasmtime::runtime::perform_call
  26: <sc_executor_wasmtime::runtime::WasmtimeInstance<H> as sc_executor_common::wasm_runtime::WasmInstance>::call
  27: sc_executor_common::wasm_runtime::WasmInstance::call_export
  28: std::panicking::try
  29: std::thread::local::LocalKey<T>::with
  30: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
  31: sc_executor::wasm_runtime::RuntimeCache::with_instance
  32: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
  33: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
  34: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  35: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  36: sp_api::runtime_decl_for_Core::initialize_block_call_api_at
  37: <node_template_runtime::RuntimeApiImpl<__SR_API_BLOCK__,RuntimeApiImplCall> as sp_api::Core<__SR_API_BLOCK__>>::Core_initialize_block_runtime_api_impl
  38: sp_api::Core::initialize_block
  39: tracing::span::Span::in_scope
  40: sc_transaction_pool::api::validate_transaction_blocking
  41: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  42: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  43: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  44: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  45: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  46: std::thread::local::LocalKey<T>::with
  47: tokio::park::thread::CachedParkThread::block_on
  48: tokio::runtime::handle::Handle::block_on
  49: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  50: tokio::runtime::task::harness::Harness<T,S>::poll
  51: tokio::runtime::blocking::pool::Inner::run
  52: std::sys_common::backtrace::__rust_begin_short_backtrace
  53: core::ops::function::FnOnce::call_once{{vtable.shim}}
  54: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/alloc/src/boxed.rs:1853:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/alloc/src/boxed.rs:1853:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys/unix/thread.rs:108:17
  55: start_thread
  56: clone


Thread 'tokio-runtime-worker' panicked at 'Failed to allocate memory: "Allocator ran out of space"', /root/.cargo/git/checkouts/substrate-7e08433d4c370a21/22d40c7/primitives/io/src/lib.rs:1284

To resolve this, we are looking at increasing the number of heap pages on this line in Service.rs:

    let executor = NativeElseWasmExecutor::<ExecutorDispatch>::new(
        config.wasm_method,
        config.default_heap_pages,
        config.max_runtime_instances,
        config.runtime_cache_size,
    );

The question is we do not know what this value is, as the documentation only says "default number of 64KB pages to allocate for wasm execution": https://docs.substrate.io/rustdocs/latest/sc_service/config/struct.Configuration.html#structfield.default_heap_pages

Any idea what the current value is of this variable? Alternatively, can we increase it sufficiently?

1
  • What transaction caused this? Was it a large batch transaction by any chance? Something similar was fixed in substrate#11206, which is not released yet AFAIK. Commented Apr 25, 2022 at 18:01

2 Answers 2

3

Modifying the client code probably is one way.

However, the canonical way is to alter the number of extra pages mounted is to change the value of :heappages in the storage.

The number is SCALE-encoded u64. The u64 is not compact-encoded, so it's just 8 byte integer encoded as little-endian.

1

We figured it out with help from @CustomBlink, the default value is

Substrate: client/executor/src/native_executor.rs: Lines 51-52:
51    /// Default num of pages for the heap
52    const DEFAULT_HEAP_PAGES: u64 = 2048

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.