We are running into an allocator memory error where the number of heap pages appears to be insufficient to run a transaction on our validator (substrate-node).
This is the error in question:
2022-04-25 13:10:40 Evicting failed runtime instance: Runtime panicked: Failed to allocate memory: "Allocator ran out of space"
2022-04-25 13:10:40 Essential task `transaction-pool-task-0` failed. Shutting down service.
^C^C2022-04-25 13:10:42 running out of space with current bumper 135395720, mem size 135397376
====================
Version: 4.0.0-dev-e9702fe-x86_64-linux-gnu
0: sp_panic_handler::set::{{closure}}
1: std::panicking::rust_panic_with_hook
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:702:17
2: std::panicking::begin_panic_handler::{{closure}}
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:588:13
3: std::sys_common::backtrace::__rust_end_short_backtrace
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:138:18
4: rust_begin_unwind
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
5: core::panicking::panic_fmt
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
6: core::result::unwrap_failed
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/result.rs:1749:5
7: tracing::span::Span::in_scope
8: sp_io::allocator::ExtAllocatorMallocVersion1::call
9: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim
11: <unknown>
12: <unknown>
13: <unknown>
14: <unknown>
15: <unknown>
16: <unknown>
17: <unknown>
18: <unknown>
19: wasmtime_runtime::traphandlers::catch_traps::call_closure
20: wasmtime_setjmp
21: wasmtime_runtime::traphandlers::catch_traps
22: wasmtime::func::invoke_wasm_and_catch_traps
23: wasmtime::func::typed::TypedFunc<Params,Results>::call
24: sc_executor_wasmtime::instance_wrapper::EntryPoint::call
25: sc_executor_wasmtime::runtime::perform_call
26: <sc_executor_wasmtime::runtime::WasmtimeInstance<H> as sc_executor_common::wasm_runtime::WasmInstance>::call
27: sc_executor_common::wasm_runtime::WasmInstance::call_export
28: std::panicking::try
29: std::thread::local::LocalKey<T>::with
30: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
31: sc_executor::wasm_runtime::RuntimeCache::with_instance
32: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
33: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
34: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
35: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
36: sp_api::runtime_decl_for_Core::initialize_block_call_api_at
37: <node_template_runtime::RuntimeApiImpl<__SR_API_BLOCK__,RuntimeApiImplCall> as sp_api::Core<__SR_API_BLOCK__>>::Core_initialize_block_runtime_api_impl
38: sp_api::Core::initialize_block
39: tracing::span::Span::in_scope
40: sc_transaction_pool::api::validate_transaction_blocking
41: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
42: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
43: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
44: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
45: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
46: std::thread::local::LocalKey<T>::with
47: tokio::park::thread::CachedParkThread::block_on
48: tokio::runtime::handle::Handle::block_on
49: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
50: tokio::runtime::task::harness::Harness<T,S>::poll
51: tokio::runtime::blocking::pool::Inner::run
52: std::sys_common::backtrace::__rust_begin_short_backtrace
53: core::ops::function::FnOnce::call_once{{vtable.shim}}
54: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/alloc/src/boxed.rs:1853:9
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/alloc/src/boxed.rs:1853:9
std::sys::unix::thread::Thread::new::thread_start
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys/unix/thread.rs:108:17
55: start_thread
56: clone
Thread 'tokio-runtime-worker' panicked at 'Failed to allocate memory: "Allocator ran out of space"', /root/.cargo/git/checkouts/substrate-7e08433d4c370a21/22d40c7/primitives/io/src/lib.rs:1284
To resolve this, we are looking at increasing the number of heap pages on this line in Service.rs:
let executor = NativeElseWasmExecutor::<ExecutorDispatch>::new(
config.wasm_method,
config.default_heap_pages,
config.max_runtime_instances,
config.runtime_cache_size,
);
The question is we do not know what this value is, as the documentation only says "default number of 64KB pages to allocate for wasm execution": https://docs.substrate.io/rustdocs/latest/sc_service/config/struct.Configuration.html#structfield.default_heap_pages
Any idea what the current value is of this variable? Alternatively, can we increase it sufficiently?
