How to benchmark huge bounded vectors?

Question

I want to have a queue system in a pallet using a storage value and a bounded vector.

/// Data related to tranmission queues.
#[pallet::storage]
#[pallet::getter(fn at_block_queue)]
pub type AtBlockQueue<T: Config> = StorageValue<_, Queue<T::BlockNumber, T::SimultaneousTransmissionLimit>, ValueQuery>;

pub struct Queue<BlockNumber: Clone + PartialEq + Debug + sp_std::cmp::PartialOrd, Limit: Get<u32>>(
    pub BoundedVec<(NFTId, BlockNumber), Limit>,
);

I would like this queue to be limited at 10_000_000 values. This seems ok while testing but the benchmarks are failing due to a failed allocation.

In my benchmarking code, I use a function to fill the queue to simulate usage :

// Benchmark / tests only
    pub fn bulk_insert(
        &mut self,
        nft_id: NFTId,
        block_number: BlockNumber,
        number: u32,
    ) -> Result<(), ()> {
        self.0.try_extend(vec![(nft_id, block_number); number as usize].into_iter())
    }

I was wondering if the Limit value was too high or my benchmarking method is wrong.

Answer 1

10M entries of u32 in a vector? Sure this will not fit in the memory of the runtime, which is capped at 32 MiB.

Three things to consider: It looks like your data structure is rather a map. Then instead of going for StorageValue<Vec< which should only be the last resort, you could try out StorageDoubleMap which maps NFTId x BlockNumber to Limit. Then you can have theoretically unlimited number of entries.

Other thing is that depending on what you mean by "queue" you could try to use the Message Queue pallet, which can be used to enqueue and later-on process messages in a PoV-safe way.

Last thing is that even if the 10M entries would go through the benchmark, the resulting PoV would be to large to deploy onto any para-chain. Currently para-chains are capped to 5MiB Proof size per block. Accessing that storage value would consume at least 40MiB in the worst case.