13

If we are using a single or double maps in our pallets to store data on-chain, is there a limit to the number of records or size of one individual map?

I understand we should be storing the minimal amount of data on-chain but say 10 years has passed and millions of records have been created in this map, is that possible? will there be any issues in terms of performance?

2 Answers 2

11

Storage size and density are both relevant concerns when it comes to the performance of your chain. There are limits and performance implications at many levels so I will break them down:

Physical Limits

The Substrate runtime executes within a wasm32 environment, which means we have the same general limitations as any 32-bit environment, for example the maximum theoretical number of items in a Vec is u32::MAX or 232 or 4,294,967,296. In practice though it will much less because that assumes a vector of bytes and does not account for any other overheads.

Wasm Limits

Our wasm environment itself is configured with a maximum allocation size which is currently set to 32 MiB. That means trying to access any item in storage which requires more than 32 MiB allocated memory will cause a panic in the runtime.

Merkle Trie Limits

Substrate's generic key-value database can support arbitrary size keys and values.

Merkle Trie Complexity

Substrate uses a base-16 merkle trie as default, and thus we can expect the complexity of accessing item in the trie to grow at log16 of the number of items in the overall state.

Size / Decoding Performance

When reading a raw value from the database, we should expect that larger files are more complex for the database to handle than smaller files.

Additionally, to convert the raw encoded bytes into a Rust type-safe object, Parity SCALE Codec needs to decode those bytes, which increases in complexity based on the number of bytes encoded.

Database Optimizations

While I cannot speak towards optimization of RocksDB with various filesizes, Parity DB is optimized for certain kinds of key/values. Those optimizations are noted in the project.

1
  • If I understand correctly adding to the size of the merkle trie adds to the complexity of accessing items. What size of merkle trie for the chain would mean that items could not be accessed (potentially due to the 16MB WAMS limit). I ask am I am working on a supply chain application that is intended to register and track a large number of products.
    – IanJScott
    Feb 11, 2022 at 16:09
13

There isn't a strict limit, however the more entries a map has, the more trie nodes need to be read/altered when accessing it, so overall the slower the I/O. It's logarithmic in complexity, so you can expand the map quite far before performance degrades substantiallya.

Regardless, you should always be thinking about economic incentives to ensure that data is not left to bloat the map needlessly. A simple one is the deposit scheme, whereby value must be taken out of circulation/usage as long as data is stored in the map. This helps ensure that (even after 10 years) the state is storing a manageable amount of data and only that which is the most economically valuable. Polkadot employs this technique ubiquitously.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.