If we are using a single or double maps in our pallets to store data on-chain, is there a limit to the number of records or size of one individual map?
I understand we should be storing the minimal amount of data on-chain but say 10 years has passed and millions of records have been created in this map, is that possible? will there be any issues in terms of performance?
Storage size and density are both relevant concerns when it comes to the performance of your chain. There are limits and performance implications at many levels so I will break them down:
The Substrate runtime executes within a wasm32 environment, which means we have the same general limitations as any 32-bit environment, for example the maximum theoretical number of items in a Vec is u32::MAX or 232 or 4,294,967,296. In practice though it will much less because that assumes a vector of bytes and does not account for any other overheads.
Our wasm environment itself is configured with a maximum allocation size which is currently set to 32 MiB. That means trying to access any item in storage which requires more than 32 MiB allocated memory will cause a panic in the runtime.
Substrate's generic key-value database can support arbitrary size keys and values.
Substrate uses a base-16 merkle trie as default, and thus we can expect the complexity of accessing item in the trie to grow at log16 of the number of items in the overall state.
When reading a raw value from the database, we should expect that larger files are more complex for the database to handle than smaller files.
Additionally, to convert the raw encoded bytes into a Rust type-safe object, Parity SCALE Codec needs to decode those bytes, which increases in complexity based on the number of bytes encoded.
While I cannot speak towards optimization of RocksDB with various filesizes, Parity DB is optimized for certain kinds of key/values. Those optimizations are noted in the project.
There isn't a strict limit, however the more entries a map has, the more trie nodes need to be read/altered when accessing it, so overall the slower the I/O. It's logarithmic in complexity, so you can expand the map quite far before performance degrades substantiallya.
Regardless, you should always be thinking about economic incentives to ensure that data is not left to bloat the map needlessly. A simple one is the deposit scheme, whereby value must be taken out of circulation/usage as long as data is stored in the map. This helps ensure that (even after 10 years) the state is storing a manageable amount of data and only that which is the most economically valuable. Polkadot employs this technique ubiquitously.
