How to iterate a large `StorageMap` over multiple extrinsic calls?

Question

Hey valuable Substrate community,

I am running into the problem that I need to iterate over an unbounded StorageMap. So I need an IterationLimit and use an extrinsic which gets called multiple times.

Example: https://github.com/zeitgeistpm/zeitgeist/blob/f81d03da05f1c3ab75b27af6964594df2dfee6d8/zrml/court/src/lib.rs#L510-L555

The problem is that I can't drain the StorageMap, because the storage is still required. So I need to iterate and continue iterating in the subsequent calls.

But it is possible that the StorageMap changes (keys inserted) between the calls that exceed the iteration limit.

Test that fails with the last assert: https://github.com/zeitgeistpm/zeitgeist/blob/f81d03da05f1c3ab75b27af6964594df2dfee6d8/zrml/court/src/tests.rs#L429-L473

With the above test I found out, that new key inserts are not ensured to be inserted after previous key inserts. So I can't use iter_from or iter_keys_from.

So my aim is to iterate over all elements and if all elements inside the StorageMap fulfill a specific condition, then it should return Ok, otherwise it throws and error.

Answer 1

If you want to iterate over something then it cannot change in the meantime.
This applies to storage maps as well. Iterating over large StorageMaps in calls is not a good idea in general since it uses up a lot of weight and proof size.

I dont know what you do but there is a chance that you can use witnesses, or look into alternative data structures. Otherwise secondary indices which provide you the information without iteration (but need to be maintained on every modification) are possible as well.

If you still really want to iterate then you somehow have to "lock" that map from modifications while the iteration happens. Like a Locked storage item or so, that would reject any modifications to the map.