Multiple remove_all calls in same block

Question

I stumbled upon an unexpected behavior when doing storage migration.

Problem Description

I want to clean-up a large DoubleStorageMap in multiple steps. And that works just fine, however I also want to be able to test this using try-runtime. So I modified the code to recursively call the clean-up when try-runtime feature is enabled.

The code looks something like this:

        pub fn storage_cleanup<T: Config>(weight_limit: Weight) -> Weight {
            ...
            let limit = ...;
            match SomeDoubleStorageMap::<T>::remove_all(Some(limit)) {
                KillStorageResult::AllRemoved(removed_entries_num) => {
                    consumed_weight += deletion_weight * removed_entries_num as u64;
                }
                KillStorageResult::SomeRemaining(removed_entries_num) => {
                    consumed_weight += deletion_weight * removed_entries_num as u64;
                    if cfg!(feature = "try-runtime") {
                        return consumed_weight + storage_cleanup::<T>(weight_limit);
                    } else {
                        return consumed_weight;
                    }
                }
            }
            ...
        }

Scenario

• Let's say that initially, SomeDoubleStorageMap has 3000 entries.
• Limit is set to 100 meaning that at most, we'll delete 100 entries in one call.
• I want to emphasize again that it works just fine when tested on-chain where each deletion is part of another block.
• When this is tested with try-runtime, it will enter an infinite loop.
• I've tried printing removed_entries_num and size of the storage map and it goes like this:
  • First Iteration
    • Initial map size is 3000
    • 100 entries are removed, 2900 entries remain
  • Second Iteration
    • Initial map size is 2900
    • entries are removed, 2900 entries remain
  • Previous step is performed until stack is exhausted

remove_all provides info as if something was removed, but the size doesn't change. I assume it has to be related to the fact I'm doing it all as part of a single block.

Question

Is this a bug or expected behavior? I've checked the docs and the code but couldn't find why this happens.

Answer 1

This is expected behavior. At least for now. I opened the following pr to improve the docs around this: https://github.com/paritytech/substrate/pull/11182