Debug corrupted on dev update

Question

We are performing upgrade on a chain, in the dev side just to ensure all is OK we do step by step using a forked state of the live chain, we got corrupted state log which we want to debug in order to solve it (example bellow), what is the best way to decode the array, is there a way to make it human readable directly in the log of the node ?

Corrupted state at [38, 170, 57, 78, 234, 86, 48, 224, 124, 72, 174, 12, 149, 88, 206, 247, 185, 157, 136, 14, 198, 129, 121, 156, 12, 243, 14, 136, 134, 55, 29, 169, 222, 30, 134, 169, 168, 199, 57, 134, 76, 243, 204, 94, 194, 190, 165, 159, 212, 53, 147, 199, 21, 253, 211, 28, 97, 20, 26, 189, 4, 169, 159, 214, 130, 44, 133, 88, 133, 76, 205, 227, 154, 86, 132, 231, 165, 109, 162, 125]

Answer 1

This error message comes from: https://github.com/paritytech/substrate/blob/master/frame/support/src/storage/unhashed.rs

/// Return the value of the item in storage under `key`, or `None` if there is no explicit entry.
pub fn get<T: Decode + Sized>(key: &[u8]) -> Option<T> {
    sp_io::storage::get(key).and_then(|val| {
        Decode::decode(&mut &val[..]).map(Some).unwrap_or_else(|_| {
            // TODO #3700: error should be handleable.
            log::error!(
                target: "runtime::storage",
                "Corrupted state at {:?}",
                key,
            );
            None
        })
    })
}

That hex is the storage key with the corrupted state.

Turning the array into a hex gives you:

0x26aa394eea5630e07c48ae0c9558cef7b99d880ec681799c0cf30e8886371da9de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d

You can then use an index of storage keys to find a match.

You may not find an exact match, but you can keep trimming the key from the end to see if a the prefix matches anything.

From that you will find it matches:

{
  "name": "system account",
  "key": "0x26aa394eea5630e07c48ae0c9558cef7b99d880ec681799c0cf30e8886371da9"
},

So this tells you that one of the accounts on your chain is not encoded correctly. Probably you did not migrate some change to the storage structure of an account.

Answer 2

You can not make it readable in the logs of the node, because Frame uses concatenated hashes to build the final key. However, as you know the scheme and your runtime, you can start reverse engineer the key. Each Frame storage key is twox_128(PALLET_NAME) ++ twox_128(STORAGE_NAME). Where PALLET_NAME being the name of your pallet as configured in your construct runtime. STORAGE_NAME is the name of the storage type in your pallet type Xy = Value<..> where Xy being this name. I think there exists some example in polakdot js somewhere to generate you a list of all of these "key prefixes". Then you just need to match these prefixes against the key from your log.