`try-runtime` doesn't start with a proper state

Question

Context:

Testing try-runtime::follow-chain utility with a local live chain.

Problem:

RemoteExternalities doesn't create a perfect copy of the state.

After follow-chain connects to a node, it starts listening for finalized heads. After encountering first such, it uses RemoteExternalities to build local state based on that block's parent. However, the created storage differs from the one on-chain.

This happens also when we use exactly the same runtime code.

Logs from follow-chain (pay attention to storage root hash):

DEBUG main try-runtime::cli: new block event: 0x3e9f789a0c13c9b9a57784c7c49fe3ca8a4484b47add564eb95116c0c6a36239 => 539, extrinsics: 1, storage root: 0xe0a9f15698564ddd1511161dba5582b9cd4e9d7a63a676129619a3492df451fc
 WARN tokio-runtime-worker jsonrpsee_core::client::async_client: Custom("[backend]: frontend dropped; terminate client")
 INFO                 main jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "localhost", host_header: "localhost:9944", _mode: Plain, path_and_query: "/" }
 INFO                 main remote-ext: scraping key-pairs from remote @ 0x781879a43612925645215738464ee16f68f84f556f7950092b0d1a46e1ec236f    
 INFO                 main remote-ext: downloading data for all pallets.    
 INFO                 main remote-ext: 👩‍👦 scraping child-tree data from 0 top keys    
 WARN tokio-runtime-worker jsonrpsee_core::client::async_client: Custom("[backend]: frontend dropped; terminate client")
 INFO                 main remote-ext: injecting a total of 721 top keys    
 INFO                 main remote-ext: injecting a total of 0 child keys    
 INFO                 main remote-ext: initialized state externalities with storage root 0xa7eca01edf1691709442b5fb7f4d748b02ba6308b580ff6c34e4dc7ae00b451e    
 INFO                 main try-runtime::cli: initialized state externalities at 539, storage root 0xa7eca01edf1691709442b5fb7f4d748b02ba6308b580ff6c34e4dc7ae00b451e

Also, screenshot from the PolkadotUI describing block 538 (the parent block, from which we are scraping state):

Reproduction:

I built a binary with cargo build --features "try-runtime". This served both as the node executable and try-runtime tool. I used chainspec that was used by nodes to start the chain (so the same runtime blob).

Note: I am aware that, in the basic scenario, storage roots will start mismatching anyway after processing first block (e.g. seal digest item will not be included in the re-executed blocks). Nevertheless, I'm trying to achieve full storage match in the beginning as a baseline for further testing. For this I'm already adding all needed digests, but since there's already an initial storage mismatch, I cannot proceed.

Answer 1

The problem was with StateVersion -- for RemoteExternalities Builder it is by default V1 and there is no way of changing that from try-runtime level. My chain has StateVersion set to V0 and hence, the mismatch occurred.