Substrate storage query response does not seems to be correct for System.digests

Question

I am developing a series of Java tools for Substrate based chains, polkadot4j (it is currently under heavy development). I have problem in querying the storage for the pallet system and storage entry named "Digest". the storage key for this entry is 0x26aa394eea5630e07c48ae0c9558cef799e7f93fc6a98f0874fd057f111c4d2d, and the response I get for this key is: 0x080642414245b50103680100009d19291100000000a8691d03f5a419f00c91ce7d1b46385f9d86d3c6d67598560a0d1522245aba2f5590ae72b150586df25e56ef109a6fee3c656b4006512a52d0629a0901df9108a9b30875c15fc8370363616c376b1fecf9b164a299d50b2eceabbf5cc1d8c80d044245454684034dc81d0cb1830f54832f93ca94511a27fc6e8aafd2fc7cfcf93ed24393a1e4d2

this response should be decoded to Digest that is:

#[derive(PartialEq, Eq, Clone, Encode, Decode, RuntimeDebug, TypeInfo, Default)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
pub struct Digest {
    /// A list of logs in the digest.
    pub logs: Vec<DigestItem>,
}

and the DigestItem is :

#[derive(PartialEq, Eq, Clone, RuntimeDebug)]
pub enum DigestItem {
    /// A pre-runtime digest.
    ///
    /// These are messages from the consensus engine to the runtime, although
    /// the consensus engine can (and should) read them itself to avoid
    /// code and state duplication. It is erroneous for a runtime to produce
    /// these, but this is not (yet) checked.
    ///
    /// NOTE: the runtime is not allowed to panic or fail in an `on_initialize`
    /// call if an expected `PreRuntime` digest is not present. It is the
    /// responsibility of a external block verifier to check this. Runtime API calls
    /// will initialize the block without pre-runtime digests, so initialization
    /// cannot fail when they are missing.
    PreRuntime(ConsensusEngineId, Vec<u8>),

    /// A message from the runtime to the consensus engine. This should *never*
    /// be generated by the native code of any consensus engine, but this is not
    /// checked (yet).
    Consensus(ConsensusEngineId, Vec<u8>),

    /// Put a Seal on it. This is only used by native code, and is never seen
    /// by runtimes.
    Seal(ConsensusEngineId, Vec<u8>),

    /// Some other thing. Unsupported and experimental.
    Other(Vec<u8>),

    /// An indication for the light clients that the runtime execution
    /// environment is updated.
    ///
    /// Currently this is triggered when:
    /// 1. Runtime code blob is changed or
    /// 2. `heap_pages` value is changed.
    RuntimeEnvironmentUpdated,
}

so we excpect a sequence of DigestItem. now look again to the response, 0x080642414245b50103680100009d1929110000..., the first byte (08) is compact encoded of value 2, this is the sequence length, so we have 2 DigestItem, the second byte (06) represent the first DigestItem's variant (u8 encoded), that is the value 6, but DigestItem has 5 variants!

what is the problem? am I missing something?

note: https://polkadot.js.org/apps gives this result:

{
  logs: [
    {
      PreRuntime: [
        BABE
        0x03c90000004c1b291100000000549c0018aaa01e9516bb25a99404a2c3ead4d6f9b55f02f80aa4f78481e0a15e2742c8379c499c0816c7cee687bad5666d00634eaaa82435b62faac7eed7ec0d6daa797ef3469f08d345f9afa646706789cbe670eba23b96a9a938df82f35a08
      ]
    }
    {
      Consensus: [
        BEEF
        0x03d584f9d4530d898fcdfc83cb205654c66d10f011983b5bd7505c6a3637b829e7
      ]
    }
  ]
}

this is two items.

thank you

Answer 1

Generally, when encoding an enum type with the SCALE codec, the first byte (u8) is supposed to indicate which variant follows. If DigestType implemented the encoding using the Encode derive macro from parity-scale-codec, it would have worked as you expected. However, it's also possible to define separate encoding methods for different types.

#[repr(u32)]
#[derive(Encode, Decode)]
pub enum DigestItemType {
    Other = 0,
    Consensus = 4,
    Seal = 5,
    PreRuntime = 6,
    RuntimeEnvironmentUpdated = 8,
}

If you look at the Encode trait implementation of DigestItem, instead of automatically encoding the variant index, it manually assigns values using the above DigestItemType. You can verify that the 0x06 points to PreRuntime.