3

We are getting the below canonicalization issue and we are wondering:

  • Is it a problem or is it normal and can be ignored?
  • Has anyone found a method to resolve it?

Here's what we know:

  • It consistently begins at block 2400, and occurs in every block thereafter
  • It occurs on --dev, --pruning archive, --unsafe-pruning (for any number) but only on validator nodes (not on sync nodes - that part makes sense)
  • It occurs on all node accounts, including Bob and other development node accounts
  • Sometimes it is followed by the comment Offchain: could not prune: no entry in pruning map for block, but not always.
  • It does not seem to affect latency, block production, etc
  • We are running the Substrate kitchen sink master 09.26

Log Sample:

2022-10-03 10:21:57 ✨ Imported #2400 (0xbcf6…a1d1)    
2022-10-03 10:21:57 Could not canonicalize elem at pos 0 using key [12, 109, 109, 114, 179, 231, 150, 242, 116, 82, 112, 40, 61, 105, 19, 211, 47, 138, 252, 83, 183, 123, 184, 153, 192, 116, 137, 163, 141, 85, 35, 94, 41, 140, 112, 6, 0, 0, 0, 0, 0, 0, 0, 0]    
2022-10-03 10:21:57 Offchain: could not prune: no entry in pruning map for block 0    
2022-10-03 10:21:57 ♻️  Reorg on #2400,0xbcf6…a1d1 to #2400,0xf59d…0abd, common ancestor #2399,0xe757…9679    
2022-10-03 10:21:57 ✨ Imported #2400 (0xf59d…0abd)    
2022-10-03 10:21:57 Could not canonicalize elem at pos 0 using key [12, 109, 109, 114, 179, 231, 150, 242, 116, 82, 112, 40, 61, 105, 19, 211, 47, 138, 252, 83, 183, 123, 184, 153, 192, 116, 137, 163, 141, 85, 35, 94, 41, 140, 112, 6, 0, 0, 0, 0, 0, 0, 0, 0]    
2022-10-03 10:21:57 Offchain: could not prune: no entry in pruning map for block 0    
2022-10-03 10:22:00 👶 New epoch 12 launching at block 0x716f…5741 (block slot 554935640 >= start slot 554935640).    
2022-10-03 10:22:00 👶 Next epoch starts at slot 554935840    
2022-10-03 10:22:00 ✨ Imported #2401 (0x716f…5741)    
2022-10-03 10:22:00 Could not canonicalize elem at pos 1 using key [12, 109, 109, 114, 231, 162, 175, 129, 3, 105, 243, 83, 25, 239, 92, 131, 52, 160, 52, 125, 39, 24, 187, 185, 18, 0, 111, 70, 240, 16, 197, 70, 27, 241, 63, 19, 1, 0, 0, 0, 0, 0, 0, 0]    
2022-10-03 10:22:00 Could not canonicalize elem at pos 2 using key [12, 109, 109, 114, 231, 162, 175, 129, 3, 105, 243, 83, 25, 239, 92, 131, 52, 160, 52, 125, 39, 24, 187, 185, 18, 0, 111, 70, 240, 16, 197, 70, 27, 241, 63, 19, 2, 0, 0, 0, 0, 0, 0, 0]    
2
  • What kind of node are you running? Is this your custom parachain or standalone chain? If so, could you maybe share a link to the code? Commented Oct 19, 2022 at 9:33
  • We are running a standalone chain using a pull we did from this repo back in August (should correlate to 9.26): github.com/paritytech/substrate Commented Oct 19, 2022 at 13:49

2 Answers 2

3

This is coming from pallet-mmr: opened https://github.com/paritytech/substrate/issues/12531 to fix it.


Is it a problem or is it normal and can be ignored?

You can definitely ignore those error logs (my guess is you are not actually using pallet-mmr).

Has anyone found a method to resolve it?

Possible workarounds:

  • remove pallet-mmr from your runtime (if you can, and don't need it),
  • run node with extra --enable-offchain-indexing=true param,
  • filter out logs with target=runtime::mmr::offchain ("problem" is still there, but you won't see these logs anymore).
1
  • Adrian! You rock. We are implementing this now. Thanks! Commented Oct 19, 2022 at 14:23
0

Potential answer found in this issue on GitHub... https://github.com/paritytech/substrate/issues/12531

Quoting user acatangiu...

pallet-mmr

uses sp_io::offchain_index API to save full MMR leaves content to offchain db, it also defines an offchain_worker() fn that does canonicalization of data in offchain db. Issue When --enable-offchain-indexing=true is not set, (1) doesn't write anything to offchain db, while (2) keeps on spamming error logs on failures to do canonicalization.

So the answer might be simply to use the flag --enable-offchain-indexing=true when starting up the node.

We will test this today and report back.

1
  • Problem Solved by running node with extra --enable-offchain-indexing=true param Commented Oct 19, 2022 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.