3

I am updating the Substrate node from v2.0.1 to v3.0.0 in live network*.

The initial plan was to merge the v3 code by following the official guidelines, build the new wasm and use a forkless runtime upgrade to deliver it to all nodes in the network. But with this approach the following error appears on nodes: WARN Cannot create a runtime: Instantiation(“Instantiation: Export ext_default_child_storage_storage_kill_version_2 not found”). After research, I found that clients should be updated first.

I've started experiments around the clients' updates. I've up a cluster with 7 validator nodes on v2.0.1 and started 1 by 1 update clients to v3.0.0. The result is the following:

  1. Updated Node-1. The network is producing and finalizing blocks. Node-1 is not finalizing blocks but sometimes producing them.
  2. Updated Node-2. The same behaviour as after the Node-1 update.
  3. Updated Node-3. The network is producing blocks. The network is not finalizing blocks. Node-3 is sometimes producing blocks. The reason for the finalization stop looks clear - less than 2/3 of nodes are finalizing blocks.
  4. Updated Node-4. The same behaviour as after the Node-3 update.
  5. Updated Node-5. The network is producing blocks. Network finalization recovered and now is working as expected.
  6. Delivered the v3.0.0 with a forkless upgrade. It applied successfully to the nodes with updated clients (1-5).

Questions:

  1. Is it the correct way to update the clients in the live network*?
  2. Is it any way to avoid stopping finalization in the live network* during clients update?

* A live network is a network with 50+ validators.

Improve this question

1 Answer 1

Reset to default
1
  1. The way of updating clients is no other than just using the new version, and for that you then find yourself in the situation where a machine has to stop providing service while the client is updated. There is no way a client can be updated without stopping it for a short period of time.
  2. For avoiding finalization stalling, as you notice on your post, you have to avoid 2/3 of your validators being down at the very same time. We can consider 50 nodes "small" as 2/3 of them is not such a big number, and even if you have control over all them, what is likely to happen is that you update the clients in batches, or maybe even all of them at once. So even if you are using some kind of orchestration. So you can find yourself in this situation if not taking care of always having at least 2/3 of them up.

For that I would recommend, in case you control all nodes in your network, grouping them in fairly small groups (~ 10 nodes each for 50 validators), and proceed with updated by groups.

Improve this answer
2
  • Do you assume that 'grouping in small groups' will avoid block finalisation issues? Is it the case that updated client is backward compatible and will finalise blocks with the rest of the not updated network?
    – Raid Ateir
    Commented Jul 21, 2022 at 15:23
  • Yes, I think that would be the assumption here.
    – Alejandro - R 0 G U E
    Commented Jul 21, 2022 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.