4

I'm particularly interested in the role of backups and state stored on disk in the recovery process. In terms of preparations, we are early enough in our project lifecycle to put in place whatever measures are pertinent in case the worst should happen.

In terms of a partial failure, my understanding is that if at least two boot nodes remain running, then the other nodes can come back up and will re-sync. Is this correct? Is it beneficial for the recovery process of a node to be able to re-attach the existing storage device?

Although very unlikely with proper redundancy, what would be the recommended recovery strategy if all the parachain nodes terminated? For example, is it enough to simply reboot the nodes with their attached storage devices and let the software handle the rest? Is it possible that the state on disk will be left in a bad state by a sudden node crash, mandating a restore from a stable point? Etc etc.

1 Answer 1

2

Even though most of the answers depend on the exact configuration of the systems being run. I feel I can give you a few pointers here.

  • Backups / snapshots.

They are the key to ensure access to the state of your chain at any point. They come handy when interested in running some tests from state at block B. In case of corruption of dbs you know you can reover. If you make them available to the community, they will have resources to sync faster.

  • Some nodes were stopped

If there are still nodes running that means that any other node can, through discovery, connect to them and sync your chain. Also, your chain will be available as long as there is at least one node publicly accessible running, that's on theory, on practice as you mention it is more like two nodes running as collators are actually being run.

  • All nodes stopped.

Well, this means that your chain is not available at any level. You can still spin up nodes again and, if dbs have not been removed, everything shall continue as soon as nodes are back running. From the top of my head I won't mention anything else on this topic.

  • Corrupted DB

The reasons of why a db can get corrupted are difficult to list, it might be that for one system loosing power unexpectedly leaves the db in a bad state, but maybe for a different system the OS / drive controllers are smart enough to avoid this scenario. Running out of space in your disk can be a problem, but I cannot promise you that hitting this will corrupt your dbs always, though I have seen it happening.

At least, these are two points I would say to pay attention to and avoid finding your nodes into.

But what I can confirm is that if your db gets corrupted you will have no option but drop that db and sync your node again, for what backups are really helpful.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.