0

We are operating a custom Solochain NPOS blockchain with Substrate. After running a stable testnet for months, we promoted to mainnet version 1.

Initially the mainnet was operating normally, with blocks being finalized regularly. However, after around 4 months of operation, validators started having problems finalizing new blocks, with the process becoming very slow.

All validators ended up being slashed due to their inability to close blocks. We have already tried updating the nodes to the latest version of Solochain, but the problem persists.

The validators appear to be synchronized on the same valid chain, so we rule out the possibility of forking. P2P connectivity was also verified and the nodes are able to communicate.

We have already reviewed logs and metrics but were unable to isolate the root cause of the problem. Has anyone experienced a similar situation or can suggest other areas for us to investigate in order to restore normal block completion? We appreciate any guidance!

Improve this question
4
  • when you say "promoted to mainnet version 1", do you mean you did a runtime upgrade? If yes, did the problems start after the upgrade?
    – dastansam
    Commented Feb 13 at 22:50
  • No, the issue occurred after block 7,344,161. When it reached that point, there were no more finalizations, and the blocks continue to be opened. At the moment I am writing this, they are at 7,349,912. All validators were slashed. What we did was stop the network, change the session keys of the nodes, etc. All the nodes we brought up are penalized.
    – Lucas Cardeal
    Commented Feb 14 at 12:08
  • I mean, did you change anything in the runtime at or before block #7,344,161? also it would help a lot if you could share your code or at least link to public RPC endpoint (to see it from the Polkadot.js explorer).
    – dastansam
    Commented Feb 14 at 13:04
  • We haven't made any changes. Our repository is: github.com/lunes-platform/lunes-nightly. The public RPC endpoint is: wss://ws.lunes.io/
    – Lucas Cardeal
    Commented Feb 14 at 13:51

1 Answer 1

Reset to default
1

I looked at your code and you have a 1s slot duration time and max 2 seconds of weight allocated for block production. 2 seconds for block production is default for solochains with 6s slot duration. Thus, if you have faster slot duration you should reduce weight allocated for block production accordingly. And I don't really know if you can have 1s slot duration in a production chain, implications of it is well described here and here.

In particular, this line seems to describe situation very similar to yours:

How exactly it behaves depends on the consensus engine that's being used, but when blocks are consistently overfull it causes a slowdown of the chain. In certain consensus engines, it might lead to a stall.

To fix it, try reducing the max available block weight to a lower value than 2 seconds, f.e, take the same ratio as with 6s slot duration, i.e third of 1s slot duration is ~0.3s (should probably be even less than this)

Improve this answer
5
  • We made the change on the testnet, but I saw in the article you sent that updating SLOT_DURATION is not so simple. Do you have any idea how we can do this safely? Is there any article with the steps? We wouldn't want to reset the mainnet and lose the data.
    – Lucas Cardeal
    Commented Feb 14 at 23:07
  • exactly, it's not really simple to change the slot duration with runtime upgrade, and I don't have any resources for this. Did you change the max available block weight?
    – dastansam
    Commented Feb 15 at 0:16
  • 1
    We were changing both the slot and the weight, but as we haven't found any information yet, we will test only updating the block size. After that, send me your wallet, and after the correction, I will send you some tokens. Thank you very much. I'll get back to you with a response if we manage to resolve it.
    – Lucas Cardeal
    Commented Feb 15 at 12:35
  • feel free to upvote or accept as an answer if the answer was useful
    – dastansam
    Commented Feb 15 at 12:47
  • We made the changes but the same problem came back. We regenerated the network, but the same problem reappeared. @dastansam do you have any idea what it could be?
    – Lucas Cardeal
    Commented Mar 8 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.