What might be the reasons that my validator nodes are receiving significant less era points at the moment?

Question

Since I upgraded my 3 KSM nodes :

• to 0.9.21 on May 11 and
• 0.9.22 on June 1

my era points dropped almost to zero.

My Polkadot node was also upgraded on May 11 but rewards were not affected until June 16th when we noticed that rewards were almost gone.

The KSM node uses the same IP, but the ports are different.

I know how era points are distributed among validators but I cannot seem to pinpoint which of those actions my nodes are not performing correctly hence missing rewards.

I had various different errors in my nodes which I list below :

• =FetchPoV(NetworkError(Network(Timeout))) which looks like it means that my node is timing out when validating parachain blocks, hence decreased rewards. However, I do not know what I can do to solve it.

• The bootnode you want to connect provided a different peer ID than the one you expect: which is also mentioned in this issue So, as they suggest, I verified that all bootnodes have the peer ID as defined in the chainspec. However, I did not see any change in my era points.

• The only down time that my nodes had were when we shut some of them down for a while to restart them.

• I also checked the specs of my hardware. My machine is running on Kubernetes with CPU limit of 3 and memory limit of 8. We have three machines and 16 nodes. I compared with the recommended hardware specs mentioned in this SE question/answers which points to another related answer here and it looks like our specs are insufficient. So I will also try to update the specs of our 3 machines.

• Other errors from the logs are the following :

  • failed to associate send_message response to the sender but not sure how relevant is this one.
  • Incoming substream exceeding maximum number of negotiating inbound streams. Dropping.

• I also checked my session keys in Polkadot-JS Apps as well (Developer tab > Chain State: session > nextKeys(AccountId32)) to see if the return value is consistent with the Session keys I have set. For the validator accountIDs that I checked, it showed 6 keys in p-js apps.

• However, I saw that in some nodes I have 5 session keys and in others I have 6 keys. I checked the keystore because based on this wiki page it can also affect the availability.

• I also checked one of the metrics shown in subscan and there is one called "Grandpa Vote”. It is a metric for checking validators grandpa online rate and this rate is also much lower than usual.

So, I suppose that the problem is multiple things since I have so many different errors. Even though I tried different things, it is still not clear to me what is the root cause and what are the necessary steps that I need to do to solve the issue of my decreased era points. Can you recommend a clear list of possible reasons and solutions to my problem ?

Answer 1

For rewards, most points are gotten from validating parachains - if your nodes aren't properly set up to validate parachains, you will be missing out on a lot of rewards.

There's a handful of requirements for parachains:

• Each node must have all the session keys required for parachains (which should be 6 keys)
• Each node must have at least the single threaded performance of the Standard Hardware machines (which at the moment is is an Intel 7700k)
• Each node must use/have disk speeds on par with NVME Drives
• Each node must have network traffic open for ingress and egress on the p2p port (which by default is 30333)
• Each node must have network bandwidth of at least 300-500 mbps

Running in Kubernetes is not really recommended, unless you really know what you're doing and understand the requirements. Ideally each node is hosted on a bare metal machine, with only 1 node per machine. Running in Kubernetes will cut a lot of performance, especially if you do not tune the resources right.

You can see if a machine has enough power by running:

./polkadot benchmark machine

as it's sort of described here

This will output a green checkmark or not if the cpu, disk and memory is on par with the Standard Hardware that is required for Parachains.

It looks like in general the loss of rewards for you is based on your setup - you can roughly see how different infrastructure providers and regions compare from this snapshot of missed parachain votes.

Implicit and Explicit votes are pretty much the same and are good. Missed votes are not good, and result in not getting rewards.

I would migrate your nodes to bare metal machines, with 1 node per machine.