Since I upgraded my 3 KSM nodes :
- to 0.9.21 on May 11 and
- 0.9.22 on June 1
my era points dropped almost to zero.
My Polkadot node was also upgraded on May 11 but rewards were not affected until June 16th when we noticed that rewards were almost gone.
The KSM node uses the same IP, but the ports are different.
I know how era points are distributed among validators but I cannot seem to pinpoint which of those actions my nodes are not performing correctly hence missing rewards.
I had various different errors in my nodes which I list below :
=FetchPoV(NetworkError(Network(Timeout)))which looks like it means that my node is timing out when validating parachain blocks, hence decreased rewards. However, I do not know what I can do to solve it.
The bootnode you want to connect provided a different peer ID than the one you expect:which is also mentioned in this issue So, as they suggest, I verified that all bootnodes have the peer ID as defined in the chainspec. However, I did not see any change in my era points.
The only down time that my nodes had were when we shut some of them down for a while to restart them.
I also checked the specs of my hardware. My machine is running on Kubernetes with CPU limit of 3 and memory limit of 8. We have three machines and 16 nodes. I compared with the recommended hardware specs mentioned in this SE question/answers which points to another related answer here and it looks like our specs are insufficient. So I will also try to update the specs of our 3 machines.
Other errors from the logs are the following :
failed to associate send_message response to the senderbut not sure how relevant is this one.
Incoming substream exceeding maximum number of negotiating inbound streams. Dropping.
I also checked my session keys in Polkadot-JS Apps as well (
session > nextKeys(AccountId32)) to see if the return value is consistent with the Session keys I have set. For the validator accountIDs that I checked, it showed 6 keys in p-js apps.
However, I saw that in some nodes I have 5 session keys and in others I have 6 keys. I checked the keystore because based on this wiki page it can also affect the availability.
I also checked one of the metrics shown in subscan and there is one called "Grandpa Vote”. It is a metric for checking validators grandpa online rate and this rate is also much lower than usual.
So, I suppose that the problem is multiple things since I have so many different errors. Even though I tried different things, it is still not clear to me what is the root cause and what are the necessary steps that I need to do to solve the issue of my decreased era points. Can you recommend a clear list of possible reasons and solutions to my problem ?