Intermitent errors with jsonrpsee: WS error: i/o error: Transport endpoint is not connected (os error 107); terminate connection 1056

Question

We've been getting some intermitent websocket connection issues on our production polkadot UI node since the upgrade to jsonrpsee.

We are running a single Polkadot 0.9.24 container image on our K8s cluster but this has occurred with an unexpected low number of connections and this only appears to have manifested itself since the upgrade.

The error message in the polkadot logs is:

WS error: i/o error: Transport endpoint is not connected (os error 107); and terminate connection 1056

the message in the k8s node is:

upstream prematurely closed connection while reading response header from upstream

Questions are:

1. Do we need to configure connection limits for websockets now? In anycase how would we do that if we did want to do that - so that we can test?

2. What is the default maximum number of connections that a single ui node can handle simultaneously?

Answer 1

1. It depends on your usecase and the load on your RPC node.
2. The default number of connections is 100 but it is configurable via the CLI flag --ws-max-connections.

If you reach the maximum number of connections you should see:

"Too many connections. Try again in a while"

upstream prematurely closed connection while reading response header from upstream

That is most likely something else related to the K8s cluster/jsonrpsee but would be good to investigate what is the cause of that as it happens "sometimes".