2

calamari parachain uses an aws dns latency based load balancer for the ws.calamari.systems dns entry used by polkadot.js for websocket interactions with calamari. there are multiple, geographically diverse nodes which will respond to the same wss://ws.calamari.systems url, depending on where the client connection comes from.

a recent increase in traffic to that endpoint resulted in nodes rejecting many ws connections because they were configured with the default --ws-max-connections 100 setting. increasing this value to 1000 for all nodes, immediately resolved the problem.

has anyone tested the upper boundaries of what this value can be set to? are there good examples of a prometheus rule that could indicate that a node has more ws connections than it can safely handle? what system properties can be monitored whilst experimenting with higher values for ws-max-connections that would give an indication of overloading?

i should add that the ws connections are reverse proxied through nginx so that they are served over ssl. i guess that nginx may have it's own limitations but i don't know if they would be hit before substrate's were. most of the nodes run on instance types like r5d.xlarge or r5ad.2xlarge. perhaps the only limitations that matter, relate to the instance type and/or hardware...

1 Answer 1

3

I can't really provide a specific number but we also increased it compared to the default value.

There number of ws connections to allow is highly dependent of what request your users are doing, as most of the resources won't be used for the connection itself but for the request going through.

Additionally, you can also start to read about the new RPC library being included in Substrate: https://github.com/paritytech/jsonrpsee

This library allows to "reserve" resources (cpu, memory, ...) for each type of request to better control the load on the RPC server.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.