calamari parachain uses an aws dns latency based load balancer for the ws.calamari.systems dns entry used by polkadot.js for websocket interactions with calamari. there are multiple, geographically diverse nodes which will respond to the same
wss://ws.calamari.systems url, depending on where the client connection comes from.
a recent increase in traffic to that endpoint resulted in nodes rejecting many ws connections because they were configured with the default
--ws-max-connections 100 setting. increasing this value to
1000 for all nodes, immediately resolved the problem.
has anyone tested the upper boundaries of what this value can be set to? are there good examples of a prometheus rule that could indicate that a node has more ws connections than it can safely handle? what system properties can be monitored whilst experimenting with higher values for ws-max-connections that would give an indication of overloading?
i should add that the ws connections are reverse proxied through nginx so that they are served over ssl. i guess that nginx may have it's own limitations but i don't know if they would be hit before substrate's were. most of the nodes run on instance types like r5d.xlarge or r5ad.2xlarge. perhaps the only limitations that matter, relate to the instance type and/or hardware...