what's a good value for `--ws-max-connections`?

Question

calamari parachain uses an aws dns latency based load balancer for the ws.calamari.systems dns entry used by polkadot.js for websocket interactions with calamari. there are multiple, geographically diverse nodes which will respond to the same wss://ws.calamari.systems url, depending on where the client connection comes from.

a recent increase in traffic to that endpoint resulted in nodes rejecting many ws connections because they were configured with the default --ws-max-connections 100 setting. increasing this value to 1000 for all nodes, immediately resolved the problem.

has anyone tested the upper boundaries of what this value can be set to? are there good examples of a prometheus rule that could indicate that a node has more ws connections than it can safely handle? what system properties can be monitored whilst experimenting with higher values for ws-max-connections that would give an indication of overloading?

i should add that the ws connections are reverse proxied through nginx so that they are served over ssl. i guess that nginx may have it's own limitations but i don't know if they would be hit before substrate's were. most of the nodes run on instance types like r5d.xlarge or r5ad.2xlarge. perhaps the only limitations that matter, relate to the instance type and/or hardware...

Answer 1

i learned a lot in the year since asking this question. some of the things i learned were difficult to discover, but the implementation of the various configuration tweaks is easy to accomplish if you know what to look for.

tl;dr:

you can set ws-max-connections to a value of 512 if your os ulimit is 1024 without making any other changes to a typical ubuntu server setup.

if you want more from your hardware and have a higher volume of inbound connections, read on...

using higher values (like 16000):

here's my findings, whilst running substrate nodes on typical bare metal servers with ubuntu 22.04 server, nginx reverse proxy (for tls), 32gb ram, a modern 8+ core cpu and a 1+ gbps network connection (the numeric values below are the ones we use for calamari and work well with the hardware example above):

• nginx and the www-data user it runs under, need to have configuration for the number of connections required and permission to open enough file handles to manage those connections:

  • configure the number of allowable active connections in the events section of the nginx config. eg:

    /etc/nginx/nginx.conf

    events {
      worker_connections 32768;
    }
    

  • configure the number of allowable open file handles to double the worker_connections value. nginx requires one for the incoming connection and another for the forward. eg:

    /etc/nginx/modules-enabled/worker_rlimit_nofile.conf

    worker_rlimit_nofile 65536;
    

  • grant the nginx service account an adequate soft limit for open file handles (check the current setting with: sudo -H -u www-data sh -c 'ulimit -nS', verify the hard limit is higher than your new soft limit: sudo -H -u www-data sh -c 'ulimit -nH'). eg:

    /etc/security/limits.d/www-data-soft-nofile.conf

    www-data soft nofile 65536
    

• the substrate node and the user it runs under, need to have configuration for the number of connections required and permission to open enough file handles to manage those connections:

  • configure the maximum number of websocket connections (ws-max-connections) in the node startup arguments. also set max-runtime-instances (i don't yet understand the significance of this value but 8 is default, 256 is max. 8 is not high enough for more than ws-max-connections=1000). eg:

    /etc/systemd/system/substrate.service

    ...
    ExecStart=/usr/local/bin/substrate \
    ...
      --ws-max-connections 16000 \
      --max-runtime-instances 256 \
    ...
    User=substrate
    Group=substrate
    ...
    

  • grant the substrate service account an adequate soft limit for open file handles (check the current setting with: sudo -H -u substrate sh -c 'ulimit -nS', verify the hard limit is higher than your new soft limit: sudo -H -u substrate sh -c 'ulimit -nH'). eg:

    /etc/security/limits.d/substrate-soft-nofile.conf

    substrate soft nofile 65536
    

Answer 2

I can't really provide a specific number but we also increased it compared to the default value.

There number of ws connections to allow is highly dependent of what request your users are doing, as most of the resources won't be used for the connection itself but for the request going through.

Additionally, you can also start to read about the new RPC library being included in Substrate: https://github.com/paritytech/jsonrpsee

This library allows to "reserve" resources (cpu, memory, ...) for each type of request to better control the load on the RPC server.