how can substrate prometheus metrics be used to monitor and alert on the number of concurrent connections?

Question

i'd like to monitor the number of connections each node is handling on its ws, rpc and p2p ports with a view to right-sizing infrastructure capacity to exceed the typical load by a learned safe margin.

has anyone figured out what the prometheus rules might look like?

i've identified these metrics that i believe contain the right information:

# HELP substrate_rpc_requests_finished Number of RPC requests (not calls) processed by the server.
# TYPE substrate_rpc_requests_finished counter
- substrate_rpc_requests_finished{protocol="http",chain="calamari"} 1580264
- substrate_rpc_requests_finished{protocol="ws",chain="calamari"} 167445457

# HELP substrate_rpc_requests_started Number of RPC requests (not calls) received by the server.
# TYPE substrate_rpc_requests_started counter
- substrate_rpc_requests_started{protocol="http",chain="calamari"} 1580264
- substrate_rpc_requests_started{protocol="ws",chain="calamari"} 167445473

# HELP substrate_rpc_sessions_closed Number of persistent RPC sessions closed
# TYPE substrate_rpc_sessions_closed counter
- substrate_rpc_sessions_closed{chain="calamari"} 1104752

# HELP substrate_rpc_sessions_opened Number of persistent RPC sessions opened
# TYPE substrate_rpc_sessions_opened counter
- substrate_rpc_sessions_opened{chain="calamari"} 1104821

# HELP substrate_sub_libp2p_connections_closed_total Total number of connections closed, by direction and reason
# TYPE substrate_sub_libp2p_connections_closed_total counter
- substrate_sub_libp2p_connections_closed_total{direction="out",reason="keep-alive-timeout",chain="calamari"} 5154
- substrate_sub_libp2p_connections_closed_total{direction="out",reason="ping-timeout",chain="calamari"} 55
- substrate_sub_libp2p_connections_closed_total{direction="out",reason="transport-error",chain="calamari"} 11846

# HELP substrate_sub_libp2p_connections_opened_total Total number of connections opened by direction
# TYPE substrate_sub_libp2p_connections_opened_total counter
- substrate_sub_libp2p_connections_opened_total{direction="out",chain="calamari"} 17093

# HELP substrate_sub_libp2p_distinct_peers_connections_closed_total Total number of connections closed with distinct peers
# TYPE substrate_sub_libp2p_distinct_peers_connections_closed_total counter
- substrate_sub_libp2p_distinct_peers_connections_closed_total{chain="calamari"} 17055

# HELP substrate_sub_libp2p_distinct_peers_connections_opened_total Total number of connections opened with distinct peers
# TYPE substrate_sub_libp2p_distinct_peers_connections_opened_total counter
- substrate_sub_libp2p_distinct_peers_connections_opened_total{chain="calamari"} 17093

# HELP substrate_sub_libp2p_incoming_connections_total Total number of incoming connections on the listening sockets
# TYPE substrate_sub_libp2p_incoming_connections_total counter
- substrate_sub_libp2p_incoming_connections_total{chain="calamari"} 0

🥷 my statistics and promql foo falls something short of ninja and i don't really understand how to write the expression i need. take the following promql queries, for example:

• substrate rpc requests finished

  substrate_rpc_requests_finished{protocol="ws"}[1m]
  

  

• rate of substrate rpc requests finished

  rate(substrate_rpc_requests_finished{protocol="ws"}[1m])
  

  

it's not apparent to me what these queries are telling me. i think substrate_rpc_requests_finished{protocol="ws"}[1m] is telling me that some nodes are handling more than 100 million ws requests per minute. these numbers surprise me. have i misinterpreted them? i don't understand what wrapping the query in rate() does. the docs suggest it's the request increase rate over the previous period. if so, it's not what i need as i'm interested in volume rather than increase.

what i really want to know is:

• what query expression would tell me that a node is approaching --ws-max-connections (see also: what's a good value for `--ws-max-connections`?)?
• what other query expressions could tell me about rpc (http) and p2p connection volumes?

Answer 1

i did not find a good metric exposed by substrate-node with which ws connections can be monitored. i did find that prometheus node-exporter provides a good enough metric in node_netstat_Tcp_CurrEstab which monitors the total number of established tcp connections being handled by the node host/instance. while this metric includes more than just the ws connections, in practice, the resolution is good enough to provide the information required on a host dedicated to running a single substrate node, since ws accounts for pretty much all of the tcp connections beyond the first 100 or so.

node-exporter can be installed like so:

version=1.3.1

# download
curl -L \
  -o /tmp/node_exporter-${version}.linux-amd64.tar.gz \
  https://github.com/prometheus/node_exporter/releases/download/v${version}/node_exporter-${version}.linux-amd64.tar.gz

# extract
sudo tar xvfz \
  /tmp/node_exporter-${version}.linux-amd64.tar.gz \
  -C /usr/local/bin \
  --strip-components=1 \
  node_exporter-${version}.linux-amd64/node_exporter

# clean up
rm /tmp/node_exporter-${version}.linux-amd64.tar.gz

it can be run as a systemd service by creating a unit file at

/etc/systemd/system/prometheus-node-exporter.service:

[Unit]
Description=prometheus node exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

and then enabled and started with:

sudo systemctl enable --now prometheus-node-exporter.service

when its metrics url is added to prometheus server configuration, the tcp metrics can be visualised, monitored and alerted on.

Answer 2

The rate function is actually what you need, increase is required in the context of a time serie db like Prometheus with a metric only increasing.

What makes you think your nodes are handling million or requests per second? The scale I see on your graph is 0-25 which makes sense.

Here is an example of this query rate(substrate_rpc_requests_started{job="astar-public-node"}[$__rate_interval]):

You can consider stacking series to have an overall usage of the network:

EDIT: rate will only give you average number of requests per second. For a different time scale to estimate whole load, better use increase instead:

increase(substrate_rpc_requests_started{}[1m])

For open connections, this makes it easily:

substrate_rpc_sessions_opened{}-substrate_rpc_sessions_closed{}