i'd like to monitor the number of connections each node is handling on its ws, rpc and p2p ports with a view to right-sizing infrastructure capacity to exceed the typical load by a learned safe margin.

has anyone figured out what the prometheus rules might look like?

i've identified these metrics that i believe contain the right information:

# HELP substrate_rpc_requests_finished Number of RPC requests (not calls) processed by the server.
# TYPE substrate_rpc_requests_finished counter
- substrate_rpc_requests_finished{protocol="http",chain="calamari"} 1580264
- substrate_rpc_requests_finished{protocol="ws",chain="calamari"} 167445457

# HELP substrate_rpc_requests_started Number of RPC requests (not calls) received by the server.
# TYPE substrate_rpc_requests_started counter
- substrate_rpc_requests_started{protocol="http",chain="calamari"} 1580264
- substrate_rpc_requests_started{protocol="ws",chain="calamari"} 167445473

# HELP substrate_rpc_sessions_closed Number of persistent RPC sessions closed
# TYPE substrate_rpc_sessions_closed counter
- substrate_rpc_sessions_closed{chain="calamari"} 1104752

# HELP substrate_rpc_sessions_opened Number of persistent RPC sessions opened
# TYPE substrate_rpc_sessions_opened counter
- substrate_rpc_sessions_opened{chain="calamari"} 1104821

# HELP substrate_sub_libp2p_connections_closed_total Total number of connections closed, by direction and reason
# TYPE substrate_sub_libp2p_connections_closed_total counter
- substrate_sub_libp2p_connections_closed_total{direction="out",reason="keep-alive-timeout",chain="calamari"} 5154
- substrate_sub_libp2p_connections_closed_total{direction="out",reason="ping-timeout",chain="calamari"} 55
- substrate_sub_libp2p_connections_closed_total{direction="out",reason="transport-error",chain="calamari"} 11846

# HELP substrate_sub_libp2p_connections_opened_total Total number of connections opened by direction
# TYPE substrate_sub_libp2p_connections_opened_total counter
- substrate_sub_libp2p_connections_opened_total{direction="out",chain="calamari"} 17093

# HELP substrate_sub_libp2p_distinct_peers_connections_closed_total Total number of connections closed with distinct peers
# TYPE substrate_sub_libp2p_distinct_peers_connections_closed_total counter
- substrate_sub_libp2p_distinct_peers_connections_closed_total{chain="calamari"} 17055

# HELP substrate_sub_libp2p_distinct_peers_connections_opened_total Total number of connections opened with distinct peers
# TYPE substrate_sub_libp2p_distinct_peers_connections_opened_total counter
- substrate_sub_libp2p_distinct_peers_connections_opened_total{chain="calamari"} 17093

# HELP substrate_sub_libp2p_incoming_connections_total Total number of incoming connections on the listening sockets
# TYPE substrate_sub_libp2p_incoming_connections_total counter
- substrate_sub_libp2p_incoming_connections_total{chain="calamari"} 0

🥷 my statistics and promql foo falls something short of ninja and i don't really understand how to write the expression i need. take the following promql queries, for example:

it's not apparent to me what these queries are telling me. i think substrate_rpc_requests_finished{protocol="ws"}[1m] is telling me that some nodes are handling more than 100 million ws requests per minute. these numbers surprise me. have i misinterpreted them? i don't understand what wrapping the query in rate() does. the docs suggest it's the request increase rate over the previous period. if so, it's not what i need as i'm interested in volume rather than increase.

what i really want to know is:

2 Answers 2


i did not find a good metric exposed by substrate-node with which ws connections can be monitored. i did find that prometheus node-exporter provides a good enough metric in node_netstat_Tcp_CurrEstab which monitors the total number of established tcp connections being handled by the node host/instance. while this metric includes more than just the ws connections, in practice, the resolution is good enough to provide the information required on a host dedicated to running a single substrate node, since ws accounts for pretty much all of the tcp connections beyond the first 100 or so.

node-exporter can be installed like so:


# download
curl -L \
  -o /tmp/node_exporter-${version}.linux-amd64.tar.gz \

# extract
sudo tar xvfz \
  /tmp/node_exporter-${version}.linux-amd64.tar.gz \
  -C /usr/local/bin \
  --strip-components=1 \

# clean up
rm /tmp/node_exporter-${version}.linux-amd64.tar.gz

it can be run as a systemd service by creating a unit file at


Description=prometheus node exporter



and then enabled and started with:

sudo systemctl enable --now prometheus-node-exporter.service

when its metrics url is added to prometheus server configuration, the tcp metrics can be visualised, monitored and alerted on. tcp connections


The rate function is actually what you need, increase is required in the context of a time serie db like Prometheus with a metric only increasing.

What makes you think your nodes are handling million or requests per second? The scale I see on your graph is 0-25 which makes sense.

Here is an example of this query rate(substrate_rpc_requests_started{job="astar-public-node"}[$__rate_interval]): enter image description here

You can consider stacking series to have an overall usage of the network: enter image description here

EDIT: rate will only give you average number of requests per second. For a different time scale to estimate whole load, better use increase instead:


enter image description here

For open connections, this makes it easily:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.