how can collator block production be monitored?

Question

most substrate documentation recommends monitoring node health using substrate's built in prometheus metrics publisher. i've started to implement this but haven't been able to find good examples of validating that a collator node is producing blocks. there are many examples of using the up metric to validate that a node is alive and moonbeam documentation mentions the substrate_proposer_block_constructed_count metric but not how one might use it in a prometheus rule expression to validate that a node's block production count is within some meaningful thresholds. an example of how to write an expression that evaluated if a collators block production was within a certain threshold of average block production for all collators within a given timeframe would be exceptionally useful. do such examples exist or do you know enough about prometheus metric calculations to provide such an example here? my current set of rules is rather generic and looks like this:

---
groups:
-
  name: substrate node
  rules:
  -
    alert: instance down
    expr: up == 0
    for: 1m
    annotations:
      summary: >-
        instance down: $labels.instance
      description: |
        [{{ $labels.instance }}] of job [{{ $labels.job }}] has been down for more than 1 minute.
    labels:
      severity: critical
  -
    alert: high cpu load
    expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 80
    for: 0m
    annotations:
      summary: >-
        high cpu load: $labels.instance
      description: |
        cpu load greater than 80%
        value: {{ $value }}
        labels: {{ $labels }}
    labels:
      severity: warning
  -
    alert: low peer count
    expr: substrate_sync_peers < 10
    for: 0m
    annotations:
      summary: >-
        low peer count: $labels.instance
      description: |
        peer count less than 10
        value: {{ $value }}
        labels: {{ $labels }}
    labels:
      severity: warning

Answer 1

Monitoring a collator node block production rate can be done in the same way as any other Substrate node.

You can find examples of Substrate Prometheus alerting rules in the Substrate Github repo under .maintain/monitoring/alerting-rule.

  - alert: BlockProductionSlow
    annotations:
      message: 'Best block on instance {{ $labels.instance }} increases by
      less than 1 per minute for more than 3 minutes.'
    expr: increase(substrate_block_height{status="best"}[1m]) < 1
    for: 3m
    labels:
      severity: warning
  - alert: BlockProductionSlow
    annotations:
      message: 'Best block on instance {{ $labels.instance }} increases by
      less than 1 per minute for more than 10 minutes.'
    expr: increase(substrate_block_height{status="best"}[1m]) < 1
    for: 10m
    labels:
      severity: critical