Metrics & footprint

Every running nest exposes Prometheus text at GET /metrics. Gauges are set to the latest value; counters only ever increase - standard scrape targets, no exporter sidecar needed.

The series

Process-level gauges:

Process-level counters:

With many nests in one runtime, the height and poll gauges above (nuthatch_tip_height, nuthatch_last_block, nuthatch_last_poll_unixtime) are solo-runtime only and do not describe any one nest; alert on the labelled per-nest series instead: nuthatch_nest_last_block, nuthatch_nest_sealed_through, nuthatch_nest_rows_decoded_total, nuthatch_nest_rows_sealed_total, and nuthatch_nest_reorgs_total.

Storage per nest, since 3.13.0: nuthatch_nest_hot_store_bytes is that nest’s own hot store, and nuthatch_nest_sealed_segments_bytes the sealed segments its manifest names. Segments are content-addressed and shared, so a segment two datasets both contain counts under both nests and the sum over nests can exceed the disk used; the unlabelled nuthatch_sealed_segments_bytes counts each store once. Before 3.13.0 that total counted a runtime’s shared store once per nest. A mount sharing another’s dataset has no storage series of its own. This is what a host meters by; see Host nests for others.

Health series (RFC-0026), which is where you look when part of a runtime is unwell:

Each declared authored incremental entity adds seven gauges, labelled by nest and entity: nuthatch_entity_applied_through, nuthatch_entity_current, nuthatch_entity_rows, nuthatch_entity_faulted, nuthatch_entity_unavailable, nuthatch_entity_seconds_since_progress, and nuthatch_entity_state_bytes. Page on faulted; investigate a current gauge that stays zero or a progress age that keeps climbing. This is the difference between a maintained answer that is fresh and one that merely happens to have a table-shaped name.

A nest started with --publish-target adds the mirror’s series, labelled by nest:

Alert on nuthatch_publish_lag_blocks staying above zero. Before 4.11.0 it was the seal watermark less the mirror’s highest block, which read millions on a quiet nest whose mirror was complete.

The footprint budget

The budget is a non-negotiable, CI-enforced: ≤2 GB RAM per active-chain cursor - one chain’s tip-following plus serving, whether that cursor hosts one nest or twelve. A single-chain runtime is one cursor (≤2 GB total, shared across its nests); a multichain runtime’s total is the sum of its cursors. Density is RAM-bounded, not free: a runtime refuses to mount a nest whose projected footprint would blow the ceiling (max_rss_mb, default 2048).

Inside the budget, the analytical path is separately bounded - each analytical session runs under its own memory cap and thread limit, and the concurrency gate bounds the aggregate. If you’re tight, lower concurrency rather than raising the per-query cap.

A small, useful alert set

These are deliberately symptoms rather than a pager for every counter. Pick the lag threshold for the chain and the service promise you make to readers. The examples assume Prometheus has a job="nuthatch" label.

groups:
  - name: nuthatch
    rules:
      - alert: NuthatchCursorStalled
        # Three poll intervals, as /ready counts a stall: 900 s at the default 5-minute poll.
        # Scale it down for a nest run with a shorter --poll-interval.
        expr: time() - nuthatch_last_poll_unixtime{job="nuthatch"} > 900
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "Nuthatch has not polled its chain for fifteen minutes"

      - alert: NuthatchFallingBehind
        expr: nuthatch_tip_lag_blocks{job="nuthatch"} > 50
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "Nuthatch remains more than 50 blocks behind the tip"

      - alert: NuthatchNestQuarantined
        expr: nuthatch_nest_health{job="nuthatch"} == 0
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "A mounted nest is quarantined"

      - alert: NuthatchNearMemoryBudget
        expr: nuthatch_rss_bytes{job="nuthatch"} > 1800 * 1024 * 1024
        for: 10m
        labels: { severity: ticket }
        annotations:
          summary: "Nuthatch is within 248 MB of its default cursor budget"

The memory number is intentionally below the default 2 GB ceiling. It gives you time to reduce query concurrency or investigate a mount before the runtime has to refuse more work. In a multichain runtime, alert per nuthatch_cursor_live{chain} and per-nest health too; the aggregate RSS cannot identify which chain has become expensive. See Troubleshooting for the corresponding diagnosis paths.

Checked against Nuthatch 5.8.2. Verify a release-specific command with the CLI reference before running it in production.