Symptom → what to look at (/metrics) → remedy. All series live on the running dev/runtime at http://127.0.0.1:8288/metrics; see Metrics & footprint for the full list.

Is it me or the endpoint?

Before diagnosing anything else, ask the endpoint directly:

nuthatch doctor --rpc https://your-endpoint.example --address 0xADDR

It probes and reports the largest getLogs window that endpoint will actually serve, its batch limit, and whether it has archive history - measured against that endpoint, not read from its documentation, which is frequently wrong and often differs by tier.

Most “backfill is stuck” reports are an endpoint that lacks archive history for the deploy block, or one whose real window is a fraction of its published one. Both show up here in seconds.

Backfill seems stuck

The most common cause is high --concurrency against a single RPC endpoint - many concurrent requests to one host can stall the whole run. Use --concurrency 1 for one endpoint, or configure several rpc_urls (then 8-16 is fine). Watch nuthatch_rows_decoded_total and nuthatch_last_block climb; on a TTY, dev shows a live progress line with events/sec and an ETA, and a frozen line is the concurrency stall above.

A sparse contract over millions of blocks isn’t stuck, just inefficient - each window comes back near-empty. Widen it: --window 50000 turns tens of thousands of near-empty requests into a few. Keep the window under your provider’s getLogs block-range cap; the concurrent backfill fails a too-big range loudly rather than silently shrinking it.

Free public RPCs: stalls and empty results

nuthatch ships a small pool of free public endpoints per chain so init -> dev works with no setup. That is deliberate, and it is the right default for a first run. It is the wrong default for production, and the failure mode is worth knowing because it does not always look like an error:

• Shared rate limits. You are queueing behind everyone else on the same free tier from the same IP range, so throughput varies by the hour.
• Silent empties. A throttled endpoint often answers with an empty result instead of an error. To the indexer that is indistinguishable from “no logs in this range”, so the window advances and you simply get less data than expected.
• Deep backfills crawl or stop. Full history over a busy contract is millions of eth_getLogs calls; expect throttling long before it finishes.
• No archive guarantees. Many free endpoints prune old state, so a backfill from a 2020 deploy block can fail partway.

Symptoms: nuthatch_last_block barely moves while nuthatch_rpc_requests_total climbs; /ready reports stalled (no successful poll within the stall window); the log shows all RPC endpoints unreachable every ~60 s.

Remedy - use your own endpoint:

nuthatch dev --rpc https://your-endpoint.example/arbitrum

--rpc is repeatable, and nuthatch round-robins across the pool with per-endpoint health tracking, so two or three endpoints buy you failover as well as throughput. Persist them as rpc_urls in nuthatch.toml rather than passing them every run.

Every endpoint in a pool must be on the same chain. nuthatch verifies this at startup and refuses to run against a mixed pool - indexing against the wrong chain corrupts state silently, so this is a hard error rather than a warning.

”block N alone exceeds the provider’s getLogs result cap”

One block’s logs are too large for the provider to return, and a single block can’t be split further. Use a provider with a higher (or no) result cap. This fails loudly by design rather than looping forever.

Tip lag

nuthatch_tip_lag_blocks is the gap between the chain head and your last indexed block (nuthatch_sealed_through trails further - past finality, by design). Persistent growth means RPC throughput, not nuthatch: add endpoints or point at your own node. The adaptive getLogs window self-tunes for density.

Reorgs

Reorgs only ever touch the hot store - sealed segments are immutable, and you should never see sealed data change. nuthatch_reorgs_total counts detections; the hot store rolls back and the IVM views retract automatically, converging to canonical state. If a plan seems to require rewriting sealed Parquet to “fix” a reorg, the plan is wrong - the hot store already handled it.

/sql returns 503 or times out

• 503 “server busy” - the analytical gate is saturated (2 concurrent). It’s node self-protection: retry, don’t raise the cap.
• 30 s timeout - the query is too heavy. Add a WHERE/LIMIT, aggregate with GROUP BY, or validate cheaply with /explain first.
• Binder and parse errors come back with a fix hint derived from the real schema: an unknown table suggests the nearest real one, from/to suggests double-quoting, sum(value) suggests value_dec. Follow the hint.

RAM near the 2 GB budget

The budget is per-cursor and CI-enforced; in a runtime it’s shared across that cursor’s nests (max_rss_mb, default 2048), and a mount projected to exceed it is refused. Check the per-nest estimated footprint (and the runtime-wide actual nuthatch_rss_bytes) in the /nests roster. DuckDB queries carry their own per-query memory cap and thread limit; if you’re tight, lower query concurrency rather than the per-query cap.

”semantic.toml drift” warnings at startup

semantic.toml describes a table or column the decode registry doesn’t have - a stale edit, or the ABI changed. Fix the file or run nuthatch schema to regenerate the derived artifacts; the footguns are always recomputed, and only the authored descriptions are yours to maintain. Stale semantics are worse than none, so dev warns loudly.

ABI won’t resolve at init

init tries Sourcify, then Etherscan-class APIs. If both miss (an unverified contract), pass the ABI directly:

nuthatch init 0xAddr --abi path/to/abi.json

One --abi per address, in the same order as the addresses; leave an entry empty to resolve that one normally. Hardhat and Foundry artifacts work as-is - the ABI is unwrapped from the "abi" key.

The nest indexes nothing and reports no error

Almost always a proxy whose implementation ABI was never found. Standard proxies (EIP-1967, EIP-1822, legacy zeppelinos, and beacon proxies) are followed automatically and resolve to the implementation’s ABI. A protocol using a bespoke proxy pattern matches none of those slots, so the resolvers return the proxy’s own ABI - typically two or three administrative events - and every event you actually wanted is missing. init succeeds, the schema looks plausible, and the tables stay empty.

Since 0.7.2, init catches this itself. It samples the contract’s real logs and tells you when none of them match the ABI it resolved:

⚠ bonding (0x35Bc…): none of its last 47 log(s) match any event in the resolved ABI.
  As configured this contract will index zero rows, silently.

The fix is the implementation’s ABI - from the project’s repo, or from a block explorer’s “read as proxy” view:

nuthatch init 0xProxyAddr --abi path/to/implementation.json

For a nest you have already scaffolded, overwrite abis/<alias>.json and run nuthatch schema to regenerate.

Two things the check will not tell you, on purpose. A contract that has emitted no logs recently proves nothing either way, so it stays quiet rather than crying wolf. And one matching log is enough to clear it - a contract may legitimately emit events its ABI omits, and only the total mismatch is the silent failure.