nuthatch

Roadmap

Roadmap

Nuthatch plans in public. The work is numbered as RFCs in the repository, the progress log carries one entry per push, and the load-bearing claims come with a measured number rather than a promise. This page is the map over that ground.

The reading key is plain. Shipped means merged with tests and, where it matters, a measured number. In progress means partially landed, with the remainder tracked. Planned means an accepted next step - an RFC in the repository, or one still to be written (marked "RFC to come"). Not planned means exactly that. Dates are not promised - order is.

Shipped

The embedded single-binary pipeline, end to end

  • init resolves each contract ABI (Sourcify, then an Etherscan-class API, with EIP-1967 and legacy-OZ proxy resolution) and scaffolds the nest: a schema.json, a semantic.toml seeded from the ABI, a commented views/ starter, an llms.txt, and a Claude Code skill.
  • Deterministic decode of every declared event of every contract - a topic0-keyed registry, one table per {alias}__{event}.
  • Every row carries provenance and ergonomics columns: block_hash, a deterministic _seq, block_timestamp, and *_dec/*_overflow siblings so big integers sum without hand-casting.
  • Reorg self-healing: block-hash checkpoints roll the hot store back; a proptest asserts convergence with the winning branch.
  • Finality-gated, content-addressed Parquet sealing, with the hot store pruned once every table's segment is durable.
  • Read-only analytical SQL over one surface (DuckDB): /sql unions the sealed segments with the live hot tip, and GET /tables / GET /table/{name} merge hot and cold - the full data model, queryable to the head.
  • Balance view maintained incrementally (DBSP): i128 base units serialised as decimal strings, a reorg is a retraction, and it replays from stored facts on restart.
  • Serving over HTTP and MCP: the MCP server speaks stdio, fully offline (see the agent-grade MCP surface below).

The delightful core (RFC-0015)

  • nuthatch sql - one-shot SQL from the terminal (aligned table, --json to pipe), plus an interactive REPL with .tables/.schema, history and line editing. Queries the local store when dev is stopped, and falls back to the running instance when it holds the writer.
  • Magical init - omit --chain and nuthatch probes each supported chain for your contract's bytecode and picks the one it lives on.
  • nuthatch add - grow a nest with another contract, no re-init; existing contracts resume from their cursor while the new one backfills from its deployment.
  • Live backfill feedback - a progress line while it catches up, and a clear "caught up to tip" the moment it does.
  • Production recipes - nuthatch mcp --print-config prints a copy-paste MCP config, and systemd / docker unit recipes run dev unattended.

Governed semantic layer & agent-grade MCP (RFC-0016, RFC-0017)

  • A per-nest semantic layer (semantic.toml, seeded from the ABI at init): the schema tool serves this nest's meaning - table/column descriptions, the hot/cold coverage seam, and derived footguns (reserved-word and big-int columns) - not one static string for every nest.
  • Errors as prompts - a failed query returns a one-line fix hint (nearest real table/column, reserved word, big-int) over the raw engine message; explain validates a query without executing it, so an agent self-corrects in one round-trip.
  • Result shaping + provenance - MCP results come back as compact aligned tables with truncation-as-guidance and a citable stamp (as of block N, sealed_through M, registry <hash>).
  • The full MCP surface - 12 tools, 3 resources (nuthatch://schema|tables|status), and 3 prompts (profile-contract, investigate-address, verify-a-number), all offline; nothing in the data path touched.
  • A repo-installable Claude Code skill (RFC-0017) teaches an agent to drive nuthatch - init, config, factories, runtimes, ops - with CLI/config references generated from the source and CI-checked for drift.

Nests

  • init --from <git-url | ./dir> - a nest is a git repo, so it needs no registry to share; the vendored ABIs and deployment blocks travel with it. A content-addressed registry is the optional package path when you want name@version and private nests.
  • A schema-version guard rejects a nest declaring a newer format, which is what makes consuming a third party's nest safe.
  • Authored views/*.sql (RFC-0018) are a validated, drift-gated layer: each view binds against the schema at load - so a column that moves out from under it is a loud nuthatch check failure, not a silent wrong number - and its meaning is described in semantic.toml and surfaced through /schema.
  • nuthatch check runs a nest's checks/*.sql against recorded fixtures - the generic parity and invariant framework, hermetic in CI.

Chains

  • Ethereum mainnet, Arbitrum One (chain 42161), and Base (chain 8453) - the L2s read their L1-aware finalized tag with a depth fallback, so a new chain is a registry entry, not a fork of the loop.

Backfill throughput, measured

  • nuthatch bench backfill reports the median of events per second, wall-clock, peak RSS and RPC requests; every published number traces to a report artifact.
  • Seal-direct plus pipelined fetching moved a fixed USDC workload from ≈289 to 2,521 to 5,837 events/sec on public RPC - about 20× over the redb baseline - with byte-identical segments regardless of the storage path.
  • An adaptive getLogs chunker targets ≈2,000 logs per response and self-heals into any provider's result cap, instead of a hand-tuned constant.
  • dev --seal-direct runs that 20× cold start in dev: fast-seal history to Parquet, rebuild the balance view, then hand off to the tip loop.

Operator surface

  • A Prometheus GET /metrics endpoint: tip height, tip lag, sealed-through, decode / seal / reorg / HTTP / SQL counters, and process RSS.
  • A GET /ready readiness endpoint (200 fresh, 503 when the tip poll has stalled) and an escalating log when RPC stops answering.
  • Health-aware RPC failover - a failing endpoint is benched on a cooldown so calls route to healthy ones first - and startup segment recovery that hash-verifies each sealed segment and quarantines any that is corrupt or tampered.
  • /sql query guards - a wall-clock timeout, a row cap, a result-byte ceiling, a concurrency limit, a filesystem-access denylist, and single-statement enforcement (a ;-stacked statement is refused before anything runs).
  • Graceful shutdown on SIGTERM (drains, checkpoints, resumes without gaps) and a loud warning when bound off localhost.
  • A 0.x config-stability contract, and a cargo-deny CI gate over advisories, licences, and sources.

WASM transforms

  • Pure wasm32-wasip2 components load with zero capabilities, so purity is checkable from a component's imports alone; the call boundary is a whole Arrow batch.
  • Effectful stages take per-component capability grants (kv today), with imports checked against the grant at load; their output is append-only annotations, never mutations of core entities.

Factories and dynamic discovery (RFC-0009)

  • [[templates]] and [[factories]] index the child contracts a factory deploys at runtime - tip, backfill, and reorg-safe, with a {template}__children view. It indexes Uniswap.
  • The discovered set flips from an address list to a topic0 filter at scale, so a factory with thousands of children stays a single deterministic getLogs stream.

Compliance pack (RFC-0008)

  • Sanctions screening: content-addressed list snapshots × a zero-capability WASM component seal replayable sanction_hit annotations; nuthatch screen and MCP screen_status answer "was this address flagged, against which list version?".
  • Threshold and velocity flags (per-transfer annotations plus a DBSP windowed velocity view, i128, reorg = retraction), served at /flags.
  • Direct counterparty-exposure view (DBSP) over content-addressed label snapshots, served at /exposure/{addr}.
  • Alert webhooks POST flag and hit annotations at-least-once through a durable outbox that never blocks the indexer.
  • Signed compliance-pack manifest (pack build/verify, ed25519) plus audit replay/report re-prove the sealed annotations by construction.

Admin UI and webhooks (RFC-0010)

  • A built-in admin UI at /_admin/ - embedded, no framework or CDN: status, table browser, SQL runner, and nest inspector, read-only and localhost-gated.
  • User webhooks: [[webhooks]] POST sealed rows matching a where clause to a URL through the shared at-least-once outbox; since suppresses backfill history.
  • Signed webhook egress: a webhook secret signs each delivery with X-Nuthatch-Signature: sha256=<hmac> over the exact bytes sent.

Nest packaging (RFC-0012)

  • nuthatch nest bundle packs a nest's authored inputs into a single portable, content-addressed .bundle file whose hash is a Merkle root over a canonical manifest; nest load verifies and installs one - from a local file or an http(s) URL - regenerating the decode registry from the installed inputs and asserting it matches the author's pinned hash. Determinism carried across the wire, not just the disk.
  • One runtime hosts several nests on the same chain under a single cursor, one hot store, and one finality view, with strict per-nest isolation and a per-cursor RSS budget: a mount whose projection exceeds the budget is refused before it starts. Verified live on Arbitrum. (2.0 retired the separate roost command - it is nuthatch dev, and the directory decides what runs.)

Nest registry (RFC-0019)

  • nest publish <bundle> --registry <path|s3://…> --as name@version writes a bundle to a content-addressed store; nest load name@version --registry … resolves, fetches, and installs it - the pulled blob hash-verified by the same install path, so a registry pull is as safe as a hash-pinned file load. A filesystem path is a registry; an S3-compatible bucket (MinIO/S3/R2) is the fleet path, behind --features object-store.
  • Private nests live behind your bucket's own auth (AWS_* env, including AWS_ENDPOINT); a missing or rejected credential fails with a clear "this nest is private", never a bare "not found". Self-hosted-first: the registry is decoupled and never mandatory - a self-built bundle and load <file|dir> touch no registry at all.

Safe upgrades - no resync tax (RFC-0020)

  • nuthatch nest diff <old> <new> classifies a nest update: compatible (additive only - a new table or column) or breaking (a consumer-observable change - removed, renamed, or retyped). nuthatch nest upgrade --to <new> then handles either kind. A compatible update is hot-swapped with zero downtime - it serves the old version, indexes the new one concurrently, and atomically flips the endpoint the moment the new one catches up, so the served address never changes. A breaking update instead serves the new version on a new endpoint (/next) alongside the old - which keeps working, now carrying a Deprecation header - so downstream migrate on their own clock before the old is sunset. Either way, updating a nest stops being a subgraph-style genesis resync. And when a compatible update's decode is unchanged, the new version mounts the old's sealed content-addressed segments instead of re-indexing history at all - a true no-re-index upgrade subgraphs structurally can't do.

Multichain roost (RFC-0021, in progress)

  • A runtime can host nests across more than one chain in a single runtime - [[chains]] in mounts.toml, and one isolated cursor per chain (its own tip, finality, and reorg boundary): a Base nest and an Arbitrum nest served from one process. The footprint budget is per active-chain cursor (≤2 GB), and each cursor is its own observable boundary, held per chain. A capability, not a mandate: one chain per runtime stays the simple default and is byte-identical to running the nest solo. The single-cursor law holds per chain - two chains are never multiplexed behind one cursor.

Derive-first - the eth_call you don't need (RFC-0023, in progress)

  • The Foundation reports that >70% of subgraphs call eth_call - but most of those reads are derivable from the events a nest already indexes; subgraphs fetch them only because they have no incremental-view engine. Nuthatch does. nuthatch recipe add total_supply drops in a derived view that computes an ERC-20's totalSupply() as Σ minted − Σ burned from Transfer events - deterministic, free, and with no archive node. It derives what a subgraph pays an archive node to fetch. The ERC-20-generic trio - total_supply, balances, and holder_count - all derive from Transfer events, and reserves derives Uniswap-V2 getReserves() as the latest Sync per pair - the AMM read a subgraph fetches on every swap. For the handful of reads that aren't derivable but never change - decimals/symbol/name - nuthatch metadata fetch calls once and caches forever. For the genuinely non-derivable residue, a verifiable eth_call fallback follows; its local-execution engine (a demand-driven state cache, not an archive node) is designed but deferred, so the derive-first path stays fully self-hosted with no external data dependency.

Fault quarantine - a runtime survives its sick nests (RFC-0026)

  • A faulting nest is quarantined rather than fatal: its healthy siblings keep indexing and serving, and it is re-admitted on a backoff if the fault was retryable. A terminal fault stays quarantined until an operator fixes it. The blast radius is now bounded in both directions - a nest's error no longer kills its cursor, and a cursor's death no longer kills the runtime.
  • The health surface to see it with: GET /nests merges each nest's live health per request, GET /ready answers runtime-wide while GET /<name>/ready answers per nest, and nuthatch_nest_health / nuthatch_nest_quarantine_total / nuthatch_cursor_live cover the metrics. --fail-fast restores exit-on-first-fault for CI and for operators who prefer a loud death to partial service.

The live roost (RFC-0027)

  • A runtime's nest set used to be frozen at boot, so onboarding one tenant's nest restarted every co-tenant's - the blast radius of a config change was larger than that of a fault. Now POST /_admin/nests mounts one and DELETE /_admin/nests/<name> unmounts one, live. A mount is admitted only if it fits the cursor's RAM budget (refused with 507, never a warning - a budget that can be quietly exceeded is not a budget), catches up before it joins so it never drags co-tenants back through history, and only then gets routes. An unmount is a drain: the cursor finishes its window and releases the store before anything is torn down.

Scaffold a nest from a subgraph (RFC-0019, #241)

  • nuthatch init --from-subgraph <CID> maps a subgraph manifest's dataSources to [[contracts]] and its templates to [[templates]], vendors every ABI from the CID the manifest pins, and carries startBlock and each dataSource's eventHandlers across - so a subgraph that handled only Transfer does not silently grow a table for every other event. The pinned ABIs beat Sourcify on proxy-heavy codebases, where Sourcify returns the proxy ABI that declares none of the events the contract emits. Factory rules are inferred only where a parameter unambiguously names a template; anything ambiguous is reported with candidates rather than guessed, because a wrong guess is a nest that quietly indexes the wrong children. Our first external contribution, from @PaulieB14.

The backfill that finishes, then flies (RFC-0028, RFC-0029)

  • Found by running someone else's benchmark: Sentio's OBIB. Case 1 - Transfer from LBTC across 22.2M Ethereum blocks - did not merely run slowly, it never finished: Alchemy returns its oversized-range refusal as HTTP 400, a status the classifier did not enumerate, so a window that needed splitting was retried unchanged forever. It now completes in 74.8 s for 294,278 events, in 321 RPC requests and 320 MB - matching the record count in Sentio's own README exactly.
  • The rest was round-trip economics rather than compute: ~85% of the wall clock was buying block_timestamp, one serial fan-out inside every window, for a column that workload never stores. Timestamps are now demand-driven - a nest that does not ask does not pay - and the log window adapts, widening while an endpoint keeps up and narrowing the instant it does not, so a provider's undocumented limits are discovered rather than configured.
  • RPC failures are also classified properly now: an auth rejection stops being retried forever, an unclassifiable wide-range failure splits speculatively rather than stalling, and a 429 escalates pool-wide.

Scaled mode - a fleet (RFC-0022)

  • One operator across many machines: a control plane holding desired state, a writer pool taking one lease per chain cursor with the fence enforced inside the store's own write transaction, and a query-FE tier scaled independently of ingestion. A plan is not permission - a scheduler that wants to rehome a live cursor is refused by the lease, which is correct rather than a bug.
  • Workers pull the nests they are assigned from a registry (RFC-0019), because the machine the scheduler picks may have nothing on disk. With a bundle_hash pinned the fetch is by content address and the registry's mutable index is never consulted, so re-tagging a version cannot change what a fleet runs.
  • Verified across real machines, not several processes on one box: a genuine lease handover with the fence advancing, a 10-minute clock jump moving expiry rather than ownership, and 377 blocks indexed through a 90-second control-plane outage - losing the control plane stops rescheduling, not ingestion.
  • And it did not write for a whole release. Until 0.9.3 worker registered, took leases, loaded secrets and reported - and contained no indexing code at all. Ten level-5 checks passed throughout, because every one of them tested the control plane and none asserted that a row appeared. We would rather you read that here than discover it.

Release and footprint

  • v2.0.0 is the current release (tagged, with prebuilt binaries and container images on ghcr). Semantic versioning applies to the 2.x line: a minor is 2.1, 2.2, and breaking changes wait for 3.0. MSRV is 1.95 - measured, not asserted.
  • 2.0 is the one release that moves data on disk, and it ships the command that does it. nuthatch migrate moves files and rewrites a config; it never re-indexes. Measured on a real two-nest deployment: 0.259s. Tenancy moved into the runtime and the roost was retired - not renamed, removed - so there is one nuthatch dev and the directory decides what runs. Nest data is keyed by content address, so two tenants mounting the same nest share one dataset and it is never indexed twice. See the upgrade notes.
  • Every prior line, for context: 0.6 brought portable nest bundles, the nest registry, zero-downtime N-1 upgrades, multichain runtimes, derive-first recipes and adaptive MCP tools; 0.7-0.8 the live mount/unmount surface and fault quarantine; 0.9 scaled mode's writer pool and the backfill work below; 1.0 the stability commitment itself. 0.9.3 is a security patch - every earlier release is vulnerable to an arbitrary file read on /sql, because DuckDB accepts a quoted function name and the guard only matched an unquoted one. 0.6.2 was also a security patch, for ;-stacked statements, which was an arbitrary file-write primitive. If you expose /sql to anyone you do not trust, upgrade. cargo install --git builds on rustc 1.95 or newer.
  • A CI footprint job fails the build above 256 MB of RAM.
  • Measured peak: ≈37 MB for a single contract, ≈58 MB across a three-contract, 23-table nest - 2.8% of the 2 GB budget.

In progress

  • The Lodestar migration (RFC-0011): the pilot is live in production - two of a public dashboard's panels serve from a single nuthatch binary instead of The Graph gateway, each parity-checked before cutover (the story is here). The published horizon-nest carries the Horizon staking/service surface; the migration is parked after the proven pilot, with the remaining panels (Indexer Directory, Curation, Epochs) folded into horizon-nest as they're needed rather than as a second nest.
  • reth ExEx wiring (RFC-0003): the Source trait, the library split, and the push-to-pull bridge are in and tested, and rustc 1.95 satisfies both reth and dbsp - the toolchain and dependency blockers are cleared. Wiring the ExEx needs a synced node.

Planned

  • RFC-0005 - the remaining release engineering: signed artifacts, a Homebrew tap, and an OCI image. The releases and the one-line installer are out; signing and the extra channels are what harden the trust path.
  • DataFusion federation (RFC-0013) - one Arrow-native engine federating hot and cold behind a single SQL surface. The benchmark gate was run, and DataFusion did not meet it: 1.6-2.7× DuckDB's latency on the fold that matters, with the gap widening as segments grow, at exact result parity. So DuckDB stays in both modes. The destination is unmet, not repudiated - measure-then-switch working as designed, and the measurement said don't.
  • reth ExEx tip mode (RFC-0003) - in-process tip-following against a colocated node, and then firehose-class extraction of traces and state diffs (RFC-0014), both gated on wiring the ExEx to a synced node.
  • A GraphQL compatibility layer (RFC to come) - for subgraph-shaped clients.
  • IVM generalisation (RFC to come) - lift more derived views from DuckDB SQL over sealed data into incrementally-maintained views.
  • Effectful HTTP egress for transform worlds (RFC to come) - a wasi:http sandboxed variant beyond today's kv capability grant.
  • Wildcard, signature-scoped indexing (RFC to come) - index an event by its signature across every emitter.

Not planned

  • A token or decentralized-network features.
  • Telemetry of any kind.
  • A Kubernetes-first deployment.
  • No gateway, auth, or metering layer in the core - those stay outside it, so independent operators can run Nuthatch behind their own.

These are design decisions, not gaps - the project's scope rules explain why.

The progress log in the README is the change-log of record, and it moves faster than this page. Where this page and the log disagree, trust the log - and read the RFCs for the reasoning behind each line.