Why a Nuthatch nest has two identities
A content-addressed package should be immutable. That does not mean a documentation edit ought to fetch the whole chain again.
Nuthatch nests are content-addressed. That is the right default for something that declares how on-chain data is decoded and exposed: a deployment can name exactly the contracts, ABIs, event selection, views, and schema it was built from. Change the authored package, and it is a different package.
There is an obvious operational cost, though. If every changed byte made a nest forget its existing history, editing a view comment would mean a complete re-index. Correctness has won, but rather expensively.
The design is two identities, each answering a different question:
authored nest package ──hashes to──> NID ──mount points to──> data/<nid>/
│
└── data identity decides whether an
existing dataset may be adoptedThe NID is deliberately strict. The data identity is deliberately narrower. Neither is an approximation of the other.
The NID: an immutable package identity
A nest’s manifest records every authored input, with a SHA-256 for each file, in a canonical order. Its NID is a domain-separated SHA-256 over that canonical manifest. The package includes the configuration, pinned ABIs, SQL views, schema, semantic material, documentation, and the rest of the authored surface. It does not include indexed data or generated build artefacts.
Consequently, every authored edit produces a new NID. A new ABI, a changed event selection, a new view, or a README correction all make a distinct package. This is not a version label that can be silently retargeted. An operator can always say which package a dataset belongs to.
One detail matters for upgrades. The NID neutralises the Nuthatch binary’s own generator version. Otherwise, a binary upgrade would give every nest a new identity and orphan every dataset even when the generated decoder was identical. The decoder registry hash remains part of the manifest, and is regenerated and verified on load. A binary which actually changes decoding therefore still changes the NID, as it should.
Mounts turn package identity into a running service
The runtime’s mounts.toml maps a (tenant, alias) to an NID. The dataset is stored under
data/<nid>/; it does not know its friendly name or the tenant that mounted it.
That gives a useful property without introducing a hosted multi-tenant service. Two aliases, or two tenants, mounting the same NID share one dataset, one backfill, and one contribution to the cursor. Removing one mount does not remove the data while another still refers to it. The tenant is an opaque ownership label. Authentication, quotas, billing, and rate limiting remain the gateway’s job.
A runtime can host nests across several chains, but each chain has its own cursor, finality view, reorg boundary, and writer. There is no cross-chain multiplexing concealed behind a cheerful config file. That way lies weather.
The data identity: can this package use existing history?
The NID answers, “is this exactly the same package?” It is intentionally too broad to answer, “would this package store the same bytes?”
For that, Nuthatch computes a separate data identity. It includes the decode registry and every
authored input that can affect stored data. It excludes only inputs which the indexing path cannot
read: views, queries.toml, semantic.toml, documentation, and scaffolded agent material.
The safety rule is simple:
- A new NID never replaces an old one.
- An existing dataset is adoptable only when its data identity and its decoder registry hash both match the incoming nest.
- The source must contain readable indexed history.
- An ambiguous or failed match falls back to an ordinary backfill.
This errs on the useful side of conservative. A false negative costs time and RPC requests. A false positive serves somebody else’s history as though it were yours, which is not a feature.
When an incoming nest is adoptable, Nuthatch copies the old derived state into the new
data/<nid>/ directory using a staging directory and an atomic rename. It then overlays the incoming
package’s own authored files. The old dataset is never moved, since it may still be mounted by
another tenant or still be the version an operator is serving. If the copy fails, the destination is
left input-only and can safely try again next time.
This is what makes a view-only, query-allowlist, semantic, or documentation change cheap. The NID moves because the package genuinely changed. The stored history is adopted because its inputs did not.
Reuse means two different things
There are two reuse mechanisms, and they are worth keeping separate.
Whole-dataset adoption
At mount or migration time, the runtime looks across existing datasets for one whose data identity and decoder registry match the incoming nest. This is not confined to a package’s direct predecessor: another nest in the runtime can supply the history if it meets the same exact test.
It is not semantic query matching. Nuthatch does not see two similar entity names and make an optimistic guess. It reuses a whole dataset only when the authored data-producing inputs prove that the stored data is the same.
Shared sealed segments
Finalised history is stored as immutable, content-addressed Parquet segments. In a multi-nest
runtime, they live in a shared segments/<content-hash>.parquet store. A dataset manifest references
the segment hash; it does not need to own another copy of the bytes.
Two nests that independently produce byte-identical segments therefore use one file on disk. This is more granular than dataset adoption and is also simpler: the segment hash is the proof. No SQL equivalence engine, no entity-name comparison, and no hopeful interpretation of what a query might mean.
The mutable hot store remains per dataset. It is affected by reorgs and has a writer, so sharing it would be an impressive way to turn one nest’s trouble into everyone else’s. Sealed Parquet is immutable, so it is the proper sharing boundary.
Updating a nest
The normal runtime update is staged and inspected first:
nuthatch migrate --dir <runtime> --dry-run
nuthatch migrate --dir <runtime>The plan tells the operator whether it will keep an existing identity, merge duplicate mounts, adopt an existing dataset, or create a new one and index it. A new schema is also compared against the currently served schema.
An additive table or column is compatible. Removing or renaming a table or column, or changing a
column’s type, is breaking. The migration refuses a breaking change by default and names the affected
surface. The operator can explicitly accept it with --allow-breaking, or mount the successor under
another alias and move consumers on their own schedule.
That compatibility check is intentionally a schema-surface check. It cannot prove that an arbitrary change to a view preserves the meaning of its results while retaining the same columns. We treat that as an operator and review concern rather than pretending a schema diff has solved program equivalence. There are already enough haunted shortcuts in this line of work.
What grafting is today, and what it is not yet
The word “grafting” can suggest a sophisticated per-view cache. That is the long-term direction, but it would be misleading to claim it is already the current cost centre.
Authored SQL views are presently defined at query time over the hot store and sealed segments. They are not materialised. Editing one has never required recomputing stored view data, because no such data exists. The historical problem was that the edit changed the package NID, and the old identity-keyed layout would consequently re-fetch the raw chain data.
Whole-dataset adoption fixes that current problem. Per-derivation DAG reuse, where a changed stored derivation causes only its dependent derivations to recompute, remains deferred until Nuthatch has materialised derivations to reuse. It is better to state that boundary now than manufacture a cache for data that does not yet exist.
Immutable packages, practical operations
Content addressing is often presented as a purity exercise. The useful version is more mundane. It lets an operator retain an exact, reproducible definition of every nest version, while the runtime reuses history whenever it can prove that the bytes are unchanged.
The NID keeps versions honest. The data identity keeps harmless edits cheap. Shared immutable segments remove duplicate cold storage. The remaining work is deliberately conservative, because a re-index is inconvenient and a false reuse is silent.
That is the trade. It is not glamorous, but it gives an indexer a proper memory rather than a series of increasingly confident guesses.