The book

1. The data problem

An Ethereum node can tell you what happened in a block. It is rather less interested in helping you answer a product question such as “which addresses delegated to this indexer last week?” That answer requires turning a stream of logs into a durable model, keeping it current as new blocks arrive, and making it cheap enough to ask repeatedly. This is the ordinary work of an indexer.

The usual answer is a hosted subgraph. That is often an entirely sensible answer. A schema and its mappings turn chain activity into a GraphQL endpoint, somebody else runs the infrastructure, and the application obtains a pleasant query surface. The awkward moment is when that endpoint is unreachable, unserved, behind the chain, or contains a capability that cannot be reproduced from the chain alone.

Nuthatch starts from a smaller promise. A nest describes a chosen set of contracts, pinned ABIs, events and SQL views. It reads logs from the chain, decodes those logs deterministically, and serves the resulting event data through SQL and HTTP. There is no allocation, indexing market or hosted control plane between the RPC endpoint and the query. A nest is something a team can keep beside the application that depends on it.

The useful boundary

A transaction log is public chain history. Given the same block range, contract addresses and ABI, two honest implementations should decode the same rows. This makes event indexing a good substrate for a portable local index. It also makes it possible to verify the result independently rather than taking an API’s word for it.

Not all subgraph data has that property. A mapping may make eth_calls whose result depends on the block, fetch IPFS content, call an external service, or maintain bespoke off-chain state. Those are not mistakes. They are simply dependencies that lie outside the event stream. Pretending that a log index automatically offers parity with every such mapping is the beginning of a fairly tedious outage.

Two of those dependencies Nuthatch now handles, and it is worth being precise about how, because the manner matters more than the fact. A [[calls]] declaration reads a contract at a fixed block, and the result is addressed by the question that produced it: chain, block, contract, calldata. An [[ipfs]] declaration resolves a document and verifies the bytes against the address that names them. Neither is the ambient, unrecorded fetch that makes a mapping hard to re-execute. The property this chapter cares about - that two honest implementations agree, and that you may check rather than trust - is preserved in both cases, because a pinned question has one answer and a content address describes exactly one document.

What remains outside is what always was: an external service, a value that depends on when you asked, bespoke off-chain state. Those are still dependencies rather than data, and naming them is still the first honest step in a port.

The first question in a port is therefore not “can Nuthatch replace this subgraph?” It is “which reads are event-derived, and what does each remaining read depend on?” Many useful reads turn out to be straightforward: transfers, swaps, delegation changes, registration records and cumulative balances. Some current-looking values can be derived from ordered events, such as ERC-20 supply from mints and burns. Others genuinely require a contract read or external input. Name the boundary before promising a fallback.

A modest but strong promise

This bounded scope gives Nuthatch several useful properties:

• The input is inspectable. A nest vendors its ABI and says which events it accepts.
• The output is repeatable. Decoding is a pure transformation of log plus authored decode rules.
• The operator owns availability. An application may query its own machine or an API it controls.
• Historical data can be sealed and retained without keeping a full node or a bespoke database cluster alive forever.

The machine does not know the business meaning of an event. It knows how to preserve and present it without quietly changing its mind. The semantic layer belongs in views and application code, where it can be reviewed as such.

A running example

Imagine a protocol dashboard that normally asks a GraphQL subgraph for recent delegations. Its critical screen needs the delegator, indexer, timestamp and amount. Those values originate in a specific event. A nest can watch the protocol contracts, decode that event into a table and expose a view with precisely those columns. The dashboard retains its normal endpoint, but gains a small and independently operated fallback for the read it cannot afford to lose.

This is not an ideological replacement programme. It is a good operational shape: use the rich surface you have, and keep the irreplaceable event data somewhere you control. The practical subgraph fallback guide walks through that exercise.

The next question is how that small description remains trustworthy after it has left your laptop. That is the work of the authored nest.

2. The authored nest

Before Nuthatch indexes a single block, somebody must make several decisions: which chain matters, which contracts matter, which ABI is authoritative, which event signatures should be decoded, and which derived reads readers may need. These choices are the nest. The database files, generated schema and decoded rows are consequences of that description, not the nest’s essence.

That distinction is easy to wave away until an ABI changes, a directory is copied, or a deployment must be reproduced on another machine. Then the authoring inputs are precisely the thing you need to preserve.

The authored inputs

A normal nest directory contains nuthatch.toml, vendored files under abis/, and optionally SQL views, semantic descriptions and checks. The TOML names contracts and their start blocks. The ABI pins the event layout used to decode their logs. A view may turn raw rows into a consumer-shaped read without modifying the underlying historical facts.

The ABI is deliberately vendored. An explorer API is useful when scaffolding a nest, but it is not an adequate long-term dependency for its definition. An explorer can change, a proxy can mislead, and a fetched ABI may not be the ABI that was intended at the time. Keeping the source artefact in the nest makes review and reproduction possible.

From those files Nuthatch generates a decode registry and a schema. The registry maps the address and event signature it receives from a log to the columns it will write. The generated artefacts are checked rather than treated as private magic. If the same authored inputs do not recreate the expected registry, the machine should stop and say so. A stale decoder producing plausible rows is not a success condition.

Events first, then views

The raw event table is the durable base layer. It records chain coordinates such as block number, transaction hash and log index alongside decoded event fields. Those coordinates are not clutter. They establish order, support audit, and give a reader a route back to the originating chain fact.

Views sit above this base layer. They are ordinary SQL declarations authored with the nest. A view can make transfers friendly to query, calculate a balance from the ordered transfer stream, or present a protocol-specific activity table. Because it is a view rather than rewritten history, a consumer can inspect the derivation and the raw events remain available when the definition needs to change.

This is an important division of labour. A decoder answers “what did this log say according to this pinned ABI?” A view answers “what result do we want from those rows?” Conflating the two makes schema upgrades unnecessarily dangerous.

Content identity

Nuthatch packages authored inputs into a canonical manifest and hashes that manifest with SHA-256. The resulting nest identity, or NID, identifies what was authored, not which directory happens to contain it and not who mounted it. Two copies of the same inputs have the same identity. A one-byte change in an ABI, configuration or view creates a new identity.

That is deliberately strict. The hash is not a version label chosen at a meeting. It is a statement that this exact package is what the runtime verified. Human names and versions remain useful for navigation, but the NID is the thing a machine can use to decide whether two claimed nests are identical.

Identity does not mean every NID requires a fresh backfill. Sometimes the package has changed in a way that leaves its event data byte-identical. How Nuthatch recognises and safely adopts that data is a later chapter. For now the point is simpler: the system begins by making the definition of a nest small, explicit and reproducible.

For the exact file layout and configuration, see authoring modes and the configuration reference. Next, the nest meets the chain.

3. The cursor

An indexer has two jobs which look similar from a distance and behave very differently in practice. It must collect old history, often millions of blocks, and it must then follow a live chain whose latest blocks are not yet final. Nuthatch calls the durable position that coordinates this work a cursor.

There is one cursor per chain in a runtime. Not per HTTP route, not per tenant and not per nest. That is a useful constraint, not a missing feature. One chain has one ordered tip, one finality boundary and one reorganisation story. Giving every nest an independent opinion about those things would multiply RPC work and make recovery needlessly inconsistent.

Backfill is a controlled walk through history

For a newly mounted nest, Nuthatch begins at each contract’s declared deployment block and asks the RPC for logs in bounded windows. Providers impose limits on ranges, result counts and concurrency, so the process does not assume that a heroic getLogs call will be welcome. It splits work into windows, retries within its policy and records progress only after the rows have been accepted by the hot store.

The details matter because RPCs are prone to giving an answer that is technically valid and operationally useless. A provider may time out, cap a response, or make a 10,000-block request feel like a personal insult. Nuthatch’s window and concurrency controls let an operator fit the walk to the provider. Faster is useful, but only if every accepted block remains attributable and repeatable.

Backfill catches a nest up to the present. It does not establish that the present is permanent.

The tip is provisional

At the chain tip, a block can be replaced by a competing block. This is a reorganisation. A reader who saw an event in the first branch must not be left with that event after the chain selects the other branch. Nuthatch therefore keeps unfinalised data in its hot store, with enough block-hash checkpoints to notice when the chain no longer agrees with the path it had followed.

When a mismatch is detected, the cursor finds the fork point and each affected hot store rolls back to that point. It then indexes forward along the winning branch. The invariant is not “we never briefly served a provisional result”. No honest near-tip system can promise that. The invariant is “provisional rows are marked by their place in a reversible part of the pipeline, and the store converges to the canonical chain.”

Finality is what ends that reversible period. Once a block sits sufficiently behind the head under the configured chain policy, Nuthatch can seal it. Sealed history is no longer subject to ordinary tip rollback and becomes the cold, durable half of the query surface.

One chain, one source of ordering

In a multi-nest runtime the cursor obtains the union of needed logs and routes them to the nests that own their address and event signature. A log may be relevant to more than one nest, in which case each gets its own decoded rows. Fetching is shared; the nests’ datasets are not silently merged. This distinction keeps ownership and rollback manageable while avoiding N copies of the same RPC polling.

Different chains need different cursors. They have different heads, different finality rules and different failure domains. A runtime can host them, but it does not pretend that Arbitrum and Ethereum form one sequence merely because they have both inconvenienced the same operator.

The cursor is the reason an index can say where it is. The next chapter is about where the data sits once it has passed through that cursor, and why it lives in two forms.

4. Storage and sealing

Chain data has an awkward temporal property. The newest blocks must remain reversible because a reorganisation may replace them. Old blocks ought to be cheap to retain and scan for years. Trying to satisfy both needs with one storage shape usually produces a database that is rather busy doing neither particularly well.

Nuthatch separates the two deliberately. The hot store holds the near-tip, reversible working set. Finalised history is sealed into immutable Parquet segments. The query layer joins the two so a reader asks one question without having to know whether the answer happened ten minutes or ten months ago.

Hot data is for change

The hot store receives decoded rows while the cursor backfills and follows the tip. It supports the operations a live index needs: inserting new event rows, recording checkpoints, rolling back a reorganisation and deleting a range that belonged to the discarded branch. It is per dataset, which gives a nest a clear write boundary and prevents an error in one package from rewriting another package’s working set.

The hot store is not a second-class cache. Before a block has crossed the finality boundary, it is the authoritative record of what the cursor currently believes the chain says. Its mutability is a feature. Treating tip data as immutable merely moves the eventual correction into an application bug, where it is more expensive and less visible.

Sealing is a commitment

When rows are final enough, Nuthatch writes them to Parquet segments and records them in a seal manifest. A segment is immutable content with a bounded block range, table identity and hash. The manifest describes the set of segments that make up the sealed historical view and is itself part of the evidence an operator can inspect.

The operation is not “move some old rows to a different folder and hope for the best”. The system writes staged output, verifies it and only then makes the new sealed state visible. If a process dies at an inconvenient moment, the previous committed manifest remains coherent. Recovery may be boring, which is the highest compliment available to storage machinery.

Because segments are immutable, they can be shared safely where their data identity matches. This does not mean every nest shares every table. It means the runtime can avoid retaining identical sealed history twice while preserving each dataset’s package and query surface. The distinction becomes important during upgrades.

One query surface

DuckDB reads sealed Parquet efficiently and can also query the current hot rows. Nuthatch builds a read-only SQL surface over the union. A query for a block range that straddles the finality boundary does not require the caller to issue two requests or reconcile duplicate rows. The storage boundary is an implementation detail, albeit one worth understanding when diagnosing performance.

There are guards around this freedom. Queries have concurrency, time, row and unsealed-row bounds. Those guards protect the node from an enthusiastic analytical query becoming a denial-of-service tool. They do not provide customer identity, quotas or billing. Those belong at the gateway, where there is actually an authenticated caller to reason about.

The full life of an event

Take one Transfer log. The cursor sees it in a block, selects the ABI decoder and writes a row with its chain coordinates into the hot store. A reader can now query it, knowing it remains inside the reorg window. After finality, the sealer writes the row into a Parquet segment, commits the updated manifest and removes the corresponding hot copy. The SQL view continues to return it, because it reads hot plus sealed history as one logical table.

The row changes physical home exactly once under normal operation. Its meaning does not change at all. That is the point of keeping decoding separate from storage and of treating finality as a first-class boundary.

For operational details, see storage and sealing and reorgs and finality. We can now turn to the thing readers actually use: the index’s query surface.

5. Reading the index

An index is useful only when somebody can ask it a question. Nuthatch exposes decoded event data as tables and makes that data available through read-only HTTP and SQL surfaces. The system is intentionally more ordinary than a bespoke query language. SQL is well understood, inspectable and capable of expressing both a simple point lookup and a careful event-derived calculation.

Start from the raw event

Every selected event has a table, including an event which the contract has not emitted yet. In that case the table is present and empty, rather than appearing only on the day the first log arrives. The table contains decoded event fields and chain coordinates: block number, transaction hash, log index, emitting address and related identity information. These columns let a consumer order simultaneous events, trace a result back to a transaction and decide how to display the distance from finality.

Raw tables are the audit trail. They may not be the API a product wants to hand to a browser, and that is quite all right. They are the stable base upon which a product-specific interface can be built. When a view looks suspicious, the raw log-derived rows provide the way to check it.

Views give event data a useful shape

An authored SQL view is part of the nest package. It can rename columns, select a narrow consumer surface, join related event tables and derive incremental-looking answers from the event history. For example, an ERC-20 total supply can be calculated from mints and burns, and a latest Uniswap V2 reserve can be selected from the latest Sync event per pair. These are not opaque mapping code. They are reviewable SQL over a known, pinned input.

This is a strong capability, but it has limits. A view is only as sound as the event model beneath it. A historical event stream cannot reproduce a state variable that was changed without an event, or content that a subgraph fetched from IPFS. An eth_call at a particular block may be necessary for some questions.

Where such a read is necessary, it is declared rather than performed quietly: a [[calls]] or [[ipfs]] block in the nest’s configuration, pinned to a block or checked against a content address, and stored in a table of its own that a reader can see and interrogate like any other. The distinction this section is drawing survives intact. Nuthatch makes the event-derived part dependable, and it does not perform other forms of data acquisition behind the reader’s back - it performs them in front of the reader, or not at all.

Serving safely

The SQL endpoint is read-only and bounded. It has a concurrency semaphore, a request timeout, a row cap and an upper limit on the unsealed rows a scan may include. A public deployment should use named queries or an allowlist when it does not intend to offer general SQL. The runtime enforces these local resource bounds, while a gateway in front supplies authentication, rate limits and tenant policy.

The HTTP API offers conventional endpoints for discovery and tables as well as /sql. Admin routes are a distinct surface and may mutate runtime state, such as mounting or unmounting a dataset. It is important not to describe the whole server as “read-only” merely because its data API is. The administrator can make changes, so the administrative listener needs the corresponding protection.

MCP and semantic descriptions provide more guided access for agents and tools. They are not a second data model. They describe and query the same underlying tables and views.

A good consumer contract

For a dashboard or service, define a small query contract: the rows, ordering, finality behaviour and error policy it actually needs. Put the contract in an authored view or named query. Test it at a fixed watermark. Then an application can switch from a degraded primary endpoint to the nest without improvising SQL during the incident, which is a period when even ordinary punctuation can become a tactical challenge.

The SQL reference, HTTP API and authored views guide provide the exact surfaces. The next chapter explains how this data can be retained and reused even as its surrounding nest package evolves.

6. Identity, upgrades and reuse

Software versioning is often an agreement among humans: this release is called 2.5.0, and these are the changes we say it contains. That remains valuable, but it is not sufficient for deciding whether two indexer packages decode the same data. Nuthatch uses content identity for that question.

The nest identity, or NID, is the SHA-256 hash of the canonical authored manifest. Change a contract selection, pinned ABI, event configuration or authored view, and the NID changes. The runtime stores data by that identity under data/<nid>/; a mount maps an operator-facing alias and tenant to it. An alias may change without changing data. Two tenants may mount the same NID without starting two indexers. The name is for people. The NID is for evidence.

Package identity is not data identity

The useful subtlety is that two different nest packages can produce exactly the same decoded data. Perhaps the only change is a view, a semantic description, a comment-like metadata field or some other authored input that does not affect event selection or decoding. The package has honestly changed, so it must have a new NID. But forcing a complete backfill merely because a view changed would be an expensive ceremony with no new information in it.

Nuthatch calculates a data identity from the inputs that actually determine decoded event rows. On migration or staged upgrade, it can compare this identity with existing datasets. If the data identity matches, it adopts the existing data into the new identity instead of indexing the chain again. The adoption is a staged, verified filesystem operation. The existing source remains; the new dataset gains the necessary package material and references the reusable sealed history.

This is not a claim that every update is free. If an ABI, event selection, contract range or decode rule changes, the decoded dataset may differ. That is a real data change. The runtime classifies the difference and requires an operator to acknowledge breaking changes rather than presenting an old table with a new label and hoping no one notices.

The guarantee has to hold on the ordinary path too

That last sentence describes an intention, and for a while it was only true where somebody had gone looking for it. A packaged nest records its expected registry hash, and mounting one regenerates the registry and verifies it. A nest run the ordinary way, from a directory on your own machine, did not compare anything.

So adding an event to a running nest’s configuration and restarting produced this: the process started, observed it was already at the chain tip, indexed nothing, and served - stamping the new registry hash onto rows the old configuration had produced. Every query then reported provenance under a decode registry that had never run. The only visible symptom was an unrelated view failing to load, and only because that view happened to reference one of the new tables; a change touching tables no view mentioned said nothing at all.

It is worth being precise about why that is worse than missing rows. A content address is a claim about what produced this data. A wrong one is not an inconvenience, it is a lie in the one field whose entire job is to be checkable. And it was a lie the system told about itself, in the direction that looks healthy.

A nest now compares the registry recorded in its store against the registry its configuration produces, and refuses to start when they differ, naming both hashes and the remedy. A store written before the check existed has no recorded hash; refusing those would break every running deployment for a fault it may not have, so it adopts the hash and logs that it was recorded rather than verified - which is a different claim, and says so.

Reuse at two levels

There are two distinct wins:

1. Exact package equality: two mounts with the same NID share one dataset immediately. This is what happens when two tenants mount the same nest.
2. Data equality across package versions: distinct NIDs can adopt data that has the same data identity. The packages remain distinct, but the indexer avoids useless re-ingestion.

Sealed segments are immutable content, so they are the natural part of history to share. The hot store remains dataset-local because it must participate in live writes and reversible reorg handling. This keeps a new package from accidentally inheriting a mutable working store it does not fully understand.

Grafting, honestly

“Grafting” is sometimes used loosely to mean any form of reuse. In the current runtime, adoption is whole-dataset reuse where data identity proves compatibility. Views are query-time definitions, not independently materialised histories that can each be grafted in a different way. There is no promise today that a changed view receives a special partial backfill or that arbitrary schemas can be spliced together. Those might be useful future capabilities, but a book should say what the machine does now.

The nuthatch migrate command plans the move, can dry-run it, and refuses named breaking changes unless the operator explicitly permits them. It is designed to be idempotent. A migration that discovers it needs to re-index data has failed its central job.

The detailed, operational companion is nest identity and reuse and upgrading a nest. Once packages and datasets have these clean boundaries, a single process can host many of them without becoming a muddle of names.

7. The runtime

A nest is a portable description plus its dataset. A runtime is the process that makes one or more such datasets live. This is where Nuthatch separates the questions people often tie together by habit: what a package is, what an operator calls it, who has mounted it, which chain is followed, and what query surface is exposed.

The runtime reads mounts.toml. A mount records an alias, tenant, NID and SQL access policy. The alias is the route a caller sees. The tenant is an opaque operator label. The NID selects the identity-keyed dataset. Because those are separate, two tenants can mount one NID and share its data while exposing different aliases or named-query allowlists.

This is multi-tenancy in the useful, modest sense: multiple independently named datasets in one runtime. It is not an identity service, quota system or billing platform. Those require knowing who the caller is, and the runtime does not. Put a gateway in front when the deployment needs that policy. The node protects its own memory and query capacity regardless of who is asking.

One cursor per chain

Within one chain, a runtime uses a shared cursor. The cursor sees the chain once, obtains the union of relevant logs and routes each log to the mounted nests that need it. This reduces duplicate RPC polling and means that finality and reorg detection have one coherent boundary. Each dataset still has isolated hot storage, package files and query context.

The rule is per chain. A multichain runtime owns a separate cursor for each chain because chains do not share a head, a finality rule or a failure mode. The runtime groups mounts by chain before spawning the cursor work. It does not make a multi-chain deployment magically one ordered stream.

The effect is worth being precise about. A runaway factory or failed decoder should quarantine the affected nest rather than corrupt its neighbour’s data. A dead cursor must not hide behind healthy HTTP routes. And an operator needs metrics that distinguish a process-level number from per-nest and per-chain health.

Capacity is a physical constraint

Nuthatch has a default 2 GB resident-set budget per active-chain cursor. It estimates the impact of a mount before accepting it and exposes the actual footprint through metrics. The number is not a marketing density claim. A runtime with two chains has two cursors and therefore two separately bounded workloads. A high-rate nest can still be expensive even if it has very few neighbours.

The shared cursor pays for chain polling once. It does not make storage, decode work or analytical queries free. The runtime keeps those costs visible so density does not become a pleasant-sounding route to an out-of-memory kill.

When one process is not enough

Scaled mode moves cursor ownership into a control plane backed by Postgres. Workers register, claim leases for chains and fence ownership with monotonically increasing values. A worker may only write while it owns the current lease. If it dies, another worker can take the lease and continue. The fence prevents a late former owner from writing as though nothing happened, which is the small but crucial detail separating failover from two machines cheerfully scribbling over the same state.

The data model remains recognisable: mounts identify datasets, cursors follow chains, hot data is reversible and sealed data is durable. Scaled mode changes who is allowed to perform the cursor work, not what the chain data means.

For configuration and practical commands, see run many nests and scaled mode. The final chapter is about operating the whole arrangement without taking a green health endpoint as proof of truth.

8. Operating a truthful index

An indexer can be available and wrong. It can answer HTTP requests while following the wrong chain, serve a stale cursor, decode against an unsuitable ABI, or retain a view whose meaning nobody has checked since the first enthusiastic afternoon. Availability matters. It is not the whole definition of health.

Nuthatch is designed to make its claims checkable. The remaining work is operational discipline: choose what to verify, give failures a route to a human, and avoid calling a system healthy merely because it remains capable of returning JSON.

Verify the inputs and the result

Start with the nest package. Inspect the contract addresses, deployment ranges, vendored ABIs and event selections. Rebuild the generated schema and decode registry from those inputs. Confirm the NID when loading or deploying a bundle. This establishes that the machine is indexing the package you intended, rather than an equally well-formatted stranger.

Then verify a result against the chain at a fixed watermark. A useful check has a known block range, a concrete expected answer and a way to replay the SQL. For an event-derived view, compare its rows with the events and, where appropriate, with an independent chain query. For an application fallback, exercise the exact named query and response shape the application will use. A test that only proves that an endpoint returned 200 has the emotional comfort of a fire alarm with its batteries removed.

Verify the endpoint, and then verify it again later

The package is not the only input. The RPC endpoint is one too, and it is the one that changes without telling you.

nuthatch doctor --rpc <url> asks an endpoint three questions before a backfill trusts it: the widest eth_getLogs range it will serve, the largest JSON-RPC batch it accepts, and whether it has archive depth. Each of those limits otherwise surfaces mid-backfill as a retry loop that looks exactly like slowness, which is the worst way to learn it. Point it at the nest with --dir and it probes with the full declared contract filter the nest will actually issue, rather than an empty one or the first contract in the file.

Read its window figure as a floor, not a ceiling. A probe with no address filter measures the provider’s raw block-range capacity; every measurement on record has address-filtered limits coming in under that, so the number is a conservative lower bound on what a real nest sustains.

The part worth building a habit around is the second probe. Nuthatch ships measured endpoints for its built-in chains, and one of them was measured on a Tuesday and had silently lost archive depth by the Wednesday - a from-deployment backfill could no longer use it at all, while the recorded figure in the source still said otherwise. A recorded measurement is a snapshot presented as a property. Nothing about an endpoint’s past behaviour is a promise, including ours, so probe before a long backfill rather than trusting a number somebody wrote down once.

Observe the pipeline, not just the server

Metrics make the stages visible. nuthatch_tip_lag_blocks tells you whether the cursor keeps up. nuthatch_last_poll_unixtime reveals a poller that has frozen. Per-nest health tells you whether a part of a shared runtime has been quarantined. nuthatch_cursor_live identifies the chain cursor that has died even if another chain in the same process remains busy. RSS and query rejection counters show whether the node is protecting itself as intended.

Alert on sustained lag, a stalled poller, quarantined nests and approaching memory limits. Treat the first three as a service issue and the last as an opportunity to reduce query concurrency or reconsider the mounting budget before the machine makes the decision rather more abruptly. The metrics guide contains runnable Prometheus examples.

Secure the separate surfaces

The data API is read-only, but a runtime may also expose administrative routes that mount or remove nests. Do not put an unauthenticated administrative listener on the public internet and then feel surprised when somebody experiments with it. Restrict the listener, use a gateway where public access is needed, and keep database credentials and RPC URLs in deployment configuration rather than the nest package.

SQL needs its own care. General SQL is powerful enough to consume resources even when it cannot write. Use named queries or an allowlist for public services. Keep the node’s built-in timeout, concurrency and result bounds on. Per-caller rates and quotas require authenticated identity, so place them at the gateway. Calling a local concurrency semaphore a rate limiter would be a category error with a pleasingly dangerous outcome.

Recover without inventing history

When something fails, first establish which layer is unhealthy: RPC reachability, cursor progress, one dataset’s decoder, storage, query load or the gateway in front. The runtime is built so a quarantined nest, a lease handover and a failed query are observable conditions rather than silent reasons to return old data forever.

Do not repair a suspected data error by editing sealed rows or retroactively decoding history under a new ABI. Preserve the evidence, identify the package and range involved, build the corrected dataset under its own identity and migrate consumers deliberately. The cost of this restraint is small compared with explaining an untraceable historical rewrite later.

That is Nuthatch’s central ethic. It is not merely a fast way of turning logs into tables. It is a way of retaining a clear chain of custody from authored definition to chain event to query result. Once that chain of custody exists, a team can operate its own critical data path without asking a remote endpoint to be both available and believed.

Use production operation, security and verification as the practical checklist alongside this chapter.

Appendix A. One log, end to end

The architecture becomes less mysterious when reduced to one ordinary event. Suppose an ERC-20 contract emits a Transfer log in block 19,000,100. The transaction succeeded, the receipt holds the log, and the chain’s RPC can return it through eth_getLogs. What must happen before a reader can safely ask Nuthatch for that transfer?

The short answer is fetch, route, decode, order, persist, then eventually seal. The longer answer is worth knowing because each verb protects a different invariant.

1. Fetch only a defined universe

The cursor has a registry built from the authored package. It knows the contract addresses and the event topic hashes that it is prepared to decode. For a solo nest it asks the source for logs inside a bounded block window. In a runtime with several mounts on the same chain, the cursor asks once for the union of those filters and routes a returned log to every live nest whose registry matches.

The RPC response is not yet application data. It is untrusted transport input. It may arrive out of order, it may include logs useful to another mounted nest, and an endpoint may reject a window or return a response too large for its local limits. The cursor’s retry, window-sizing and concurrency policy exist here. They deal with the ordinary weather of RPC infrastructure before a row reaches durable storage.

An empty result still matters. The cursor must advance over a block range with no selected events, otherwise it would return to the same quiet range forever. Progress is about blocks successfully accounted for, not merely rows received.

2. Route and decode with a pinned registry

For the transfer log, the registry looks at the emitting address and topic zero, identifies the Transfer(address,address,uint256) decoder from the vendored ABI, and decodes indexed and data fields into a typed row. It also supplies the implicit chain columns: block number, transaction hash, transaction index, log index, address and, where configured, the block timestamp.

No schema inference happens at this point. The table layout was generated from the nest’s inputs. An unknown event is not invited to become a new column because it happened to look interesting. Likewise, a malformed log does not earn a creative interpretation. The point of a pinned registry is that the mapping from byte sequence to row is reviewable and repeatable.

Factories add one wrinkle. A factory event can discover child contracts which should be indexed thereafter. During backfill, Nuthatch performs discovery before the authoritative decode for the window so a child discovered in the window is already known when its own logs are decoded. The final row path remains the same. Discovery is not allowed to become a second, differently behaving decoder.

3. Establish a canonical local order

Providers are not entitled to return equivalent log sets in the same order. Nuthatch therefore sorts decoded rows by chain coordinates before writing or sealing. The salient ordering is block, transaction and log index. This is not cosmetic. A view calculating a balance, a factory registry or a content-addressed segment must behave the same when two RPC endpoints return the same chain facts in different sequence.

This is also why concurrent backfill needs care. Fetches may happen in parallel, but their results must enter the sealing path in deterministic block order. Otherwise speed has changed the bytes of historical storage, and a supposedly content-addressed result has become dependent on timing. That would be a rather expensive way of saving a few seconds.

4. Commit to the hot store

For an unfinalised block, the sorted row is written to the dataset’s hot store alongside the cursor checkpoint. The write establishes two things together: the transfer is visible to live reads, and the cursor now has evidence of which chain block it believes it processed. A later reorg check compares that evidence with the chain’s current answer.

Derived views and incremental state receive the same event in this phase. Their update is part of the reversible hot path. If the log later belongs to a discarded branch, the rollback replays its effect with the opposite weight or rebuilds the affected hot state. A derived answer is therefore not permitted to outlive the raw event it depends upon.

5. Seal after finality

Once block 19,000,100 crosses the finality watermark, Nuthatch serialises its rows into a content-addressed Parquet segment. The segment is staged and verified; the seal manifest is updated only when the output is ready to become the committed sealed history. The matching hot rows can then be pruned. The event is no longer subject to ordinary reorg rollback.

--seal-direct uses the same sealed representation for old, already-final history, bypassing the hot store during an initial bulk backfill. It still resolves every declared [[calls]] input at its pinned block before it commits the segment. It is an optimisation with an important precondition: the range must be past finality. It does not use a fast path to omit declared inputs or declare recent, reversible blocks permanent.

6. Answer a query with provenance

When a reader asks for the transfer or runs SQL over its table, the serving layer reads sealed segments and the hot tail as one logical surface. The response carries watermarks and source information so a caller can tell how current the answer is and whether the hot contribution was available. A hot-store failure must not quietly make a query look complete while returning only sealed history.

That final detail is representative of the design. A Nuthatch answer is useful not only because it contains a number, but because it can say which package decoded it, how far the cursor had reached and what portion of history was sealed when the answer was formed.

Appendix B. A reorganisation, walked through

Reorganisations are where an indexer either demonstrates that it understands a blockchain or quietly begins preserving fiction. The normal case is recoverable precisely because Nuthatch keeps the recent part of history hot and reversible.

Consider a cursor that has processed blocks 100 through 110. Its finality policy has sealed through block 104. Blocks 105 through 110 remain in the hot store, along with block-hash checkpoints. A transfer in block 108 has incremented a derived balance view. At this instant that is a valid, useful result, but it is not yet final.

The chain changes its mind

On the next poll, the cursor obtains a head which does not agree with its recorded checkpoint for block 110. It walks backwards through the known checkpoints and the source’s current block hashes until it finds the deepest common ancestor. In this example, block 106 still agrees; blocks 107 to 110 were replaced. The ancestor is 106.

The cursor does not try to patch individual logs based on a hunch. It performs a rollback to the ancestor. The hot store deletes rows above 106, resets its last-block metadata to 106 and removes the checkpoints that belonged to the old branch. The derived balance view retracts the effect of the former transfer in block 108. Factory child discovery that occurred only on the discarded branch is rolled back as well. The cursor then starts again at 107 and indexes the canonical replacement blocks in the normal path.

The application may have observed a provisional balance before the rollback. That is inherent to serving near-tip data. What Nuthatch promises is convergence: once it notices the reorg, neither the raw table nor a maintained derivation may retain the old branch.

Shared cursor, many datasets

Now place three nests on the same chain cursor. The cursor detects the hash disagreement once at its shared boundary, then fans the rollback out to every live dataset. Each dataset may have a different sealed watermark because it may have been mounted at a different time or progressed differently. A nest already at or below the ancestor does nothing. A nest whose hot range contains the discarded blocks retracts them. If one nest cannot roll back, it is quarantined rather than making the other datasets lie about their state.

This is why a shared cursor does not require shared mutable tables. Fetching and reorg detection are shared chain work. Row ownership, store mutation and local failure handling remain per dataset. The structure is slightly more machinery than one giant database, but much less machinery than trying to explain which tenant’s rows were inadvertently removed by a global repair.

The finality line

Return to the example. A reorg to ancestor 106 is repairable because the seal watermark was 104. Every affected block is still in the reversible hot layer. But imagine the source reports an ancestor of 102. Blocks 103 and 104 have already been sealed as immutable history. Deleting only the hot rows above 104 would leave sealed data from the discarded branch in the query surface.

Nuthatch refuses this condition. It reports a finality violation and halts the affected index rather than silently presenting a half-correct history. This is not a graceful recovery in the marketing sense. It is the only honest behaviour once the external finality assumption has been violated. The operator must investigate the chain source, finality configuration and recovery procedure instead of allowing a plausible but inconsistent index to keep serving.

The same line applies to direct sealing. That fast backfill path only processes a range already behind the finality boundary. Its performance comes from avoiding hot writes, not from relaxing the definition of permanent history.

What to look for in practice

nuthatch_reorgs_total records ordinary detected reorgs. The health and readiness surfaces expose the last indexed and sealed watermarks. A sudden tip lag accompanied by reorg growth calls for a look at the source and chain conditions. A finality violation is a hard incident, not a counter to wave away.

The important operational habit is to distinguish “we are behind” from “we are wrong”. Being behind can often be fixed by an RPC change or smaller windows. A reorg below seal means the system has lost the ability to correct historical facts automatically, and it should be treated accordingly.

Appendix C. A lease handover, walked through

Running a cursor on one machine is simple because there is one process that can write its hot state. Running it across machines introduces a more awkward possibility: the first worker is slow or partitioned, the control plane gives the work to a second worker, and then the first worker comes back convinced it still owns the chain. Without a fencing rule, both can write. This is how a failover exercise becomes a data-corruption exercise with better branding.

Scaled mode assigns cursor work through a Postgres-backed control plane. Workers register and claim a lease for a chain. The lease has an owner and an owner_fence, a monotonically increasing number that identifies one particular period of ownership.

The normal sequence

Worker A starts, registers and claims the lease for Arbitrum. The control plane records A as owner with fence 41. A runs the Nuthatch cursor, advances through blocks and renews its lease. Its writes and progress reports carry the ownership context expected by the control plane.

At this point there is one writer. The control plane does not infer that from a worker’s optimism. It has a current lease record which says so.

A fails, B takes over

Suppose Worker A is killed, loses network access, or otherwise stops renewing. Once its lease expires, Worker B can claim the same chain. The control plane updates the owner to B and increments the fence to 42. B now starts the cursor under fence 42.

The increment is not an ornament for logs. It distinguishes this ownership epoch from A’s old one. Any action that still arrives from A carrying fence 41 can be rejected as stale. Worker A cannot resume and extend its former claim merely because its process remained alive long enough to regain connectivity. The control plane has moved on.

A correctly written worker must also stop its local ingestion task when it learns that it has lost the lease. This reduces unnecessary work and narrows the time during which it could attempt stale operations. The fence remains necessary because process shutdown and network delivery are not atomic events. It is the backstop that makes delayed messages and slow death harmless.

Control plane outage is not automatic eviction

There is a separate failure worth naming. If the control plane is unavailable but the worker still holds a valid lease and its data store is available, the cursor may continue indexing according to the system’s lease and outage policy. A control-plane outage is not evidence that a second worker has taken ownership. In the two-machine test, workers continued processing during a deliberate control-plane outage while Postgres remained available, then reconciled when the service returned.

The boundary is always ownership, not mere connectivity. If the worker can no longer establish that its lease is current, it must not keep acting as the sole authority by force of habit. Conversely, if the control-plane service is briefly unavailable but the durable lease record still protects the writer, stopping the cursor needlessly can turn a control-plane incident into a data availability incident.

What this buys the operator

With leases and fences, failover is observable and testable. The worker roster shows registrations. The lease record shows the owner and fence. A deliberate handover should move ownership and increment owner_fence. A healthy test is not “two workers exist”. It is “the old holder stopped being allowed to write, the new holder took over, and indexing continued without two authorities claiming the same cursor.”

Scaled mode does not alter the event model, decode registry or sealing rules described elsewhere in this book. It gives those same rules a single writer across machines. The real achievement is not that a worker can restart. It is that the system can tell the difference between a restart and two writers, which is where the difficult bits live.