Documentation Internals: how it actually works

Internals: how it actually works

This is the long version, for people who need to know why it is safe rather than that it is. It follows one log entry from an RPC response to a row someone queries, then covers what each component refuses to do, which is usually the more interesting half.

Line counts are given as a sense of weight, not a metric. The crate’s src/ alone was about 204,000 lines of Rust in October 2026.

The shape of it

                        ┌─── one cursor per chain ───────────────────────┐
   eth_getLogs ───►  ingest ──► decode ──► hot store ──► seal ──► segments
                        │        registry     (redb)     past      (Parquet,
                        │                        │      finality    immutable)
                        │                        │                     │
                        │                        └──► IVM circuit      │
                        │                             (DBSP)           │
                        └────────────────────────────────┬─────────────┘
                                                         ▼
                                       serve: point-reads from redb,
                                       analytical SQL from Burrmill over
                                       segments ∪ tip, MCP over the same

The whole design turns on one split: the hot store is mutable and the cold store is not. Every other guarantee is downstream of it.

Since 5.14.0 the stateful middle of that picture lives in its own workspace crate, nuthatch-core (core/, RFC-0075): a nest’s window path, reorg detection and rollback (cursor.rs), the redb hot store (redb_store.rs), the DBSP circuits (views.rs, exposure.rs, velocity.rs, flags.rs, the entity_* modules) and the bodies of the query routes (answer.rs), over traits that a node and a browser tab both implement. The binary calls it, and also builds it for wasm32 to run a whole nest in a tab at /play/live. For a binary user nothing changed: the routes answer byte for byte as before, and the segment-hash pins and reorg suites pass with their bodies untouched. The file names below are the binary’s where it still owns the code, and core/src/ where it moved.

Ingest - rpc.rs, indexer.rs

indexer.rs is the loop that makes it alive: poll logs, decode, store, and serve concurrently. One cursor per chain, each its own observable failure boundary; a runtime can run several. A lone nest runs one loop; a runtime’s cursor, since 5.14.0, is an extraction stage and one ingest task per nest (below, under the runtime).

rpc.rs is a deliberately thin JSON-RPC client - eth_blockNumber, eth_getLogs, block headers, and eth_call - with round-robin failover across the configured endpoints and per-endpoint health tracking. Two behaviours are worth calling out because they were both learned the hard way:

Failures are classified, not merely retried. A rate limit, a transport blip, an oversized-range refusal and a rejected credential are four different things. An auth rejection is cooled down loudly rather than retried forever; a 429 escalates across the pool; an oversized range is split and retried, taking the provider’s own suggested range when it offers one. A refusal of the filter’s address list (publicnode on BSC and Polygon refuses ten or more addresses in one eth_getLogs) is a fifth kind: the list is halved until the endpoint accepts it, and that group size is remembered, rather than the endpoint being cooled down as if its credentials had been refused. An archive refusal stops a backfill at once with a hint to supply an archive endpoint. Since 5.18.0 an eth_getLogs over more than one block that gets no answer within the client timeout is narrowed like an oversized range, rather than retried at the same width as a transport blip.

A failure we cannot classify is split once anyway. Providers phrase the same refusal a dozen ways and invent new phrasings without telling anyone. Running Sentio’s OBIB benchmark found the cost of getting this wrong: Alchemy returns its oversized-range refusal as HTTP 400, a status the classifier did not enumerate, so the window was retried unchanged - forever. The benchmark did not run slowly, it never finished. Speculative splitting means an endpoint whose phrasing we have never seen degrades instead of hanging.

The log window adapts. Every provider caps eth_getLogs differently, most document it wrongly, and several vary it by tier. So the window widens while an endpoint keeps up and narrows the moment it does not - the real limit is discovered rather than configured. nuthatch doctor runs that discovery on demand and reports what an endpoint will actually serve.

Decode - registry.rs

registry.rs is a thin re-export; the decode logic lives in the nuthatch-decode crate under decode/, so fuzz targets build without the rest of the binary. The decode registry is the determinism boundary. Given each contract’s resolved ABI, it builds one immutable map from topic0 → decoders, filtered by emitting address, and turns any log into a typed row keyed to a per-(alias, event) table.

It is plain deterministic Rust. No LLM is anywhere near this path, by rule - models generate code and tests, never runtime data.

The registry’s content hash (a sha256 over a canonical, order-independent serialisation) is recorded in every sealed segment’s manifest and returned in query provenance. That hash is what makes re-execution checkable: two operators indexing the same range with the same registry produce byte-identical segments, and a query result can be traced to the exact decode that produced it.

Decodings are versioned, never retroactive. When an ABI improves, history is not silently re-decoded - that would change answers under consumers who already have them.

The hot store - core/src/redb_store.rs

redb, embedded, holding the mutable tip. Four tables: entities, meta, blocks, outbox. A --features graph build adds a fifth, checkpoint_hashes, a reverse index over the checkpoints; a default build never creates it.

Entities are keyed {block:012}-{log_index:06}, zero-padded so lexicographic iteration is chain-ordered - which makes “everything above block N” a range scan rather than a filter, and that is exactly what a reorg needs.

Four tasks write it: the ingest loop (entities, checkpoints, meta and the alerts’ outbox enqueue), the webhook collector (one per nest since 5.21.0, queueing each webhook’s rows onto the outbox in the same commit as the cursor that covers them, so a seal only wakes it and never waits on a delivery query), the alert-delivery worker draining the outbox, and the IPFS resolver storing fetched documents and their typed rows. The resolver shares the ingest loop’s key range, so it writes only in a transaction that finds the row naming the document still there from the same block; if a reorg rollback or a seal’s prune has removed that row, the late write is a no-op. redb serialises write transactions, so each check and its write are atomic. What is single is the cursor: one ingestion task per chain advances the chain state. (The module once claimed a single writer; it was wrong, and now says what is true.)

Sealing - seal.rs

Once a range is final, each table’s rows in that range are written to their own content-addressed Parquet segment - {table}-{hash}.parquet, or segments/<hash>.parquet in a runtime’s shared store - catalogued in a manifest, and then pruned from hot.

Finality is per chain, not a constant. chains.rs carries the policy as data:

chainfinalitydefault getLogs window
mainnetDepth(64) - 64 blocks behind tip20
arbitrum-oneFinalizedTag - the node’s L1-aware finalized tag, falling back to a depth2000
baseFinalizedTag, same fallback1000

An L2 is a data entry here, not a fork of the indexer.

Sealing is where the append-only guarantee is made: a segment is written strictly past finality, so the columnar layer never sees a reorg. If a change ever requires mutating a sealed segment, the design is wrong and we go back rather than add a mutation path.

All tables in a nest seal together per finalized range, so sealed_through stays a single global watermark - which is what lets hot and cold be unioned without deduplication.

Sealing does not read its output back; the hash is taken from the bytes in memory. Checking is a separate job, and since 5.21.0 it is off the start path: a start rereads only the segment files whose length, times, inode or expected hash moved since a pass last hashed them (segments/verified.json), and a segment-verify thread hashes every segment a minute after start, at up to 64 MiB/s across the process, and every 24 hours after. A segment that does not hash to its address is quarantined: moved to the dataset’s quarantine/, or, in a runtime’s shared store, to quarantine/<hash>.parquet beside the store, reducing every dataset that names it. Where segments/ is a link onto another filesystem the segment is copied aside and removed, and one that cannot be moved is refused in place. Nothing retracts derived state built from a segment quarantined later, so /ready reports derived_state_stale until a restart rebuilds it from what remains.

Reorgs - core/src/cursor.rs, redb_store.rs::rollback_to

A reorg only ever touches the hot store. Detection asks whether the last block we committed against is still canonical - the blocks table stores each block’s hash, so detect_reorg re-checks that hash against the chain and walks back to find the common ancestor. Recovery is rollback_to(block), a range delete above the fork point inside one write transaction, which is exactly the operation the zero-padded key ordering makes cheap.

Because sealing is strictly past finality, there is no ordinary case where a reorg reaches a segment. A reorg below sealed_through is a finality violation this model cannot repair - the doomed blocks are already in immutable segments and pruned from hot, so a retraction would be silently incomplete and the two layers would permanently disagree. So it bails rather than quietly rewriting history, and that nest stays down until an operator looks at it rather than being backed off and re-admitted.

One detail worth having, because it was a real blind spot: if a boundary’s hash could not be stored, treating “no hash here” as “nothing to verify” would skip the check entirely. It walks to the nearest stored checkpoint instead.

Random reorg depths are property-tested to converge on canonical chain state.

Derived state - core/src/views.rs, exposure.rs (DBSP)

The differentiator, and it is real rather than aspirational: derived entities are incremental views maintained by DBSP circuits, not hand-written update logic.

Nobody writes “on transfer, load balance, add, save”. Balance is stated as Σ(in) − Σ(out) and the circuit maintains it. The payoff is that a reorg is a retraction - the same circuit that consumes a +1 delta consumes a −1, so rollback correctness is a property of the engine rather than of remembering to write the inverse of every handler.

Counterparty exposure (the compliance path) is a second circuit over the same facts.

Query - analytics.rs, serve.rs

Two surfaces over the same data.

Point-reads take three paths. An entity by id reads redb first; on a miss (the row has been sealed and pruned) it scans the sealed segments through Burrmill, under the same admission gate and deadline as /sql, so a busy server answers 503. Balances and flags are read from the in-memory IVM views, not redb, with no SQL engine involved.

Analytical SQL runs on Burrmill, an embedded engine on DataFusion. A session opens empty and registers, for each table a statement binds, the segment list and the hot rows staged for it, so the ingestion path never writes to the engine.

analytics.rs keeps every policy decision - the read-only gates, the allowlist walk, the deadline - and Burrmill sits behind the engine trait (engine.rs) in engine_burrmill.rs. DuckDB sat there until 4.1, and a shadow-burrmill feature ran the two side by side before the cutover (Replacing DuckDB, after all).

The union is exact without deduplication, and that is structural rather than careful: cold includes only segments at or below sealed_through, hot only rows above it. The two sets cannot overlap, even during the brief window between sealing and pruning.

Results carry provenance: the as_of and sealed_through watermarks, the nest’s nid and registry_hash, each referenced entity’s watermark, and source, which is hot+sealed normally and sealed when the tip could not be scanned and the answer is cold-only. A number can be cited and re-derived. It names the dataset and the cut, not the individual segments. Since 5.22.0 an answer asked at a block (at_block or at_time) adds segments_sha256, one hash over the sealed segments it could read (seal::read_set_hash, absent while one is a provisional tail), and an answer that read labels adds the label set’s labels_sha256, since the NID no longer pins one.

Answers are remembered by sqlmemo.rs, and the memo is not a TTL cache. Its key is every input the answer depends on: the statement and its row cap, the sealed segments it is served, a generation that moves only when a commit changes hot rows, each entity’s watermark, and the authored files. A cursor poll that commits no rows leaves the key alone, so a quiet nest at the tip answers a repeated statement from memory ("cached": true), and a degraded answer is never remembered. It is bounded by NUTHATCH_SQL_MEMO_BYTES (64 MiB by default, 0 turns it off) and cleared by a restart.

/sql is a genuine analytical surface exposed to callers, so it is guarded: a 30-second timeout, a 50,000-row cap, a 64 MiB result ceiling, 2 concurrent analytical queries, a 16 KiB query-length limit, and a refusal past 2,000,000 unsealed tip rows or 64 MiB of stored tip row JSON, whichever comes first. Those tip limits refuse rather than truncating, because a partial tip silently changes the answer to an aggregate, and a wrong number is worse than an error. The 64 MiB tip limit bounds input copied from the hot store, separately from the 64 MiB result ceiling. A declared (named) query is also refused before it runs if the sealed segments plus the hot snapshot it would read pass 512 MiB.

Two controls stop /sql touching the filesystem, deliberately with different failure modes - see security.

Factories - factory.rs

Uniswap-class protocols are unindexable without dynamic data sources: the children do not exist when you write the config. A factory watches an event (PoolCreated) and indexes the child it announces under a template ABI, into shared {template}__* tables.

Each window asks for logs by the factory and child addresses, in the backfill, the tip loop and a runtime’s cursor alike, until 500 children have been discovered. Past that the fetch flips to topic0-only and discards non-children locally, so N children cost roughly one child’s RPC budget rather than N. A chain whose shipped endpoint refuses address-less eth_getLogs (BSC) keeps asking by address past 500, and an endpoint that refuses a topic0-only fetch at runtime switches the cursor to addresses. Since 5.13.0 a solo nest’s tip fetch and a seal-direct backfill flip per template: static contracts are never fetched without an address, a template past 500 children of its own is fetched by its own topic0s, and many small templates that cross 500 together all go wide.

Packaging - blob.rs, distribution.rs

A nest’s authored inputs - config, ABIs, views, skills, and since 5.22.0 never label sets - canonicalise into a bundle pinned by a Merkle root. nest load verifies the manifest, every file’s bytes, and that the decode registry regenerated from those inputs matches the author’s pinned hash. A nest that cannot reproduce its own registry is refused, and since 5.11.0 so is one whose installed files do not compute the NID its manifest claims. Every input must be a regular file: a symbolic link is refused by name, and since 5.18.0 so is a file that is not UTF-8. A .gitignore is not an input (5.18.0), and neither are labels/ and labels.manifest.json (5.22.0): identity::is_input leaves them out of the NID and the data identity, and a manifest packed before 5.22.0 that lists them names the nest without them.

That makes a bundle tamper-evident, which is not the same as safe - a valid bundle can still declare outbound endpoints you would rather it did not, so load warns about every non-loopback URL and ranks link-local/metadata addresses at error.

A registry (a directory, or any S3-compatible bucket) adds publish/pull by name@version.

The runtime - runtime.rs

One runtime, many nests, one isolated cursor per distinct chain. Nests on the same chain share a cursor and one getLogs per window - N nests for roughly one nest’s RPC cost - with per-nest namespaced stores.

Since 5.14.0 a runtime’s cursor is the buffered cursor (indexer/buffered.rs, RFC-0076): an extraction stage fetches each range once with the union filter, reads the header that ends the window, decides any reorg, and appends the window to a bounded buffer that spills to <runtime>/cursors/<chain>/buffer/; one ingest task per nest reads the buffer from its own position and is the only writer of its dataset. A nest behind the buffer, or back from quarantine, fetches for itself by its own filter until it reaches the buffer, so extraction never moves back. A reorg drops the buffered windows above the ancestor and leaves a marker; each nest rolls back from its own position with the same rollback_reorg, and a nest’s position counts toward extraction’s floor only once it has applied every marker issued. A lone nest outside a runtime keeps its own loop.

The law: a cursor is always single-chain, single-writer, one observable failure boundary. Two chains are never multiplexed behind one cursor. A second chain means a second cursor.

The RAM budget is ≤2 GB per active-chain cursor, CI-enforced. A mount whose projected footprint would exceed it is refused with a 507 rather than admitted with a warning - a budget that can be quietly exceeded is not a budget.

A faulting nest is quarantined, not fatal: its healthy siblings keep indexing and serving, and it is re-admitted on a backoff if the fault was retryable. Blast radius is bounded in both directions - a nest’s error no longer kills its cursor, a cursor’s death no longer kills the runtime. Since 5.13.0 that includes a panic in one nest’s window, prepare, reorg rollback or rejoin, which quarantines that nest terminally while the cursor carries on, and a nest that fails to build at boot, which boots quarantined rather than failing the runtime. Since 5.18.0 so does a nest that cannot be loaded or whose data directory is missing, and a panic while one nest is built. Since 5.16.0 a store whose ownership another holder has claimed (LostOwnership) is a terminal fault for the nest on it, not a retryable one.

A window is split at its commit (5.14.0): the part that decodes, reads, checks and binds has no side effect a retry would see, and the commit, once begun, always finishes, so the views it feeds and its rows land together or not at all. That split is what lets window_deadline_ms abandon a slow window safely, and what lets each request answer from one published snapshot: a hot-store read transaction and one immutable version of each view.

What changes in scaled mode - pgstore.rs, worker.rs, scheduler.rs, controlplane.rs

The same business logic, pointed at different substrates behind a HotStore trait. There are no #[cfg] forks of the pipeline; it is a role flag, and it is opt-in at build time so the default binary carries no database driver.

  • pgstore.rs - the same HotStore contract over Postgres.
  • controlplane.rs - desired state. What should run, not what does.
  • scheduler.rs - placement, written as a pure function of (workers, desired nests, current assignments). No database, no clock, no network, because placement is where the interesting mistakes live - double assignment, silent under-scheduling, churn - and none of them need distribution to reproduce.
  • worker.rs - the reconcile loop: heartbeat, take a lease per assigned cursor, index what it holds, pull from the registry what it does not have.

Ownership is enforced by the store, not by agreement. Every write carries a fence; a stalled worker that wakes to find its lease reassigned has its writes refused inside the same transaction as the write. That is what makes --scale writer=N safe. The worker does not lean on the fence alone: each tick takes what it holds from the leases it actually renewed or acquired, never from memory, and stops ingesting a cursor whose lease was taken on that tick while its other cursors carry on.

The control plane and the lease are deliberately independent: a control-plane outage stops rescheduling, not ingestion. Measured - 377 blocks indexed through a 90-second outage.

Scaled mode is younger than embedded and has not carried a production workload for anyone. Until v0.9.3 the writer pool took leases and ran no indexing at all; see verifying a deployment for what that cost and how it is checked now.

Where to start reading

indexer.rs for the loop, registry.rs for what makes it deterministic, core/src/redb_store.rs and seal.rs for why a reorg cannot corrupt history. The design record is in the RFCs - numbered, and honest about what was wrong.

Checked against Nuthatch 5.22.0 (2026-11-17). Verify a release-specific command with the CLI reference before running it in production.