Host nests for others

This is the guide for running nuthatch as a host: one runtime, started with nothing in it, that nests are added to, paused, moved and removed over an HTTP API while it keeps serving the rest. It is the shape a platform uses, and it is equally the shape of one operator who wants to manage a box by script instead of by editing files and restarting.

The whole contract fits in a sentence: you hand the runtime a nest identity (NID), and the runtime fetches it, verifies it, stores it, indexes it and serves it. Nothing on your side unpacks bundles, runs schema tools or starts processes. Everything below is available since 3.13.0.

What stays outside the runtime, by design: who your callers are, what they may do, what they pay, and how the process is supervised. nuthatch sees a tenant as an opaque label and knows nothing else about it. Sign-in, plans, per-tenant authorisation, quotas and billing belong to a gateway in front of the runtime; process supervision belongs to systemd or a container restart policy.

The moving parts

• A nest is authored inputs (config, ABIs, views) packed into a content-addressed bundle.
• A NID is the nest’s identity: the key its data is stored under, data/<nid>/. Change anything that changes what the nest indexes and you get a new NID; upgrading the nuthatch binary does not.
• A registry is where bundles live: a directory or an S3-compatible bucket. See the nest registry.
• A runtime is one nuthatch dev --dir <dir> process over a directory with a mounts.toml. It hosts many nests, with one cursor per chain. See run many nests.
• A mount gives a NID a name, served at /<name>/. Two mounts may share one NID and one dataset.

1. Publish the nests you will host

On the machine where a nest is built:

nuthatch nest bundle ./usdc                  # → usdc-<hash>.bundle
nuthatch nest publish usdc-<hash>.bundle --registry /srv/registry --as usdc@1.0.0
nuthatch nest nid --dir ./usdc               # the NID a host mounts it by

nest publish prints the NID as well as the hash. The registry indexes every bundle by NID, so a NID is all a host needs: nuthatch nest load <nid> --registry /srv/registry pulls it anywhere, and refuses a bundle whose own manifest does not compute to the NID asked for.

2. Start an empty runtime

Declare the chains you will serve, and nothing else:

# /srv/runtime/mounts.toml
[runtime]
name = "host-1"
max_rss_mb = 2048              # per-cursor RAM ceiling

[[chains]]
chain = "mainnet"
chain_id = 1
rpc_urls = ["https://…"]

[[chains]]
chain = "base"
chain_id = 8453
rpc_urls = ["https://…"]

export NUTHATCH_ADMIN_TOKEN=$(openssl rand -hex 32)
nuthatch dev --dir /srv/runtime --listen 0.0.0.0:8288 --registry /srv/registry

• Chains are declared at boot, and only there. A chain’s RPC is dialled when the first nest is mounted onto it, so an unused chain, or one whose endpoint is down when the process starts, cannot stop the runtime. Adding a chain means a restart.
• --registry lets a mount fetch a NID the runtime does not hold. Without it, only NIDs already under data/ can be mounted.
• The admin API needs NUTHATCH_ADMIN_TOKEN off localhost. Bound anywhere but localhost without it, the runtime serves no admin routes at all, and an empty runtime then refuses to start, since nothing could ever be mounted into it. --no-admin removes the routes everywhere.

Every admin call below takes the token as Authorization: Bearer <token> (or ?token=<token>); the examples set it once in a shell alias, with the Content-Type: application/json every JSON body needs (without it the API answers 415):

alias nh='curl -s -H "Authorization: Bearer $NUTHATCH_ADMIN_TOKEN" -H "Content-Type: application/json"'

3. Price a nest before you mount it

nh -XPOST 'localhost:8288/_admin/nests?dry_run=true' -d '{"name":"usdc","nid":"9f2c…"}'

{
  "name": "usdc", "nid": "9f2c…", "fetched": true, "shares": null,
  "chain": "mainnet", "start_block": 6082465, "tip": 21000000, "blocks_to_backfill": 14917535,
  "has_data": false, "per_block_rpc": ["[extract] blocks"],
  "incoming_mb": 180, "projected_mb": 420, "ceiling_mb": 2048,
  "refusal": null, "refusal_status": null
}

A dry run runs the same admission checks a real mount runs and mounts nothing, so its refusal_status is exactly what the mount would answer. What each field tells you:

• blocks_to_backfill is the history ahead of it. tip is known only for a chain whose cursor is already running, since a dry run dials nothing; has_data: true means a dataset is already on disk and the figure is an upper bound.
• per_block_rpc lists extraction that costs RPC calls for every block on top of the shared eth_getLogs: [extract] blocks, traces, top_level_calls, state, [[calls]]. An empty list is the cheap case. See costs.
• projected_mb is the chain cursor’s footprint with this nest added, against ceiling_mb.
• shares names a mount that already indexes this dataset: mounting it again costs nothing further.

If the NID had to be fetched, it was fetched and verified, and it stays installed for the real mount.

4. Mount it

nh -XPOST localhost:8288/_admin/nests -d '{"name":"usdc","nid":"9f2c…"}'
# 202 {"name":"usdc","nid":"9f2c…","phase":"accepted","since_unixtime":1790700000}

A mount is a job, because fetching and catching up can take minutes. Poll it:

nh localhost:8288/_admin/mounts/usdc     # one job
nh localhost:8288/_admin/mounts          # {"mounts": [...]} - every mount the runtime knows

Reading a job never waits on a mount in progress. Jobs are written to mount-jobs.json in the runtime directory: after a restart an unfinished job resumes, and a failed one stays readable until the name is mounted again or unmounted.

Posting again is safe. The same name and NID answers 202 with the running job, or 200 once it is live. The same name with another NID is 409: changing a live mount’s nest is a move (step 7).

?wait=true answers only when the mount has finished, with the synchronous statuses below. It suits a script; a platform should poll.

On the job route the same refusals end the job failed with the same reason. A nest the runtime fetched and then refused is removed again: a refusal leaves nothing on disk.

Once live, the nest’s full API is under its name: /usdc/sql, /usdc/tables, /usdc/ready. GET /nests lists every live mount with its health, and GET /ready answers for the whole runtime.

5. Tenants and names

A mount’s name is its route. For more than one party, name mounts tenant/alias:

nh -XPOST localhost:8288/_admin/nests -d '{"name":"acme/usdc","nid":"9f2c…"}'
nh -XPOST localhost:8288/_admin/nests -d '{"name":"globex/usdc","nid":"9f2c…"}'

Both mounts serve one dataset, indexed once. The tenant is a label nuthatch refcounts and knows nothing else about: it never authenticates it, limits it or bills it.

:::caution[Route shape is decided at boot] Routes are /<alias>/ while the mount table holds one tenant and /<tenant>/<alias>/ once it holds two or more. A live mount does not reshape the routes already served, but the next restart applies the rule to all of them. If you host more than one party, mount everything as tenant/alias from the start and route your gateway by the name you mounted. :::

6. Pause and resume

nh -XPOST localhost:8288/_admin/suspend/usdc     # 200 {"suspended":"usdc"}
nh -XPOST localhost:8288/_admin/resume/usdc      # 202, a mount job

A suspended mount leaves its cursor and releases its store, and its routes answer 503 with "suspended": true, so a caller can tell a paused nest from a missing one. Its data and its record are kept, and it stays suspended across a restart (mounts.toml lists it under suspended). A suspended mount costs no RPC and no memory; it keeps its disk.

Resume is an ordinary mount of the recorded NID: it catches up from where it stopped, and serves the 503 until it has joined. Resuming a quarantined mount is its explicit release. Suspending a name that is not mounted is 404; resuming one that is not suspended is 404. While paused, its job reads suspended, and posting a mount for the name resumes it just as resume does.

7. Move a name to a new version

A new version of a nest is a new NID. To repoint a name without a gap:

nh -XPOST localhost:8288/_admin/move/usdc -d '{"nid":"4a71…"}'    # 202, a job under "usdc"

The runtime mounts the new NID beside the old one, catches it up, and then switches the name’s routes in one step. A reader polling /usdc/ sees the old nest until that step and the new nest after it, and never an error in between. The old nest is then taken off its cursor. A move keeps its chain; a move whose new nest is refused leaves the old one serving and removes anything it fetched.

To keep the old version reachable, mount its NID under another name first; the move leaves that mount alone.

8. Unmount and reclaim disk

nh -XDELETE localhost:8288/_admin/nests/usdc                    # 200 {"unmounted":"usdc"}
nh -XDELETE 'localhost:8288/_admin/nests/usdc?reclaim=true'     # and free the disk
nh -XDELETE localhost:8288/_admin/datasets/9f2c…                # free one unmounted earlier

An unmount keeps the dataset, so mounting it again is free. Reclaiming removes it once no mount names it; a dataset another mount still uses is kept and the answer says by whom:

{"outcome": "kept", "nid": "9f2c…", "mounted_by": ["globex/usdc"]}

DELETE /_admin/datasets/<nid> answers 200 when it reclaimed, 409 when it kept, 404 when there was nothing there. Inside a running runtime a reclaim removes the dataset and the segments only it references. Segments a live nest may still be reading are left for nuthatch prune, run offline.

9. Meter it

/metrics breaks the runtime down per mount, labelled {nest="<name>"}:

Segments are content-addressed and shared: a segment two datasets both contain counts under both nests. For disk actually used, read the unlabelled nuthatch_sealed_segments_bytes, which counts each segment store once. A mount that shares another’s dataset has no storage series of its own; its bytes are reported under the mount that indexes it. See metrics for the full list.

10. Put a gateway in front

The runtime is the data plane. What it protects is itself: a RAM budget per cursor, query timeouts, a bounded SQL surface per mount, and the admin token. What it does not do is know your users. A gateway in front should:

• terminate TLS and authenticate callers;
• map a caller to the mounts they may read, and route /<name>/ only for those;
• hold the admin token and never expose /_admin/ to callers;
• meter and bill from the per-nest series above and from its own request log.

Keep the admin API on a private interface or behind the gateway only. The token is the one thing between the network and a mount.

11. Restarts and upgrades

• Restart at any time. Mounts come back from mounts.toml, suspended mounts stay suspended, unfinished jobs resume, and a fetch a killed process left half-staged is cleared first.
• Upgrade the binary by replacing it and restarting. NIDs are independent of the nuthatch version, so no dataset moves and nothing re-indexes. See upgrades.
• Supervise the process with systemd or a container restart policy. A runtime whose every cursor has died exits non-zero so the supervisor restarts it; a runtime with nothing mounted, or whose nests have all been unmounted, stays up and waits.

The whole walkthrough

# build side
nuthatch nest bundle ./usdc
nuthatch nest publish usdc-<hash>.bundle --registry /srv/registry --as usdc@1.0.0
NID=$(nuthatch nest nid --dir ./usdc)

# host side
nuthatch dev --dir /srv/runtime --listen 0.0.0.0:8288 --registry /srv/registry &
nh -XPOST 'localhost:8288/_admin/nests?dry_run=true' -d "{\"name\":\"acme/usdc\",\"nid\":\"$NID\"}"
nh -XPOST localhost:8288/_admin/nests -d "{\"name\":\"acme/usdc\",\"nid\":\"$NID\"}"
nh localhost:8288/_admin/mounts/acme/usdc          # until "phase": "live"
curl -s 'localhost:8288/acme/usdc/sql?q=SELECT%201'

# later
nh -XPOST localhost:8288/_admin/suspend/acme/usdc
nh -XPOST localhost:8288/_admin/resume/acme/usdc
nh -XPOST localhost:8288/_admin/move/acme/usdc -d '{"nid":"<new nid>"}'
nh -XDELETE 'localhost:8288/_admin/nests/acme/usdc?reclaim=true'

The same API is documented route by route in the HTTP API reference.

Checked against Nuthatch 3.8.5. Verify a release-specific command with the CLI reference before running it in production.