Farcaster on infrastructure you own

Run your own
Farcaster client.

A Snapchain read node you own, and a client that reads from it. No API keys, no hosted provider, nobody's rate limit.

~/snapstart3 commands
1
Compute
$ ./snapstart provision
2
Snapchain
$ ./snapstart node up
3
Client
$ ./snapstart client create

Quickstart

Clone it, then run three commands.

Everything below explains what each command does, what the protocol is doing underneath, and the ways it goes wrong — all of which we hit while building it.

git clone https://github.com/webthethird/snapstart && cd snapstart

./snapstart provision      # 0. furnish the compute
./snapstart node up        # 1. start a Snapchain read node
./snapstart client create  # 2. issue an app key and launch the client
Cost
~$10/day

Hourly billing. Destroy it when you're done.

First sync
9–14 hours

Docs say two. Two restores timed end to end: 13h 22m and 9h 09m, same instance type.

API keys
Zero

The node holds no credentials once running.

Licence
MIT

Every number here measured on a live node.

Read this before step 1

The restore can lose a race with the snapshot it is reading.

Your node fetches all three shard manifests when it starts, then restores shards one at a time, and shard 0 is always last (shard_ids.push(0), src/main.rs:413). Snapshots are replaced daily and the previous one is eventually deleted. A restore slower than that gap reaches shard 0 holding a pointer to something that no longer exists.

Restore takes
9–14 hours

5,692 chunks. Two runs on an m5.xlarge with gp3: 13h 22m and 9h 09m. Download throughput is what varies.

At risk
shard 0

Restored last, so it has waited the longest. It is also the smallest — 371 chunks, about 30 GB.

Failure signature
404s, then panic

Repeated Failed to download. The panic that follows is not unique to this — see below.

What we measured, and what we did not

We hit this twice while building, so the hazard is real. But we never pinned down the one number that would let us hand you a safe start time, and we would rather say so than publish a window that only looks authoritative. An earlier draft of this page did exactly that; shard 2 then moved two hours in a day and broke it.

Generation time, from each snapshot's own timestamp
  shard 0   10-17 05:18:45  10-18 05:18:46  10-19 05:18:52  10-20 05:18:49
            steady to within 7 seconds across four days
  shard 1   10-17 13:23:10  10-19 13:29:15
            +6m05s over two days
  shard 2   10-17 08:04:08  10-19 08:04:38  10-20 10:04:58
            +30s over two days, then +2 HOURS in one

latest.json is updated LATER than that timestamp
  10-20 06:51  still served the 10-19 shard-0 key
  10-20 07:45  first served the 10-20 key, generated 05:18:49

The old snapshot is deleted -- but we never caught WHEN
  10-20 06:38  node downloaded all 371 shard-0 chunks from the 10-19
               snapshot successfully, hours after it was "rotated"
  10-20 11:48  those same chunks return 404
  10-18 13:38  404s on the 10-17 snapshot, then the panic

Deletion is the event that actually matters, and it is the one we did not characterise. So treat any start-time rule — including one you might derive from the table above — as a guess. Our own node was saved by winning this race, not by planning for it.

What to do instead

Recovery is cheap, which is the useful part. A missing shard can be reloaded on its own — shard 0 is 30 GB and takes minutes, against 10–15 hours to redo the whole restore. Fetch the shard's current manifest, download its chunks, verify them, and untar into .rocks while the node is stopped. The technique is from reload-shard.sh by Sats100M (MIT), linked from snapchain#1044. PR #1045 fixes the root cause upstream by refetching each shard's metadata immediately before downloading it.

Whatever you do, check the manifests yourself rather than trusting times on a web page — including this one:

R2=https://pub-d352dd8819104a778e20d08888c5a661.r2.dev/FARCASTER_NETWORK_MAINNET
for s in 0 1 2; do
  curl -s "$R2/$s/latest.json" | python3 -c "
import sys, json, datetime as d
t = json.load(sys.stdin)['timestamp'] / 1000
print('shard $s', d.datetime.fromtimestamp(t, d.timezone.utc))"
done

A node that loses this race fails loudly. What is quiet is the aftermath: it still answers reads, so it looks healthy while rejecting every write.

A panic does not mean you lost the race

We assumed for two days that the panic at the end of a restore was this bug. It is not. Every successful restore ends in a panic, for an unrelated reason, and the two look identical from the outside.

called `Result::unwrap()` on an `Err` value:
  IoError(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" })

download_snapshots finishes by calling remove_dir_all on its staging directory, which deletes the contents and then the directory itself — but .rocks.snapshot is a bind mount in snapchain's own compose file, and removing a mount point returns EBUSY. That error reaches the same .unwrap() a rotation 404 would, so both produce the same crash at the same line.

It is invisible in the documented setup: the restart policy brings the node back, it finds a complete database, and it serves. Run without one and the process exits non-zero after a 9–14 hour restore with a perfectly good database on disk. We reproduced it on two nodes, the second with zero 404s, and reported it as snapchain#1076.

Concepts

What you are actually building.

If you have run an Ethereum node, you already understand this. You can read chain data from Alchemy with an API key and a rate limit, or run Reth yourself and query localhost with nobody's permission. Farcaster has the same split: Neynar is Alchemy, a Snapchain read node is your own Reth.

EthereumFarcaster
Reth / Geth full nodeRead node — stores all state, serves queries, cannot propose blocks
ValidatorValidator — permissioned, gated on a key in a registry contract
eth_sendRawTransactionsubmitMessage
eth_syncingGET /v1/info
EOA private keyCustody key — owns the account
Session keyApp key — scoped, expiring, revocable

You cannot run a validator — the validator set lives in a contract and you are not in it. That is fine. A read node stores full state and serves every read API, which is everything a client needs.

Three identities, not one

FID
Your account

A uint256 in a registry contract. No cryptography of its own.

Custody address
Owns it

secp256k1, derived from your seed phrase. Never touches the node.

App key
Signs posts

Ed25519, generated per client. Scoped, expiring, revocable.

An app key is a session key. One account can have many — Warpcast holds one, your client holds another, and revoking one does not touch the rest. That is also why a cast can tell you which client posted it: the signature identifies the software, not just the person.

Command 0

Provision the compute.

./snapstart provision

Four API calls and a boot script: an SSH keypair dedicated to this node, a firewall, an instance, a disk — then cloud-init installs Docker. Nothing mystical.

New AWS accounts land on the Free Plan, which cannot run this. It refuses any instance type that is not free-tier eligible, and the largest eligible type is 2 vCPU / 8 GiB where Snapchain needs 4 / 16. No quota increase fixes it — you have to upgrade to a paid plan. Credits still apply.

Hetzner is a third the price and cannot finish on a new account. Its default volume quota is 1024 GB and the restore peaks above that — verified by probe, a resize to 1030 GB is refused. Use SNAPSTART_PROVIDER=hetzner only if your quota has been raised.

One credential, and the node never sees it

The cloud API key belongs to the provisioning script. provision, status and destroy need it; node and client do not. Once running, the node holds no credentials of any kind.

Command 1

Start the read node.

./snapstart node up

Installs upstream's snapchain.sh, writes its config without prompting, and starts the node. It then restores from a snapshot — the slow part, and the part with teeth. See the timing window above.

Check readiness properly

./snapstart node ready

"The API responds" and "the database has data" are both true of a broken node. This verifies every shard is present and within 100 blocks of the tip.

  shard           height          delay          messages  state
  0             49007206              1                 0  ok
  1             49007206              1         458606290  ok
  2             48842741              1         455621742  ok

  ready: all shards present and within 100 blocks of the tip

Shard 0 holds no user messages — which is exactly why a message-count check would call a broken node healthy.

Command 2

Create the client.

./snapstart client create

Asks for your seed phrase, uses it once to authorize an app key, then forgets it. From that point the client holds only the app key.

Use a throwaway account. Not one you care about.

A client holding an app key can post as you. A client holding your seed phrase can transfer your account away, change its recovery address, and spend from the custody wallet. Those are not the same risk.

Registering a fresh account costs about $0.20 — the docs say $10, but IdGateway.price() returns roughly 0.0001 ETH, including the first storage unit. There is no good reason to test with an account that matters.

Only paste a seed phrase into software whose source you have read, from people you have reason to trust. That includes this project. The code that touches the phrase is scripts/app-key.ts — about 140 lines, worth the three minutes.

What issuing an app key actually does

Registering a key used to need an Optimism transaction. Under FIP-272 it does not: you submit a KEY_ADD message authorized by an EIP-712 signature from your custody address. No gas, no transaction. Three signatures are involved:

  • a SignedKeyRequest — the app saying "I asked for this key"
  • a KeyAdd signature from your custody address, authorizing the key, its scopes and lifetime
  • the envelope itself, signed by the new key — proof whoever submits it holds it
Scopes
6 types

Only what the client sends. A key scoped to casting cannot follow anyone.

Lifetime
90 days

Sliding — it refreshes on use, so abandoned keys expire by themselves.

Replay
Nonce

Strictly increasing per account, so an old authorization cannot be reused.

Writing this yourself? The gasless EIP-712 domain is { name: "Farcaster KeyAdd", version: "1", chainId: 1 } with no verifyingContract, and it sits beside near-identically named onchain helpers. Using the wrong one produces a signature that verifies locally and is rejected by the network. Also, deadline is in Farcaster seconds (epoch 2021-01-01), not Unix seconds.

Writes are confirmed, not assumed

A node returns 200 once your message is well-formed and handed to the gossip layer — then discards what happens next. A node with no peers logs InsufficientPeers, drops your cast, and still returns 200. So the client never trusts the status code: it knows the message hash before submitting and polls until the node reports it back.

Do not post immediately after starting your node. It needs gossip peers first, and until it has them your writes vanish with a 200.

The interesting part

Filter the feed by the client that posted.

Every cast carries the Ed25519 key that signed it. That key's registration records the FID of the app that requested it, and that FID has a profile with a name:

cast.signer  →  the key's registration  →  requestFid  →  that app's name

Which means a feed can be filtered by the software that produced it:

feed: 60 casts from 23 authors

   45  warpcast
   13  fireflyapp
    1  greg
    1  unknown client

include ["warpcast"] -> 45
exclude ["warpcast"] -> 15

That is a real feed on a real node, not a mockup. Casts that genuinely cannot be attributed — keys registered before the metadata existed — render as unknown client and are never silently dropped: a filter that hides posts because a lookup failed is a bug that looks like a feature.

The same field, two encodings

requestFid is reported two different ways depending on how the key was registered, which is easy to miss because the obvious one works:

onchain   signer.onChainEvent.signerEventBody.metadata
          base64, ABI-encoded SignedKeyRequestMetadata, decode to get requestFid

gasless   signer.requestFid
          a plain top-level field. `metadata` is absent entirely.

We shipped the bug this creates: decoding only the blob sends every gasless key to unknown client, including the one your own client just issued itself. Nothing errors. The feature simply fails to recognise your own posts, which is exactly the case you are least likely to test. Check both.

Money

What it costs, and how to stop paying.

AWS (default)Hetzner
Instancem5.xlarge — 4 vCPU / 16 GBcx43 — 8 vCPU / 16 GB
Disk2048 GB gp31024 GB volume
Cost~$0.42/hr — ~$10/day~€0.14/hr — ~€3.30/day
New account can finish?yesno — volume quota

Both bill hourly. Run it for a day or two, then:

./snapstart destroy

It names what it is about to delete, verifies the resources are actually gone before discarding local state, and keeps that state if anything survived — so you can retry rather than being told you have stopped paying when you have not.

Troubleshooting

When it goes wrong.

Every entry here is something we hit. Most are not documented anywhere.

The node serves reads but every post fails with unknown fid

A shard did not restore. KEY_ADD routes to shard 0 unconditionally, and the block engine there needs your account's registration record — so with shard 0 missing, every write fails, for every account.

The confusing part is that everything else works, including /v1/validateMessage, which checks structure rather than shard state.

Confirm with ./snapstart node ready. A shard showing a delay in the millions never restored and will not recover on its own — ours was closing at ~390 blocks/min against a 49.5 million block deficit, about 85 days. Wipe and re-bootstrap inside the timing window.

The restore panicked with a 404
Failed to download .rocks.snapshot/shard-0/chunk_0001.bin: HTTP 404
thread 'main' panicked at src/main.rs:422

The snapshot it was downloading got replaced and deleted mid-restore. Start again inside the timing window. Restarting without wiping does not fix it: the node sees a non-empty database and skips the snapshot entirely.

provision fails: "not eligible for Free Tier"

Your AWS account is on the Free Plan, which refuses instance types large enough to run Snapchain. Upgrade to a paid plan in Billing — credits still apply.

resource_limit_exceeded on Hetzner

New Hetzner accounts cap volume storage at 1024 GB, below what the restore peaks at. Request an increase, or use AWS.

reactionsByCast returns 400 for a type the API itself reports

Message bodies report REACTION_TYPE_LIKE, but the query parameter wants Like. Feeding back the value you just read gets you unknown variant `REACTION_TYPE_LIKE`, expected one of `None`, `Like`, `Recast`. Same for Recast.

/v1/info returns nothing

Normal for the first several hours — the node does not serve until the restore completes, so the obvious health check reads as "broken" for most of a day. Watch ./snapstart node status instead.

Posts return 200 but never appear

The node accepted your message and failed to gossip it — most likely it has no peers yet. Wait a few minutes after startup. This is why the client confirms by reading back rather than trusting the status code.

Permission denied deleting node data

The container writes as root. On AWS you log in as ubuntu, so cleanup needs sudo. On Hetzner you are already root and never see this.

Honest limitations

Worth knowing before you spend a day on this.

  • The restore took 13h 22m and 9h 09m on two runs, and interruption is worse than it sounds. A restart does not resume a shard whose snapshot download failed — it sees a non-empty .rocks, skips the snapshot, and starts syncing that shard from genesis, which takes weeks. You have to reload that shard deliberately. snapchain#1044.
  • It needs 2 TB at peak for a database that settles around 557 GB, because snapshot staging is not reclaimed until the end. We proposed a fix.
  • Writing through someone else's validator is not an option — they require authentication. Your own read node is the only keyless write path.
  • This is a tutorial client, not a product. It reads a feed, posts, follows, reacts and filters by source. It is not trying to be Warpcast.

Giving back

Corrections we filed upstream.

Where the documentation disagreed with what we measured, we reported it rather than silently working around it.

FindingStatus
Registration costs ~$0.20, not ~$10PR #1047
Snapshot staging never reclaimed — 961 GB retainedPR #1046
Grafana and statsd bind to all interfaces by defaultPR #1048
/v1/signersByFid missing from the API referencePR #1049
A rotated snapshot breaks writes, silentlyissue #1044