Farcaster on infrastructure you own
Run your own
Farcaster client.
A Snapchain read node you own, and a client that reads from it. No API keys, no hosted provider, nobody's rate limit.
Quickstart
Clone it, then run three commands.
Everything below explains what each command does, what the protocol is doing underneath, and the ways it goes wrong — all of which we hit while building it.
git clone https://github.com/webthethird/snapstart && cd snapstart
./snapstart provision
./snapstart node up
./snapstart client create
Hourly billing. Destroy it when you're done.
Docs say two. Two restores timed end to end: 13h 22m and 9h 09m, same instance type.
The node holds no credentials once running.
Every number here measured on a live node.
Read this before step 1
The restore can lose a race with the snapshot it is reading.
Your node fetches all three shard manifests when it starts, then restores
shards one at a time, and shard 0 is always last
(shard_ids.push(0), src/main.rs:413). Snapshots are
replaced daily and the previous one is eventually deleted. A restore slower
than that gap reaches shard 0 holding a pointer to something that no
longer exists.
5,692 chunks. Two runs on an m5.xlarge with gp3: 13h 22m and 9h 09m. Download throughput is what varies.
Restored last, so it has waited the longest. It is also the smallest — 371 chunks, about 30 GB.
Repeated Failed to download. The panic that follows is not unique to this — see below.
What we measured, and what we did not
We hit this twice while building, so the hazard is real. But we never pinned down the one number that would let us hand you a safe start time, and we would rather say so than publish a window that only looks authoritative. An earlier draft of this page did exactly that; shard 2 then moved two hours in a day and broke it.
Generation time, from each snapshot's own timestamp
shard 0 10-17 05:18:45 10-18 05:18:46 10-19 05:18:52 10-20 05:18:49
steady to within 7 seconds across four days
shard 1 10-17 13:23:10 10-19 13:29:15
+6m05s over two days
shard 2 10-17 08:04:08 10-19 08:04:38 10-20 10:04:58
+30s over two days, then +2 HOURS in one
latest.json is updated LATER than that timestamp
10-20 06:51 still served the 10-19 shard-0 key
10-20 07:45 first served the 10-20 key, generated 05:18:49
The old snapshot is deleted -- but we never caught WHEN
10-20 06:38 node downloaded all 371 shard-0 chunks from the 10-19
snapshot successfully, hours after it was "rotated"
10-20 11:48 those same chunks return 404
10-18 13:38 404s on the 10-17 snapshot, then the panic
Deletion is the event that actually matters, and it is the one we did not characterise. So treat any start-time rule — including one you might derive from the table above — as a guess. Our own node was saved by winning this race, not by planning for it.
What to do instead
Recovery is cheap, which is the useful part. A missing shard can be reloaded
on its own — shard 0 is 30 GB and takes minutes, against
10–15 hours to redo the whole restore. Fetch the shard's
current manifest, download its chunks, verify them, and untar into
.rocks while the node is stopped. The technique is from
reload-shard.sh
by Sats100M (MIT), linked from
snapchain#1044.
PR #1045
fixes the root cause upstream by refetching each shard's metadata immediately
before downloading it.
Whatever you do, check the manifests yourself rather than trusting times on a web page — including this one:
R2=https://pub-d352dd8819104a778e20d08888c5a661.r2.dev/FARCASTER_NETWORK_MAINNET
for s in 0 1 2; do
curl -s "$R2/$s/latest.json" | python3 -c "
import sys, json, datetime as d
t = json.load(sys.stdin)['timestamp'] / 1000
print('shard $s', d.datetime.fromtimestamp(t, d.timezone.utc))"
done
A node that loses this race fails loudly. What is quiet is the aftermath: it still answers reads, so it looks healthy while rejecting every write.
A panic does not mean you lost the race
We assumed for two days that the panic at the end of a restore was this bug. It is not. Every successful restore ends in a panic, for an unrelated reason, and the two look identical from the outside.
called `Result::unwrap()` on an `Err` value:
IoError(Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" })
download_snapshots finishes by calling
remove_dir_all on its staging directory, which deletes the
contents and then the directory itself — but
.rocks.snapshot is a bind mount in snapchain's own compose
file, and removing a mount point returns EBUSY. That error
reaches the same .unwrap() a rotation 404 would, so both
produce the same crash at the same line.
It is invisible in the documented setup: the restart policy brings the node back, it finds a complete database, and it serves. Run without one and the process exits non-zero after a 9–14 hour restore with a perfectly good database on disk. We reproduced it on two nodes, the second with zero 404s, and reported it as snapchain#1076.
Concepts
What you are actually building.
If you have run an Ethereum node, you already understand this. You can read chain
data from Alchemy with an API key and a rate limit, or run Reth yourself and query
localhost with nobody's permission. Farcaster has the same split:
Neynar is Alchemy, a Snapchain read node is your own Reth.
| Ethereum | Farcaster |
|---|---|
| Reth / Geth full node | Read node — stores all state, serves queries, cannot propose blocks |
| Validator | Validator — permissioned, gated on a key in a registry contract |
eth_sendRawTransaction | submitMessage |
eth_syncing | GET /v1/info |
| EOA private key | Custody key — owns the account |
| Session key | App key — scoped, expiring, revocable |
You cannot run a validator — the validator set lives in a contract and you are not in it. That is fine. A read node stores full state and serves every read API, which is everything a client needs.
Three identities, not one
A uint256 in a registry contract. No cryptography of its own.
secp256k1, derived from your seed phrase. Never touches the node.
Ed25519, generated per client. Scoped, expiring, revocable.
An app key is a session key. One account can have many — Warpcast holds one, your client holds another, and revoking one does not touch the rest. That is also why a cast can tell you which client posted it: the signature identifies the software, not just the person.
Command 0
Provision the compute.
./snapstart provision
Four API calls and a boot script: an SSH keypair dedicated to this node, a firewall, an instance, a disk — then cloud-init installs Docker. Nothing mystical.
New AWS accounts land on the Free Plan, which cannot run this. It refuses any instance type that is not free-tier eligible, and the largest eligible type is 2 vCPU / 8 GiB where Snapchain needs 4 / 16. No quota increase fixes it — you have to upgrade to a paid plan. Credits still apply.
Hetzner is a third the price and cannot finish on a new account.
Its default volume quota is 1024 GB and the restore peaks above that — verified
by probe, a resize to 1030 GB is refused. Use
SNAPSTART_PROVIDER=hetzner only if your quota has been raised.
One credential, and the node never sees it
The cloud API key belongs to the provisioning script. provision,
status and destroy need it; node and
client do not. Once running, the node holds no credentials of any kind.
Command 1
Start the read node.
./snapstart node up
Installs upstream's snapchain.sh, writes its config without prompting,
and starts the node. It then restores from a snapshot — the slow part, and the part
with teeth. See the timing window above.
Check readiness properly
./snapstart node ready
"The API responds" and "the database has data" are both true of a broken node. This verifies every shard is present and within 100 blocks of the tip.
shard height delay messages state
0 49007206 1 0 ok
1 49007206 1 458606290 ok
2 48842741 1 455621742 ok
ready: all shards present and within 100 blocks of the tip
Command 2
Create the client.
./snapstart client create
Asks for your seed phrase, uses it once to authorize an app key, then forgets it. From that point the client holds only the app key.
Use a throwaway account. Not one you care about.
A client holding an app key can post as you. A client holding your seed phrase can transfer your account away, change its recovery address, and spend from the custody wallet. Those are not the same risk.
Registering a fresh account costs about $0.20 — the docs say $10,
but IdGateway.price() returns roughly 0.0001 ETH, including the
first storage unit. There is no good reason to test with an account that matters.
Only paste a seed phrase into software whose source you have read, from people you
have reason to trust. That includes this project. The code that
touches the phrase is
scripts/app-key.ts
— about 140 lines, worth the three minutes.
What issuing an app key actually does
Registering a key used to need an Optimism transaction. Under FIP-272 it does not:
you submit a KEY_ADD message authorized by an EIP-712 signature from
your custody address. No gas, no transaction. Three signatures are
involved:
- a SignedKeyRequest — the app saying "I asked for this key"
- a KeyAdd signature from your custody address, authorizing the key, its scopes and lifetime
- the envelope itself, signed by the new key — proof whoever submits it holds it
Only what the client sends. A key scoped to casting cannot follow anyone.
Sliding — it refreshes on use, so abandoned keys expire by themselves.
Strictly increasing per account, so an old authorization cannot be reused.
Writing this yourself? The gasless EIP-712 domain is
{ name: "Farcaster KeyAdd", version: "1", chainId: 1 } with no
verifyingContract, and it sits beside near-identically named onchain
helpers. Using the wrong one produces a signature that verifies locally and is
rejected by the network. Also, deadline is in Farcaster
seconds (epoch 2021-01-01), not Unix seconds.
Writes are confirmed, not assumed
A node returns 200 once your message is well-formed and handed to the
gossip layer — then discards what happens next. A node with no peers logs
InsufficientPeers, drops your cast, and still returns 200.
So the client never trusts the status code: it knows the message hash before
submitting and polls until the node reports it back.
Do not post immediately after starting your node. It needs gossip
peers first, and until it has them your writes vanish with a 200.
The interesting part
Filter the feed by the client that posted.
Every cast carries the Ed25519 key that signed it. That key's registration records the FID of the app that requested it, and that FID has a profile with a name:
cast.signer → the key's registration → requestFid → that app's name
Which means a feed can be filtered by the software that produced it:
feed: 60 casts from 23 authors
45 warpcast
13 fireflyapp
1 greg
1 unknown client
include ["warpcast"] -> 45
exclude ["warpcast"] -> 15
That is a real feed on a real node, not a mockup. Casts that genuinely cannot be attributed — keys registered before the metadata existed — render as unknown client and are never silently dropped: a filter that hides posts because a lookup failed is a bug that looks like a feature.
The same field, two encodings
requestFid is reported two different ways depending on how the key was
registered, which is easy to miss because the obvious one works:
onchain signer.onChainEvent.signerEventBody.metadata
base64, ABI-encoded SignedKeyRequestMetadata, decode to get requestFid
gasless signer.requestFid
a plain top-level field. `metadata` is absent entirely.
We shipped the bug this creates: decoding only the blob sends every gasless key to unknown client, including the one your own client just issued itself. Nothing errors. The feature simply fails to recognise your own posts, which is exactly the case you are least likely to test. Check both.
Money
What it costs, and how to stop paying.
| AWS (default) | Hetzner | |
|---|---|---|
| Instance | m5.xlarge — 4 vCPU / 16 GB | cx43 — 8 vCPU / 16 GB |
| Disk | 2048 GB gp3 | 1024 GB volume |
| Cost | ~$0.42/hr — ~$10/day | ~€0.14/hr — ~€3.30/day |
| New account can finish? | yes | no — volume quota |
Both bill hourly. Run it for a day or two, then:
./snapstart destroy
It names what it is about to delete, verifies the resources are actually gone before discarding local state, and keeps that state if anything survived — so you can retry rather than being told you have stopped paying when you have not.
Troubleshooting
When it goes wrong.
Every entry here is something we hit. Most are not documented anywhere.
The node serves reads but every post fails with unknown fid
A shard did not restore. KEY_ADD routes to shard 0 unconditionally,
and the block engine there needs your account's registration record — so with shard 0
missing, every write fails, for every account.
The confusing part is that everything else works, including
/v1/validateMessage, which checks structure rather than shard state.
Confirm with ./snapstart node ready. A shard showing a delay in the
millions never restored and will not recover on its own — ours was closing at ~390
blocks/min against a 49.5 million block deficit, about 85 days. Wipe and
re-bootstrap inside the timing window.
The restore panicked with a 404
Failed to download .rocks.snapshot/shard-0/chunk_0001.bin: HTTP 404
thread 'main' panicked at src/main.rs:422
The snapshot it was downloading got replaced and deleted mid-restore. Start again inside the timing window. Restarting without wiping does not fix it: the node sees a non-empty database and skips the snapshot entirely.
provision fails: "not eligible for Free Tier"
Your AWS account is on the Free Plan, which refuses instance types large enough to run Snapchain. Upgrade to a paid plan in Billing — credits still apply.
resource_limit_exceeded on Hetzner
New Hetzner accounts cap volume storage at 1024 GB, below what the restore peaks at. Request an increase, or use AWS.
reactionsByCast returns 400 for a type the API itself reports
Message bodies report REACTION_TYPE_LIKE, but the query
parameter wants Like. Feeding back the value you just read gets you
unknown variant `REACTION_TYPE_LIKE`, expected one of `None`, `Like`, `Recast`.
Same for Recast.
/v1/info returns nothing
Normal for the first several hours — the node does not serve
until the restore completes, so the obvious health check reads as "broken" for most of
a day. Watch ./snapstart node status instead.
Posts return 200 but never appear
The node accepted your message and failed to gossip it — most likely it has no peers yet. Wait a few minutes after startup. This is why the client confirms by reading back rather than trusting the status code.
Permission denied deleting node data
The container writes as root. On AWS you log in as
ubuntu, so cleanup needs sudo. On Hetzner you are already
root and never see this.
Honest limitations
Worth knowing before you spend a day on this.
- The restore took 13h 22m and 9h 09m on two runs, and interruption is worse than it sounds. A restart does not resume a shard whose snapshot download failed — it sees a non-empty
.rocks, skips the snapshot, and starts syncing that shard from genesis, which takes weeks. You have to reload that shard deliberately. snapchain#1044. - It needs 2 TB at peak for a database that settles around 557 GB, because snapshot staging is not reclaimed until the end. We proposed a fix.
- Writing through someone else's validator is not an option — they require authentication. Your own read node is the only keyless write path.
- This is a tutorial client, not a product. It reads a feed, posts, follows, reacts and filters by source. It is not trying to be Warpcast.
Giving back
Corrections we filed upstream.
Where the documentation disagreed with what we measured, we reported it rather than silently working around it.
| Finding | Status |
|---|---|
| Registration costs ~$0.20, not ~$10 | PR #1047 |
| Snapshot staging never reclaimed — 961 GB retained | PR #1046 |
| Grafana and statsd bind to all interfaces by default | PR #1048 |
/v1/signersByFid missing from the API reference | PR #1049 |
| A rotated snapshot breaks writes, silently | issue #1044 |