0xDocs 0x.limo

Docs / P2P and the swarm

Mounts, mirror and readahead

Partly live

Put a game's big external data tree (tens of GB, served by your own server) on the swarm in 1 MiB pieces with mounts, and speed it up with the x1.json mirror tier and readahead.

A mount puts a large data tree on the swarm, such as a game's data/ directory with thousands of files and tens of GB, without copying it into the site. 0x publish hashes the tree where it is, in 1 MiB pieces, and writes small index files (shards) next to the swarm manifest. When the game asks for a byte range of a mounted file, the service worker fetches only the pieces that cover that range. Each piece comes from the worker's memory, the page's kept copy, a starter pack on disk, other players, the mirror, or your origin, in that order.

Two x1.json settings go with mounts. mirror adds a fast HTTPS copy of the swarm's files that the worker asks before your origin. readahead warms the next pieces of a file that is read in order.

Self-hosted sites only, runtime 0.2.x

Mounts, mirror and readahead need the site runtime from @0xlimo/swarm-sw 0.1.8 (runtime 0.2.6). They work only for sites you host yourself with 0x publish --to:

  • 0x hosting doesn't take mounts. Its site builder still installs runtime 0.1.4, and the x1.json it writes has no mirror or readahead.
  • The 0x CLI: --mount, --mount-exclude, --mount-cache and the mounts key in 0x.config.json are in the CLI source (x1-sdk main), but not in any published @0xlimo/cli (0.1.7 and earlier). The published CLI also installs runtime 0.1.3.
  • Pieces from other players need a meshtorrent browser client with fetchRanges. The @x1/meshtorrent 0.2.1 on the registry doesn't have it. Without it, mounts still work, but every piece the page doesn't already have comes from the mirror or your origin.

#When to use a mount

Use a mount when:

  • the data is too big for a site: more than the 64 MiB per-file limit, or more files and bytes than you want to copy into every version;
  • your own server already serves it at a URL prefix, such as https://game.example/data/;
  • the game reads parts of files with Range requests (an engine reading archives, streaming a world), not whole files.

For ordinary assets, use the normal swarm policy. A file under a mount prefix can't also be in the site: the publish fails if it is.

#Publishing a mount

From the command line (repeat --mount for several trees):

bash
0x publish ./web --to /srv/www/game \
  --mount /data/=/srv/gamedata \
  --mount-exclude "**/*.idx" --mount-exclude "cache/**" \
  --mount-cache ./.x1

Or in 0x.config.json. A relative dir is resolved against the game directory:

json
{
  "id": "my-game",
  "mounts": [
    { "prefix": "/data/", "dir": "../gamedata", "exclude": ["**/*.idx", "cache/**"], "minBytes": 16384 }
  ]
}
Key / flagTypeDefaultMeaning
prefix / --mount <prefix>=<dir>stringrequiredThe URL path your server serves the tree at, under the site root, such as /data/. A leading and trailing / is added. / alone, /./ and /../ are refused. Two mounts can't have the same name (the prefix with / stripped and other characters turned into _).
dirpathrequiredThe directory on disk. Symlinks are followed, and each directory is walked once.
exclude / --mount-exclude <glob>glob[]noneFiles that stay on the origin. Globs are relative to the mount and matched case-insensitively (** any depth, * within a name, ? one character). --mount-exclude applies to every --mount on the command line, not to the mounts in 0x.config.json.
minBytesnumber16384Smaller files stay on the origin. It can be set only in 0x.config.json.
--mount-cache <dir>path<game dir>/.x1Where the piece hashes are kept between publishes (mount-<name>.json).

Mounts need --to. 0x publish --mount without --to fails with --mount needs --to: an external data tree is only for a self-hosted site. A hosted publish ignores mounts in 0x.config.json. An origin-only site ("swarm": false) can't have mounts.

The publish prints one line per mount, then a summary when the hashing is done:

text
0x publish: mount /data/ <- /srv/gamedata: 8412 files (37 excluded, 1290 under 16384 B), 8412 to hash (41.7 GiB)
0x publish: mount /data/: hashed 12.3 / 41.7 GiB (610 MiB/s)
0x publish: mount /data/: 8412 files, 42803 pieces, 41.7 GiB, 91 pages (70 s)

Hashing runs on up to 8 threads (one fewer than the CPU count). The cache is keyed by path, size and modification time. On the next publish, only files that changed are hashed again, and files that left the tree are dropped from the cache.

#What a mount writes

Under swarm/ in the site:

PathContent
manifest.json mountsOne item per mount: {prefix, base: "x/<name>/", pieceLen: 1048576, pagePieces: 512, buckets: [256 names or null], files, pieces, bytes, packs: true}
x/<name>/b/<hh>.<sha12>.jsonBucket hh: the files whose lower-cased path hashes there (FNV-1a 32, & 255). {v: 1, files: {"<lower-case path>": {size, id, pages, pack, n?, alt?}}}. id is the file's SHA-256, pack its pack torrent's infohash, n its name on disk when that isn't all lower case, and alt: 1 when other files differ from it only in case.
x/<name>/p/<sha16>.jsonA page of up to 512 pieces: {v: 1, pieces: [[sha256, infoHash], …]}
x/<name>/packs.json{v: 1, packs: {"<pack infoHash>": "<path under the prefix>"}}: where a seeder or mirror can fetch each pack from your origin (<origin><prefix><path>)
t/<pack infoHash>.torrentOne torrent per mounted file, its bytes named <sha256><ext>, with 1 MiB pieces, so piece i of the torrent is mount piece i

Bucket and page names carry their content hash, so a cache in front of the site can keep them forever. The manifest names the current ones.

#How a mounted file is served

The worker matches a request's path against each mount's prefix, case-insensitively, and looks the file up in its bucket. Buckets are kept while the manifest names them, and the last 96 pages are kept as well.

  • A Range request is answered from the pieces that cover it. The page's swarm is asked only for pieces the page keeps or somebody is known to seed (from the tracker's /p2p/seeds, looked up in the background, never inside a read), or for every piece when the page holds a starter pack. It gets 400 ms for them. Whatever is missing then is fetched in one request per run of consecutive pieces: first from the mirror, then from your origin. After 3 reads in a row where the page missed the 400 ms, it is left out for 30 s.
  • An aligned read that nobody seeds, such as an engine reading whole 1 MiB blocks, is streamed straight from the mirror or origin to the game. A copy is split into pieces for the memory and the page.
  • A request without Range is streamed 8 pieces at a time, never held whole. If a window fails in the middle, that window comes from the origin.
  • A file the shards don't list (excluded, too small, a case twin): a page with a starter pack is asked for the whole file first (400 ms), then the request goes to your origin as it is.
  • Several ranges in one request, or a shard that fails to load: the request goes to your origin as it is.

The origin and the mirror are trusted like your server: their bytes answer the read at once. Each piece is checked against its SHA-256 after that. Only matching pieces are kept in the worker's memory and handed to the page to keep and seed (at most 8 MiB/s, because the page's main thread is the game's too). A piece that comes from a peer or from the page's cache is checked before the game gets it.

Responses carry X-X1-Source: swarm | origin | mixed and X-X1-Pieces: memo=…,cache=…,swarm=…,origin=… (mirror pieces count as origin), plus the same COEP/COOP/CORP headers as other swarm responses.

#What your server must do

  • Serve the tree at the prefix, with single Range requests answered 206 and a correct Content-Range. A server that answers 200 makes every read fetch the whole file.
  • Serve each file under its name on disk. Paths match case-insensitively: whatever case the game uses, the worker asks your server for the name as published (n, or the lower-case path). A file whose name differs from another only in case is served by the origin only, under its exact name.
  • Keep the tree as published. A changed file fails its piece hashes, so it is served by the origin but never kept or seeded, until you publish again.

#mirror: a fast copy before the origin

json
{ "manifest": "swarm/manifest.json", "timeoutMs": 20000, "mirror": "https://mirror.example" }
KeyTypeDefaultMeaning
mirrorstringnoneAn https:// base URL. Other schemes are ignored, and trailing / are removed.

The mirror is any HTTPS server that answers GET <mirror>/file/<infoHash> with the bytes of that torrent's single file, and single Range requests with 206. A seedbox that holds the site's torrents is the intended mirror. The worker uses it:

ForWhenCheck
A manifest file of 16 MiB or more (an engine's wasm)before the page's swarmsize and SHA-256 before answering
Any other manifest fileafter the swarm failed, before the originsize and SHA-256 before answering
A run of missing mount piecesbefore the origin (Range on the file's pack infohash)like origin bytes, after answering
A streamed aligned mount readbefore the originlike origin bytes, after answering

Every mirror request has a timeout of max(30 s, bytes / 1000) ms, a floor of about 1 MB/s. After any failure, such as an HTTP error, a timeout, a short range or bad bytes, the mirror sits out for 60 s and the origin is asked. Mirror bytes are reported as origin bytes in transfer reports.

The worker calls the mirror from your site's origin, so the mirror must send CORS headers: Access-Control-Allow-Origin for your site, and Access-Control-Expose-Headers: Content-Range for range answers. Mount pieces from the mirror reach the game before they are hash-checked, so use only a mirror you trust as much as your origin.

#readahead: warm the next pieces

json
{ "manifest": "swarm/manifest.json", "timeoutMs": 20000, "readahead": true }
KeyTypeDefaultMeaning
readaheadbooleanfalseOnly true turns it on

After each ranged read of a mounted file, the worker fetches the next 6 pieces (6 MiB) of that file in the background into its 64 MiB memory, by the same path as a read. Turn it on for a game that reads its data in order, such as a world streamed while the player moves. Leave it off when reads jump around, because readahead then fetches bytes nobody uses. It applies only to mounted files.

#Setting mirror and readahead

0x publish --to writes x1.json with manifest, timeoutMs and, if given, tracker. Add mirror and readahead to the published x1.json yourself. Each 0x publish --to writes the file again, so add them after every publish. boot.js reads x1.json on every page load and passes the values to the worker's registration URL (readahead=1, mirror=<url>).

#Limits

WhatValue
Piece size1 MiB (fixed)
Pieces per page512
Buckets per mount256
Smallest mounted fileminBytes, 16 KiB by default
Page swarm budget per read400 ms
Peer wait per piece / longest peer download1.5 s / 4 s
Pages kept by the worker96
Worker memory (whole files and pieces)64 MiB
Origin pieces handed to the page8 MiB/s

See also: starter packs, which answer a mount's reads from a file on the player's disk, and the @0xlimo/swarm-sw reference.