# Hosting your own Pop Boll corpus *Draft, 9 November 2026. This page describes how an artist keeps their recordings on their own server and lets Pop Boll pull from it. The folder layout and the discovery file are fixed enough to build to today. The Pop Boll side (the automatic handshake, heartbeat and hiding) is not built yet: until it is, Graham does the same steps by hand after you email him. Preparing the folder as described here is useful now and will not need redoing.* This page is written for the agent that sets up the artist's server, and for the artist reading over its shoulder. Part 1 is the plain version. Part 2 is the specification. Part 3 is a checklist. --- ## Part 1. The plain version ### Why host it yourself Your recordings stay on a server you control. Pop Boll reads them once to transcode them, keeps what your licence allows and nothing more, and checks in now and then to see the folder is still there. If you take the folder down, your corpus is hidden on every Pop Boll after 7 days of failed check-ins (on a Pop Boll that is offline, when it next reaches the network). If you put it back, it comes back. You never have to ask anyone to delete anything, and you can add albums by adding files. ### What you need - A web server you control, reachable over HTTPS with a valid certificate, that will stay up. A static file host or a storage bucket with a read-only key works. A home computer that sleeps does not. - A private folder on it, with a login. Pop Boll fetches with HTTP Basic auth or a bearer token. - Your recordings: MP3 preferred, or WAV, FLAC, AIFF or M4A, flat or in album subfolders. Multiple albums' worth. - The current Pop Boll Shake (version 4), filled in, in the folder. - Two small files the agent writes for you: a manifest listing every file with its size and hash, and a discovery file that tells Pop Boll what is here and what you agreed to. - Cover art, if you like. Optional. ### What happens, step by step 1. **You set up the folder** (Part 2 has the layout). 2. **You tell Pop Boll where it is.** Email the folder's address with "Sounds good to me!" the way the Shake asks. The login does not go in the email: you enter it in the Pop Boll backstage on bollawax.org, where artists manage their corpora, or Graham sends you a one-time form until that page exists. 3. **Pop Boll fetches and checks.** It reads the discovery file, the Shake and the manifest, checks that the Shake is the real text with your choices ticked, checks a sample of the hashes, and archives the Shake, the discovery file and the fetch record as evidence in Pop Boll's signed corpus archive. Your corpus gets an id like `0007-YOURNAME`. 4. **Pop Boll transcodes, for free.** The pipeline downloads the recordings once, makes the MIDI and notes, and keeps what your scope allows: for MIDI-only, nothing but the transcriptions; for Full, a copy of the recordings too, so playback does not depend on your server being fast. 5. **Your corpus goes live.** It appears in the corpus list that every Pop Boll reads. 6. **Pop Boll checks in.** About every six hours it asks your server whether the discovery file is still there and unchanged. That is one tiny request. If the manifest changed, it fetches the new files and transcodes them. Nothing else touches your server. ### If your server goes away After 7 days of failed check-ins, your corpus is **hidden**: it leaves the list, every Pop Boll drops it within one refresh (a Pop Boll that is offline, when it is next online), playback stops, and the musicians stop studying it. Nothing is deleted. The hide lasts until you tell Pop Boll the server is back (email, or the backstage "I'm back" button), and then the same corpus comes back under the same id, with nothing re-transcoded if the files are the same. If you want it gone rather than hidden, set `"status": "revoked"` in the discovery file or just email. That starts the takedown in the Shake: new songs stop at once, and the recordings and transcriptions are deleted within about 30 days. If you want a pause, set `"status": "paused"`. It hides at the next check-in and keeps everything. ### Adding and removing tracks Add files, add their lines to the manifest, save. The next check-in sees the manifest changed and fetches the new tracks. Remove a track the same way: its line leaves the manifest, and the track leaves the list and the stored copy within a day. Notes the musicians already made from it stay, as the Shake describes. ### The free transcoding, and its limits Transcoding runs on Pop Boll's cloud and is paid for by Pop Boll. To keep that true: one corpus per artist at a time in the queue, up to about 20 GB of recordings per corpus, and files that are real recordings (a folder of silence or noise is refused). If you have more than that, say so in the email. ### Hosts that work and hosts that do not | Works | Does not | | :--- | :--- | | Your own web server (nginx, Apache, Caddy) with a protected folder | Dropbox, Google Drive and iCloud share links (they redirect to HTML pages) | | A storage bucket (S3, Google Cloud Storage, Cloudflare R2, Backblaze B2) with a read-only key, served over plain HTTPS GET | Bandcamp (no file access) | | Nextcloud or any WebDAV server, as long as plain GET with Basic auth works | A server that needs a browser login, a cookie or JavaScript | | A static host with Basic auth in front of a folder | A home machine that sleeps, or a residential connection without a stable address | --- ## Part 2. The specification ### 2.1 Folder layout Everything lives under one base URL, called `` below. The base URL ends with a slash. ``` popboll-corpus.json discovery file (required) POP_BOLL_SHAKE_V4_11112026.md the agreement, with your two boxes ticked (required) MANIFEST.tsv every recording: path, size, sha256 (required) art/ optional cover images (jpg or png), one per album folder or one for the corpus flat, or in album subfolders; any depth up to 3 ``` Rules: - HTTPS only, with a certificate a normal browser trusts. Plain HTTP is refused. - Every file is served by plain `GET`, and the server answers `HEAD` for at least the discovery file and the manifest. `HEAD` on the discovery file must return `ETag` or `Last-Modified`. - The same credential covers everything under ``. - No redirects away from ``'s host. A redirect inside the same host is followed once. - Paths in the manifest are relative to ``, use `/` as the separator, contain no `..`, no leading `/`, and no control characters. Spaces are allowed and are percent-encoded when fetched. - Recording formats: `.mp3`, `.wav`, `.flac`, `.aif`, `.aiff`, `.m4a`. Anything else in the manifest is refused. - Total size up to about 20 GB; the discovery file up to 64 KiB; the manifest up to 4 MiB. ### 2.2 The discovery file: `popboll-corpus.json` UTF-8 JSON, one object. Keys not listed here are ignored. ```json { "schema": "popboll.origin/1", "corpus": { "name": "The Band", "artist": "The Band", "legal_name": "Pat Doe", "contact_email": "pat@example.com", "description": "Three albums and the 2019 EP.", "website": "https://theband.example" }, "licence": { "shake_version": 2, "shake_file": "POP_BOLL_SHAKE_V4_11112026.md", "scope": "full", "hosting": "self", "accepted_by": "Pat Doe", "accepted_email": "pat@example.com", "accepted_at": "2026-11-09T18:00:00Z", "statement": "Sounds good to me!" }, "files": { "manifest": "MANIFEST.tsv", "art": ["art/cover.jpg"] }, "auth": { "type": "basic" }, "status": "live", "notes": "Anything you want Graham to read." } ``` | Key | Required | Meaning | | :--- | :-: | :--- | | `schema` | yes | exactly `popboll.origin/1` | | `corpus.name` | yes | the corpus name Pop Boll shows and credits | | `corpus.artist` | yes | the artist or project name | | `corpus.legal_name` | yes | the person who agreed to the Shake | | `corpus.contact_email` | yes | the address the Shake email came from | | `licence.shake_version` | yes | `2`: the version of the terms. Version 3 of the Shake changed only the address to write to, so its terms are version 2's and this number stays `2` | | `licence.shake_file` | yes | the filled-in Shake, in the folder | | `licence.scope` | yes | `full` or `morpus_only` (MIDI-only); must match the box ticked in the Shake | | `licence.hosting` | yes | `self` | | `licence.accepted_by`, `accepted_email`, `accepted_at`, `statement` | yes | who agreed, when (UTC, ISO 8601), and the words they used | | `files.manifest` | yes | the manifest path, normally `MANIFEST.tsv` | | `files.art` | no | cover images, relative paths | | `auth.type` | yes | `basic`, `bearer` or `none`. Never put the secret here | | `status` | yes | `live`, `paused` or `revoked` (§2.5) | | `notes` | no | free text | Pop Boll compares the Shake file against the published text of the current Shake. Only the two checkbox lines and the signature block may differ. A Shake with other edits is refused with a message that says where. ### 2.3 The manifest: `MANIFEST.tsv` One line per recording, tab-separated, LF line endings, UTF-8, sorted by path, no header: ``` pathsize_bytessha256_hex ``` For example: ``` 2019-ep/01 First Song.mp3 6213504 3f1c…e9 2019-ep/02 Second Song.mp3 5890221 a07d…11 album-one/01 Opener.mp3 7120033 c9b2…54 ``` Cover images are not in the manifest; they are named in `files.art`. The manifest lists recordings only, so that its hash changes exactly when the music changes. Generate it with the standard tools on the server, from inside the folder: ```bash find . -type f \( -iname '*.mp3' -o -iname '*.wav' -o -iname '*.flac' -o -iname '*.aif' -o -iname '*.aiff' -o -iname '*.m4a' \) \ | sed 's|^\./||' | LC_ALL=C sort \ | while IFS= read -r f; do printf '%s\t%s\t%s\n' "$f" "$(stat -c %s "$f" 2>/dev/null || stat -f %z "$f")" "$(shasum -a 256 "$f" | cut -d' ' -f1)"; done > MANIFEST.tsv ``` Or with Python, which behaves the same on every platform: ```bash python3 - <<'EOF' import hashlib, os exts = {".mp3", ".wav", ".flac", ".aif", ".aiff", ".m4a"} rows = [] for root, dirs, files in os.walk("."): dirs[:] = [d for d in dirs if not d.startswith(".") and d != "art"] for name in files: if os.path.splitext(name)[1].lower() in exts: p = os.path.relpath(os.path.join(root, name), ".").replace(os.sep, "/") h = hashlib.sha256() with open(p, "rb") as fh: for chunk in iter(lambda: fh.read(1 << 20), b""): h.update(chunk) rows.append((p, os.path.getsize(p), h.hexdigest())) rows.sort() with open("MANIFEST.tsv", "w", encoding="utf-8", newline="\n") as out: for p, size, digest in rows: out.write(f"{p}\t{size}\t{digest}\n") print(len(rows), "recordings") EOF ``` ### 2.4 Authentication - `basic`: HTTP Basic over HTTPS. Make a dedicated read-only user for Pop Boll; do not reuse your own. - `bearer`: an `Authorization: Bearer ` header. Same idea: one token, read-only, for Pop Boll. - `none`: the folder is public. Allowed, but then anyone can download your recordings; on popboll.com Pop Boll will still only stream them to paid supporters. The secret is entered once in the bollawax.org backstage (artist tier) and stored encrypted. It is never written to the discovery file, the manifest, an email or a corpus manifest. To rotate it, enter the new one in the backstage first, then change it on the server; Pop Boll retries with the new credential on the next check-in, and a 401 for seven days counts as unreachable, not as revoked. ### 2.5 Status and the handshake, from Pop Boll's side The states a self-hosted corpus moves through: | State | Meaning | How it is left | | :--- | :--- | :--- | | `submitted` | the email arrived and the credential is in the backstage | the first fetch | | `verifying` | discovery file, Shake and manifest fetched; Shake text checked; a sample of hashes checked | pass → `attesting`; fail → `refused` with a reason the artist sees | | `attesting` | the Shake, the discovery file and the fetch record (URL, headers, time, hashes) are archived as evidence in the signed corpus archive with the first edition | done → `transcoding` | | `transcoding` | every recording is fetched once, hashed against the manifest, transcoded; the scope decides what is kept | done → `live` | | `live` | listed; heartbeat every 6 hours: `HEAD` on the discovery file, `GET` if the ETag changed, then the manifest | manifest change → additions and removals processed in place; 7 days unreachable → `hidden`; `status: paused` → `hidden`; `status: revoked` → `revoking` | | `hidden` | not listed; devices drop it within one refresh; playback and study stop; everything kept | the artist says it is back, or sets `status: live` and asks → re-verify → `live` under the same id | | `revoking` | the Shake's takedown: new songs stop at once, listing dropped, deletion of recordings and transcriptions within about 30 days | → `retired` | | `retired` | gone; the id is never reused; the locker keeps the evidence of what was licensed and deleted | never | Removal of one track: when a path leaves the manifest, the track leaves the listing and any stored copy within a day. Its morpus pair is retired from study at the next DNA rebuild. Cells musicians already extracted stay, as the Shake says. What Pop Boll stores, by scope, for a self-hosted corpus: | | Full | MIDI-only | | :--- | :--- | :--- | | Recordings | mirrored into Pop Boll's private bucket after the first fetch, so playback does not load your server | fetched once into a one-day staging bucket, transcoded, deleted | | Cover art | kept | not kept | | Transcriptions and notes | kept | kept | | Your Shake, discovery file and fetch record | archived and signed in the signed corpus archive | same | | Your credential | encrypted, in the backstage store | same | ### 2.6 Where this lands in Pop Boll's own records The corpus manifest Pop Boll writes (`corpus.json`) gains an `origin` block: ```json "origin": { "kind": "self_hosted", "base": "https://files.theband.example/popboll/", "discovery_etag": "\"5f2a…\"", "manifest_sha256": "…", "last_ok": "2026-11-10T06:00:02Z", "failures": 0, "state": "live", "hidden_since": null } ``` The licence block records `"hosting": "self"` and `"shake_version": 2`. The archived edition carries `evidence/popboll-corpus.json`, `evidence/fetch_record.json` and `licenses/POP_BOLL_SHAKE_V4_11112026.md`. ### 2.7 The validator Planned: `POST https://bollawax.org/api/origin/check` with `{"base": ""}` and the credential already in the backstage, returning a report with one line per rule in §2.1 to §2.4. Until it exists, Part 3 is the validator. --- ## Part 3. Checklist for the agent Run from a machine that is not the server, with the credential in `$CRED` (`user:pass` for basic) and the base URL in `$BASE` (ending with a slash). 1. The certificate is valid and plain HTTP is not used: ```bash curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' -u "$CRED" "${BASE}popboll-corpus.json" ``` Expect `200 0`. 2. `HEAD` works on the discovery file and returns an ETag or Last-Modified: ```bash curl -sS -I -u "$CRED" "${BASE}popboll-corpus.json" | grep -iE '^(etag|last-modified):' ``` 3. The discovery file is valid JSON with the required keys, `schema` is `popboll.origin/1`, `status` is `live`, and `auth.type` matches how you actually protected the folder: ```bash curl -sS -u "$CRED" "${BASE}popboll-corpus.json" | python3 -c ' import json,sys; d=json.load(sys.stdin) assert d["schema"]=="popboll.origin/1", d["schema"] for k in ("name","artist","legal_name","contact_email"): assert d["corpus"][k] L=d["licence"]; assert L["shake_version"]==2 and L["scope"] in ("full","morpus_only") and L["hosting"]=="self" for k in ("accepted_by","accepted_email","accepted_at","statement"): assert L[k] assert d["files"]["manifest"] and d["auth"]["type"] in ("basic","bearer","none") and d["status"] in ("live","paused","revoked") print("discovery ok:", d["corpus"]["name"], L["scope"], d["status"])' ``` 4. The Shake is present and is the current one (version 4) with exactly one box ticked in each "Choose one": ```bash curl -sS -u "$CRED" "${BASE}POP_BOLL_SHAKE_V4_11112026.md" | grep -c '^- \[x\]' ``` Expect `2`. 5. The manifest downloads, has three tab-separated columns, is sorted, lists only allowed formats, and every path exists on the server with the stated size: ```bash curl -sS -u "$CRED" "${BASE}MANIFEST.tsv" -o /tmp/MANIFEST.tsv python3 - <<'EOF' import sys, re rows=[l.rstrip("\n").split("\t") for l in open("/tmp/MANIFEST.tsv", encoding="utf-8")] assert all(len(r)==3 for r in rows), "three columns" paths=[r[0] for r in rows]; assert paths==sorted(paths), "sorted" assert all(re.search(r"\.(mp3|wav|flac|aif|aiff|m4a)$", p, re.I) for p in paths), "formats" assert not any(p.startswith("/") or ".." in p.split("/") for p in paths), "paths" print(len(rows), "rows,", sum(int(r[1]) for r in rows)//(1<<20), "MiB") EOF ``` Then spot-check three files by `HEAD` and compare `Content-Length` to the manifest's size. 6. One full hash check on the smallest file: ```bash curl -sS -u "$CRED" "${BASE}$(python3 -c 'import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))' "")" | shasum -a 256 ``` 7. A wrong credential is refused (so the folder really is private): ```bash curl -sS -o /dev/null -w '%{http_code}\n' -u "nobody:wrong" "${BASE}MANIFEST.tsv" ``` Expect `401` or `403`, unless `auth.type` is `none`. 8. Email the Shake (the current one is at [bollawax.org/pdf](https://bollawax.org/pdf)) and `` from the artist's own address to shake@bollawax.org with the words in the Shake. Enter the credential in the backstage, or wait for the one-time form. Do not put the credential in the email. What is built on 10 November 2026 (Bollawax 0.2.4): the discovery file has a schema, `origin`, in the open Bollard SDK, and the SDK can fetch and check one (`bollardeck.corpus.fetch_origin`); the backstage records that a corpus is self-hosted ("I host them myself", `hosting` in the licence block). What is not built: the hub states in §2.5, the heartbeat, `hidden`, the `origin` block, the backstage credential store, and a validator you can run on your folder. So Pop Boll does not pull from anyone's server yet, and until it does a self-hosting artist uploads the zip as well. Everything in Part 2 that the artist's server must do is final enough to build against.