DRAFT

Hosting your own Pop Boll corpus

Draft, 9 November 2026. This page describes how an artist keeps their recordings on their own server and lets Pop Boll pull from it. The folder layout and the discovery file are fixed enough to build to today. The Pop Boll side (the automatic handshake, heartbeat and hiding) is not built yet: until it is, Graham does the same steps by hand after you email him. Preparing the folder as described here is useful now and will not need redoing.

This page is written for the agent that sets up the artist's server, and for the artist reading over its shoulder. Part 1 is the plain version. Part 2 is the specification. Part 3 is a checklist.

Part 1. The plain version#

Why host it yourself#

Your recordings stay on a server you control. Pop Boll reads them once to transcode them, keeps what your licence allows and nothing more, and checks in now and then to see the folder is still there. If you take the folder down, your corpus is hidden on every Pop Boll after 7 days of failed check-ins (on a Pop Boll that is offline, when it next reaches the network). If you put it back, it comes back. You never have to ask anyone to delete anything, and you can add albums by adding files.

What you need#

• A web server you control, reachable over HTTPS with a valid certificate, that will stay up. A static file host or a storage bucket with a read-only key works. A home computer that sleeps does not.
• A private folder on it, with a login. Pop Boll fetches with HTTP Basic auth or a bearer token.
• Your recordings: MP3 preferred, or WAV, FLAC, AIFF or M4A, flat or in album subfolders. Multiple albums' worth.
• The current Pop Boll Shake (version 4), filled in, in the folder.
• Two small files the agent writes for you: a manifest listing every file with its size and hash, and a discovery file that tells Pop Boll what is here and what you agreed to.
• Cover art, if you like. Optional.

What happens, step by step#

1. You set up the folder (Part 2 has the layout).
2. You tell Pop Boll where it is. Email the folder's address with "Sounds good to me!" the way the Shake asks. The login does not go in the email: you enter it in the Pop Boll backstage on bollawax.org, where artists manage their corpora, or Graham sends you a one-time form until that page exists.
3. Pop Boll fetches and checks. It reads the discovery file, the Shake and the manifest, checks that the Shake is the real text with your choices ticked, checks a sample of the hashes, and archives the Shake, the discovery file and the fetch record as evidence in Pop Boll's signed corpus archive. Your corpus gets an id like 0007-YOURNAME.
4. Pop Boll transcodes, for free. The pipeline downloads the recordings once, makes the MIDI and notes, and keeps what your scope allows: for MIDI-only, nothing but the transcriptions; for Full, a copy of the recordings too, so playback does not depend on your server being fast.
5. Your corpus goes live. It appears in the corpus list that every Pop Boll reads.
6. Pop Boll checks in. About every six hours it asks your server whether the discovery file is still there and unchanged. That is one tiny request. If the manifest changed, it fetches the new files and transcodes them. Nothing else touches your server.

If your server goes away#

After 7 days of failed check-ins, your corpus is hidden: it leaves the list, every Pop Boll drops it within one refresh (a Pop Boll that is offline, when it is next online), playback stops, and the musicians stop studying it. Nothing is deleted. The hide lasts until you tell Pop Boll the server is back (email, or the backstage "I'm back" button), and then the same corpus comes back under the same id, with nothing re-transcoded if the files are the same.

If you want it gone rather than hidden, set "status": "revoked" in the discovery file or just email. That starts the takedown in the Shake: new songs stop at once, and the recordings and transcriptions are deleted within about 30 days.

If you want a pause, set "status": "paused". It hides at the next check-in and keeps everything.

Adding and removing tracks#

Add files, add their lines to the manifest, save. The next check-in sees the manifest changed and fetches the new tracks. Remove a track the same way: its line leaves the manifest, and the track leaves the list and the stored copy within a day. Notes the musicians already made from it stay, as the Shake describes.

The free transcoding, and its limits#

Transcoding runs on Pop Boll's cloud and is paid for by Pop Boll. To keep that true: one corpus per artist at a time in the queue, up to about 20 GB of recordings per corpus, and files that are real recordings (a folder of silence or noise is refused). If you have more than that, say so in the email.

Hosts that work and hosts that do not#

Part 2. The specification#

2.1 Folder layout#

Everything lives under one base URL, called <base> below. The base URL ends with a slash.

<base>
  popboll-corpus.json                discovery file (required)
  POP_BOLL_SHAKE_V4_11112026.md      the agreement, with your two boxes ticked (required)
  MANIFEST.tsv                       every recording: path, size, sha256 (required)
  art/                               optional cover images (jpg or png), one per album folder or one for the corpus
  <recordings>                       flat, or in album subfolders; any depth up to 3

Rules:

• HTTPS only, with a certificate a normal browser trusts. Plain HTTP is refused.
• Every file is served by plain GET, and the server answers HEAD for at least the discovery file and the manifest. HEAD on the discovery file must return ETag or Last-Modified.
• The same credential covers everything under <base>.
• No redirects away from <base>'s host. A redirect inside the same host is followed once.
• Paths in the manifest are relative to <base>, use / as the separator, contain no .., no leading /, and no control characters. Spaces are allowed and are percent-encoded when fetched.
• Recording formats: .mp3, .wav, .flac, .aif, .aiff, .m4a. Anything else in the manifest is refused.
• Total size up to about 20 GB; the discovery file up to 64 KiB; the manifest up to 4 MiB.

2.2 The discovery file: popboll-corpus.json#

UTF-8 JSON, one object. Keys not listed here are ignored.

{
  "schema": "popboll.origin/1",
  "corpus": {
    "name": "The Band",
    "artist": "The Band",
    "legal_name": "Pat Doe",
    "contact_email": "pat@example.com",
    "description": "Three albums and the 2019 EP.",
    "website": "https://theband.example"
  },
  "licence": {
    "shake_version": 2,
    "shake_file": "POP_BOLL_SHAKE_V4_11112026.md",
    "scope": "full",
    "hosting": "self",
    "accepted_by": "Pat Doe",
    "accepted_email": "pat@example.com",
    "accepted_at": "2026-11-09T18:00:00Z",
    "statement": "Sounds good to me!"
  },
  "files": {
    "manifest": "MANIFEST.tsv",
    "art": ["art/cover.jpg"]
  },
  "auth": {
    "type": "basic"
  },
  "status": "live",
  "notes": "Anything you want Graham to read."
}

Pop Boll compares the Shake file against the published text of the current Shake. Only the two checkbox lines and the signature block may differ. A Shake with other edits is refused with a message that says where.

2.3 The manifest: MANIFEST.tsv#

One line per recording, tab-separated, LF line endings, UTF-8, sorted by path, no header:

path<TAB>size_bytes<TAB>sha256_hex

For example:

2019-ep/01 First Song.mp3   6213504 3f1c…e9
2019-ep/02 Second Song.mp3  5890221 a07d…11
album-one/01 Opener.mp3 7120033 c9b2…54

Cover images are not in the manifest; they are named in files.art. The manifest lists recordings only, so that its hash changes exactly when the music changes.

Generate it with the standard tools on the server, from inside the folder:

find . -type f \( -iname '*.mp3' -o -iname '*.wav' -o -iname '*.flac' -o -iname '*.aif' -o -iname '*.aiff' -o -iname '*.m4a' \) \
  | sed 's|^\./||' | LC_ALL=C sort \
  | while IFS= read -r f; do printf '%s\t%s\t%s\n' "$f" "$(stat -c %s "$f" 2>/dev/null || stat -f %z "$f")" "$(shasum -a 256 "$f" | cut -d' ' -f1)"; done > MANIFEST.tsv

Or with Python, which behaves the same on every platform:

python3 - <<'EOF'
import hashlib, os
exts = {".mp3", ".wav", ".flac", ".aif", ".aiff", ".m4a"}
rows = []
for root, dirs, files in os.walk("."):
    dirs[:] = [d for d in dirs if not d.startswith(".") and d != "art"]
    for name in files:
        if os.path.splitext(name)[1].lower() in exts:
            p = os.path.relpath(os.path.join(root, name), ".").replace(os.sep, "/")
            h = hashlib.sha256()
            with open(p, "rb") as fh:
                for chunk in iter(lambda: fh.read(1 << 20), b""):
                    h.update(chunk)
            rows.append((p, os.path.getsize(p), h.hexdigest()))
rows.sort()
with open("MANIFEST.tsv", "w", encoding="utf-8", newline="\n") as out:
    for p, size, digest in rows:
        out.write(f"{p}\t{size}\t{digest}\n")
print(len(rows), "recordings")
EOF

2.4 Authentication#

• basic: HTTP Basic over HTTPS. Make a dedicated read-only user for Pop Boll; do not reuse your own.
• bearer: an Authorization: Bearer <token> header. Same idea: one token, read-only, for Pop Boll.
• none: the folder is public. Allowed, but then anyone can download your recordings; on popboll.com Pop Boll will still only stream them to paid supporters.

The secret is entered once in the bollawax.org backstage (artist tier) and stored encrypted. It is never written to the discovery file, the manifest, an email or a corpus manifest. To rotate it, enter the new one in the backstage first, then change it on the server; Pop Boll retries with the new credential on the next check-in, and a 401 for seven days counts as unreachable, not as revoked.

2.5 Status and the handshake, from Pop Boll's side#

The states a self-hosted corpus moves through:

Removal of one track: when a path leaves the manifest, the track leaves the listing and any stored copy within a day. Its morpus pair is retired from study at the next DNA rebuild. Cells musicians already extracted stay, as the Shake says.

What Pop Boll stores, by scope, for a self-hosted corpus:

2.6 Where this lands in Pop Boll's own records#

The corpus manifest Pop Boll writes (corpus.json) gains an origin block:

"origin": {
  "kind": "self_hosted", "base": "https://files.theband.example/popboll/",
  "discovery_etag": "\"5f2a…\"", "manifest_sha256": "…", "last_ok": "2026-11-10T06:00:02Z",
  "failures": 0, "state": "live", "hidden_since": null
}

The licence block records "hosting": "self" and "shake_version": 2. The archived edition carries evidence/popboll-corpus.json, evidence/fetch_record.json and licenses/POP_BOLL_SHAKE_V4_11112026.md.

2.7 The validator#

Planned: POST https://bollawax.org/api/origin/check with {"base": "<base>"} and the credential already in the backstage, returning a report with one line per rule in §2.1 to §2.4. Until it exists, Part 3 is the validator.

Part 3. Checklist for the agent#

Run from a machine that is not the server, with the credential in $CRED (user:pass for basic) and the base URL in $BASE (ending with a slash).

1. The certificate is valid and plain HTTP is not used: bash curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' -u "$CRED" "${BASE}popboll-corpus.json" Expect 200 0.
2. HEAD works on the discovery file and returns an ETag or Last-Modified: bash curl -sS -I -u "$CRED" "${BASE}popboll-corpus.json" | grep -iE '^(etag|last-modified):'
3. The discovery file is valid JSON with the required keys, schema is popboll.origin/1, status is live, and auth.type matches how you actually protected the folder: bash curl -sS -u "$CRED" "${BASE}popboll-corpus.json" | python3 -c ' import json,sys; d=json.load(sys.stdin) assert d["schema"]=="popboll.origin/1", d["schema"] for k in ("name","artist","legal_name","contact_email"): assert d["corpus"][k] L=d["licence"]; assert L["shake_version"]==2 and L["scope"] in ("full","morpus_only") and L["hosting"]=="self" for k in ("accepted_by","accepted_email","accepted_at","statement"): assert L[k] assert d["files"]["manifest"] and d["auth"]["type"] in ("basic","bearer","none") and d["status"] in ("live","paused","revoked") print("discovery ok:", d["corpus"]["name"], L["scope"], d["status"])'
4. The Shake is present and is the current one (version 4) with exactly one box ticked in each "Choose one": bash curl -sS -u "$CRED" "${BASE}POP_BOLL_SHAKE_V4_11112026.md" | grep -c '^- \[x\]' Expect 2.
5. The manifest downloads, has three tab-separated columns, is sorted, lists only allowed formats, and every path exists on the server with the stated size: bash curl -sS -u "$CRED" "${BASE}MANIFEST.tsv" -o /tmp/MANIFEST.tsv python3 - <<'EOF' import sys, re rows=[l.rstrip("\n").split("\t") for l in open("/tmp/MANIFEST.tsv", encoding="utf-8")] assert all(len(r)==3 for r in rows), "three columns" paths=[r[0] for r in rows]; assert paths==sorted(paths), "sorted" assert all(re.search(r"\.(mp3|wav|flac|aif|aiff|m4a)$", p, re.I) for p in paths), "formats" assert not any(p.startswith("/") or ".." in p.split("/") for p in paths), "paths" print(len(rows), "rows,", sum(int(r[1]) for r in rows)//(1<<20), "MiB") EOF Then spot-check three files by HEAD and compare Content-Length to the manifest's size.
6. One full hash check on the smallest file: bash curl -sS -u "$CRED" "${BASE}$(python3 -c 'import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))' "<a path from the manifest>")" | shasum -a 256
7. A wrong credential is refused (so the folder really is private): bash curl -sS -o /dev/null -w '%{http_code}\n' -u "nobody:wrong" "${BASE}MANIFEST.tsv" Expect 401 or 403, unless auth.type is none.
8. Email the Shake (the current one is at bollawax.org/pdf) and <base> from the artist's own address to shake@bollawax.org with the words in the Shake. Enter the credential in the backstage, or wait for the one-time form. Do not put the credential in the email.

What is built on 10 November 2026 (Bollawax 0.2.4): the discovery file has a schema, origin, in the open Bollard SDK, and the SDK can fetch and check one (bollardeck.corpus.fetch_origin); the backstage records that a corpus is self-hosted ("I host them myself", hosting in the licence block). What is not built: the hub states in §2.5, the heartbeat, hidden, the origin block, the backstage credential store, and a validator you can run on your folder. So Pop Boll does not pull from anyone's server yet, and until it does a self-hosting artist uploads the zip as well. Everything in Part 2 that the artist's server must do is final enough to build against.