# Brioso Docs - Full Context > Generated by docmd --- ## [Brioso Docs](https://fitxers.usebrioso.com/) # Brioso Docs Internal documentation for the Brioso monorepo — runbooks, architecture, policies, and product docs. Built from repo-root `docs/` (this tree) and `.agents/docs/` (published under [`/internal/`](internal/)) on every push to `beta` by `deploy-docs.yml`. The site and its machine feeds sit behind Cloudflare Access — humans log in with their team email, agents use a service token (below). ## Quickstart ### Humans 1. Open https://fitxers.usebrioso.com and log in with your team email (one-time PIN; 12h session). 2. Browse the sidebar or press `⌘K` to search. Everything under **Internal** mirrors `.agents/docs/` — start with [Architecture](internal/architecture.md), the [PRD](internal/prd.md), or the [roadmap](internal/roadmap.md). Ops material lives in the runbooks (GPU gateway, home worker, deploy, secrets…). ### Agents Fetch the docs corpus headlessly with the Access service token, pulled from Infisical (`prod`, `/infra` — the same source of truth as the other infra secrets): ```bash infisical run --env=prod --path=/infra -- \ curl -sS -o /tmp/llms-full.txt \ -H "CF-Access-Client-Id: $CF_ACCESS_CLIENT_ID" \ -H "CF-Access-Client-Secret: $CF_ACCESS_CLIENT_SECRET" \ https://fitxers.usebrioso.com/llms-full.txt ``` Machine-readable feeds, generated at build time: - [`/llms.txt`](llms.txt) — page index - [`/llms-full.txt`](llms-full.txt) — full corpus in one request - [`/llms.json`](llms.json) — machine-readable manifest Token rotation and troubleshooting: [docs-site runbook](internal/docs-site-runbook.md). --- ## [2026-07-15 — Drizzle Gateway + otelite + Tailscale HTTPS: session learnings](https://fitxers.usebrioso.com/internal/2026-07-15-drizzle-gateway-otelite-tailscale-learnings/) (Internal) Superseded by deploy v2 (BRI-338) + windmill-logmcp-decommission-runbook.md. # 2026-07-15 — Drizzle Gateway + otelite + Tailscale HTTPS: session learnings Session date: 2026-07-15. Worked directly on the VPS (`ovhcloud.tail876c20.ts.net`). Source debrief lives at `/home/debian/Documents/prjcts/docs/drizzle-gateway-otelite-tailscale.md` on the VPS and is preserved verbatim upstream of this document. This file is the in-repo summary, extended with two related bugs the original debrief missed and with the deploy-automation gap quantified. ## Outcome Both observability backends are now reachable over the tailnet with HTTPS: - `https://ovhcloud.tail876c20.ts.net:8000` → Drizzle Gateway (`127.0.0.1:4983`) - `https://ovhcloud.tail876c20.ts.net:8001` → otelite web UI (`127.0.0.1:16686`) All fixes were applied on the VPS as `systemctl edit` drop-ins, not by hand-patching `/opt/brioso/current` or `/opt/brioso/repo`, per the project's deploy convention. They need to land in the actual unit files and deploy scripts in git so the next redeploy does not reintroduce them. ## Bugs in `deploy/systemd/brioso-drizzle-gateway.service` All five are present in the file as committed (verified 2026-07-15 against HEAD): ### Bug 1 — `User=brioso` / `Group=brioso` does not exist Only a `brioso` *group* (gid 1003) exists on the VPS; `web-services` is its only member. The unit runs as a non-existent user and systemd refuses to start it. Every other unit (`brioso-web`, `brioso-litestream`, `brioso-otelite`) runs as `User=web-services` / `Group=web-services` (or `Group=brioso` in the web case, both exist). Fix: use `User=web-services` / `Group=web-services`. ### Bug 2 — `ReadWritePaths=` uses comma separator Systemd's `ReadWritePaths=` is space-separated. The unit sets `ReadWritePaths=/opt/brioso/shared/data,/etc/drizzle-gateway`, and systemd parses that whole comma-joined string as one literal path. It does not exist, so the unit fails with `status=226/NAMESPACE` ("Failed to set up mount namespacing: ... No such file or directory"). Fix: change to `ReadWritePaths=/opt/brioso/shared/data /etc/drizzle-gateway` (space between the two paths). ### Bug 3 — `StartLimitIntervalSec=60` and `StartLimitBurst=5` under `[Service]` Both keys are `[Unit]`-section directives in current systemd. Placed under `[Service]` they are silently ignored (`Unknown key 'StartLimitIntervalSec' in section [Service], ignoring`), so the crash-loop rate limit configured alongside `StartLimitBurst=5` never actually applies. Fix: move both lines under `[Unit]`. ### Bug 4 — `Environment=DB_FILE=...` is dead config `strings`-d the drizzle-gateway binary: it only reads `MASTERPASS` and `STORE_PATH` from the environment. There is no `DB_FILE`, `DATABASE_URL`, or similar env knob. Drizzle Gateway is configured through its own web UI on first login (dialect: SQLite, type: file, path `/opt/brioso/shared/data/local.db` — matches the web app's `DATABASE_URL` in `/opt/brioso/shared/web.env`). Fix: remove the `Environment=DB_FILE=...` line. It is dead config that misleads the next reader into thinking the DB is pre-wired. ### Bug 5 — Missing `STORE_PATH` env var Drizzle Gateway persists its connection config to `./drizzle-gateway-volume/store.json` by default (relative to `WorkingDirectory`, which is `/opt/brioso`). Two compounding problems: 1. `/opt/brioso` is read-only under `ProtectSystem=strict` and not covered by `ReadWritePaths`, so every startup fails with `Failed to load data: EROFS: read-only file system, mkdir './drizzle-gateway-volume'`. 2. The `WorkingDirectory=/opt/brioso` is also conceptually wrong — that path is the deploy symlink farm and never the right place to write app state. Fix: add `Environment=STORE_PATH=/opt/brioso/shared/data/drizzle-gateway-volume` and ensure that directory exists, is owned by `web-services:web-services`, and is covered by `ReadWritePaths` (it already is once Bug 2 is fixed — both paths are under `/opt/brioso/shared/data`). ## Bug 6 — `deploy/systemd/brioso-otelite.service` runs as a non-existent user `User=otelite` / `Group=otelite` — `otelite` is not a user on the VPS (verified 2026-07-15 via `id otelite` → "no such user"). `install-otelite.sh:72-74` *does* create the user conditionally, but the deploy path on this VPS apparently never ran that branch. Same class of bug as Bug 1 (drizzle-gateway's `User=brioso`/`Group=brioso`). Fix: use `User=web-services` / `Group=web-services`. Same dependency on the `web-services` group already existing on the VPS — verified. ### Adjacent hazard — repo `brioso-otelite.service` and the live unit have drifted Confirmed live on the VPS via `/etc/systemd/system/brioso-otelite.service`: the unit the operator actually built and is running: - Wraps everything in `nono run` - Uses `User=web-services` / `Group=web-services` - Uses positional flags `--addr` / `--storage-path` (no `--web-port`) - Has no `StartLimit*` lines at all The repo canonical file (`deploy/systemd/brioso-otelite.service`) at the time of writing uses `User=otelite`, the named `--web-port 16686` flag, and the `StartLimit*` rate-limit lines under the wrong section. If `setup-observability.sh:51` re-copies the unit from `$REPO_DIR` on a future run, the working live config is replaced by a broken one. This means: **the next time `setup-observability.sh` is run, otelite will break** independently of Bug 6 getting fixed. The two paths have to be reconciled before the next redeploy. Recommended approach (plan commit 1 below): reconcile the live unit's `nono run` / positional-flag style into a unified canonical, commit, then redeploy — not treat the repo unit as the source of truth. ## Same `[Service]` vs `[Unit]` bug in the other two units (not flagged in VPS doc) `grep -n 'StartLimit' deploy/systemd/*.service` shows the same directive-in-wrong-section mistake in both: - `deploy/systemd/brioso-otelite.service:12-13` — `StartLimitIntervalSec=60` and `StartLimitBurst=5` under `[Service]`. - `deploy/systemd/brioso-web.service:17-18` — same two lines under `[Service]`. Both services happen to run, but the rate limit is silently inactive. Fold them into the same unit-file fix when Bug 3 is addressed, otherwise they drift out of sync again. ## otelite: misconfiguration, not a unit bug otelite was swept into the 2026-07-13 decommission of the old FastAPI stack (`systemctl disable --now brioso-api brioso-refine brioso-otelite`) and never re-enabled. Its dashboard had been rebound to `127.0.0.1:3100` during that cutover (to avoid colliding with `brioso-web` on `:3000`). Per the original design, `apps/web/src/server/otelite.ts` hardcodes `OTELITE_BASE_URL = "http://127.0.0.1:16686"`, so the dashboard **must** stay on `:16686`. Repointed `ExecStart`'s listen args back to `:16686` and re-enabled the unit. **Worth a repo-side audit:** is there any script or workflow step that re-runs `systemctl disable brioso-otelite` (or similar) during a decommission-style cleanup, that could silently re-disable it again on a future redeploy? Nothing in the current `deploy/` tree does (verified with `grep -r "systemctl disable" deploy/`), but worth keeping in mind when adding any new cleanup task. ## `tailscale serve` mode `:8000` and `:8001` had been set up in `tls-terminated-tcp` mode pointing at themselves (`127.0.0.1:8000`, `127.0.0.1:8001` — nothing ever listened there). That state was left behind from an earlier pass by another agent session and surfaced as the `NS_ERROR_NET_EMPTY_RESPONSE` the user originally reported (the cert was for the hostname but no upstream was behind the proxy). Reset and redone as HTTP reverse proxies: ``` tailscale serve --bg --https=8000 http://127.0.0.1:4983 # Drizzle Studio tailscale serve --bg --https=8001 http://127.0.0.1:16686 # otelite ``` This is live tailscaled state on the box, not config tracked in the repo, but the `setup-observability.sh` script (which already calls `tailscale serve reset` followed by the two serve rules) is the idempotent re-applier. Important: if a future deploy adds a `tailscale serve` rule but does not go through `setup-observability.sh`, it must not assume any particular existing config — always reset. ## Deploy automation gap (still open) Verified that the following are not wired in: - `deploy/scripts/install-drizzle-gateway.sh` is called from `setup-observability.sh:39`, but neither that script nor `deploy-vps.sh` is itself called from any GitHub Actions workflow (verified by inspection of `.github/workflows/`). Means a fresh VPS never gets Drizzle Gateway unless an operator SSHes in and runs `setup-observability.sh` by hand. The `install-drizzle-gateway.sh` script is already idempotent. - `VITE_OTELITE_URL` / `VITE_DRIZZLE_GATEWAY_URL` are never exported before `deploy-vps.sh:102` runs `bun run build`, so the built bundle's fallback URLs are `localhost:16686` and `localhost:4983` (confirmed by grep on `.output/public/assets/*.js`). A service restart will not fix this — it is baked in at build time. Needs the two env vars set to `https://ovhcloud.tail876c20.ts.net:8001` and `https://ovhcloud.tail876c20.ts.net:8000` respectively, then a rebuild + redeploy. - All five unit bugs above plus this automation gap should ideally be fixed together so a fresh `deploy-vps.sh` run produces a working `brioso-drizzle-gateway` unit without any manual `systemctl edit` drop-in. ## Tailscale ACL — not a repo issue, recorded for context ACL grant used on the tailnet: ```json { "src": ["autogroup:member", "carloshpiedra@gmail.com"], "dst": ["tag:server"], "ip": ["tcp:8000", "tcp:8001", "tcp:4318", "tcp:3000"] } ``` Note that this is the `grants` schema — no `action` key is needed there; only the legacy `acls` array requires a sibling `"action": "accept"`. Structurally fine as written. One teammate (`carloshpiedra@gmail.com`) initially could not reach `:8000`/`:8001` despite being in `src`. Diagnosis, confirmed via `tailscale status --json` on the VPS: - OS-level firewall was **not** the cause — `ufw`/`iptables` allow all inbound on the `tailscale0` interface unconditionally. - His devices showed up only as anonymized **shared** nodes (`Hostname: device-of-shared-to-user`, blank `DNSName`) — unlike fully-joined members who get real `*.tail876c20.ts.net` MagicDNS names. MagicDNS could not resolve `ovhcloud.tail876c20.ts.net` on his end even though the ACL `src` rule matching his email would apply once traffic actually reached the node. - He had been given access via both "share this device" and a tailnet member invite, but only the device-share had taken effect; the member invite was likely not accepted yet. - **Workaround used:** he added a `/etc/hosts` entry on his own machine mapping the VPS Tailscale IP (`100.105.198.36`) to `ovhcloud.tail876c20.ts.net`. Works per-device but is brittle to IP changes and won't help him reach other tailnet services. Getting him fully accepted as a member is the durable fix and is an operational/Tailscale-admin task, not a repo fix. ## What changed on the VPS (for `systemctl edit` diff context) Patches were applied as `systemctl edit brioso-drizzle-gateway` / `brioso-otelite` drop-ins, leaving `/etc/systemd/system/.service` unchanged from `git`'s view but layering overrides on top. Canonical fix lands in `deploy/systemd/brioso-drizzle-gateway.service` (and the other two units) so the next `setup-observability.sh` run produces the right unit without the drop-in. --- ## [2026-07-17 — VPS state management: migration vs reconciliation research](https://fitxers.usebrioso.com/internal/2026-07-17-vps-state-management-research/) (Internal) Superseded by state-mgmt-policy.md (most findings incorporated). # 2026-07-17 — VPS state management: migration vs reconciliation research Status: **researched, not implemented.** Filed for later. Owner: TBD. ## Problem statement (verbatim from user) > "it seems to me that with deployment we're putting a lot of effort on setup and then cleanup of unused/dead services. Can't we just make it in a db-migration-like basis (ie on deployment we do a series of checks which verify the server is in a 'healthy' state and we can automate some scripts for cleanup)?" User's pain, concretely observed during 2026-07-15 work: - Manual `systemctl edit` drop-ins accumulating on the VPS (drizzle-gateway had 5 bugs patched this way). - Dead config in unit files (e.g. `DB_FILE=` nobody reads). - Old units leaving behind state after decommission (FastAPI stack: `brioso-api`, `brioso-refine`). - 5+ orphan nono profiles at `/home/web-services/.config/nono/profiles/` from the FastAPI cutover, repo only has 2. - Repo unit files drifting from live canonical (otelite drift; would have broken the next `setup-observability.sh`). - `setup-observability.sh` only manages 2 of 4 units. `brioso-web` and `brioso-litestream` have no automated installer. - `deploy-vps.sh` only invokes the reconciler on a fallback branch (`if [[ ! -x drizzle-gateway ]]`). ## TL;DR (both research lanes agree) **This is not a migration problem. It is a reconciliation problem.** The repo unit files are already the desired state; the gap is enforcing them and reaping orphans. Build a shell-only declarative reconciler wired into every deploy, plus a read-only drift gate. ~150 lines of shell, zero new dependencies. Skip migrations, Ansible, NixOS, Packer — all overkill for 1 VPS / 4 units / 1 developer. The half-built reconciler (`setup-observability.sh`) already exists and is correct in spirit; the work is to generalize it, make it unconditional, add a manifest, add a cleanup loop, and pair it with a verify gate. ## Why migrations are the wrong model (oracle) Numbered migrations (`0001_initial.sh`, `0002_...`) earn their keep when transitions are **irreversible** (DB schema with live data). Systemd units are **already declarative** — the file at `deploy/systemd/brioso-otelite.service` *is* the desired end state. A migration system would re-derive that state through N stacked steps, and you'd still need a verify step on top. Additional migration failure modes at this scale: - Migrations don't compose (migration 12's effect silently undone by migration 28). - Decommissions need a paired `0028_remove_stale_api.sh` — easy to forget, easy to get wrong, no enforcement. - No diff/plan output — run it and hope. - No standard library; each `ensure_*` is hand-rolled. - Real-world adoption: rare. Mostly personal infra dotfiles repos. No widely-adopted framework for shell-based server-state migrations. ## Approaches considered and rejected | Approach | Setup cost | Verdict for this project | |---|---|---| | Migration-style shell scripts | Low | Wrong model (see above). Migrations add a second source of truth alongside the declarative unit files. | | Ansible playbooks | Medium | Python toolchain on controller, inventory, playbook, learning curve. Tractor for a balcony at 1 VPS / 4 units. | | NixOS / Nix deploy | Very High | NixOS as full OS, Nix language learning curve, atomic rollback is nice but overkill. | | Packer / immutable infra | Very High | VPS rebuild on every deploy, ~5-10 min per deploy, not zero-downtime. | | testinfra (Python) | Medium | Python dependency on test runner, steeper than goss for equivalent validation. | | InSpec (Chef) | Medium-High | Ruby + Chef Infra dependency. Overkill. | | goss alone | Very Low | Excellent as the *verify* step but cannot *apply* state. Needs a reconciler alongside. | ## The recommendation **One shell script that owns VPS state, called unconditionally on every deploy. Plus a read-only verify step that fails CI on drift.** ### `reconcile-vps.sh` (the apply step) Reads a manifest (array literal at top — no YAML). Three responsibilities: 1. **Install + enable every unit in the manifest** — copy from `deploy/systemd/` to `/etc/systemd/system/`, including tracked drop-ins under `deploy/systemd/$unit.service.d/`. Run the existing idempotent installers (`install-drizzle-gateway.sh`, `install-otelite.sh`). Set tailscale serve rules. 2. **Cleanup** — disable + remove any `brioso-*.service` in `/etc/systemd/system/` NOT in the manifest. This is the automatic dead-service removal: drop a unit from the repo, next deploy reaps it on the VPS. 3. **`systemctl revert`** to flush untracked drop-ins before copying — the canonical files in the repo are the source of truth, drop-ins are an escape hatch that should not survive a deploy. Folds in (or imports) `setup-observability.sh`. Ponytail: fold it, one fewer file to remember. ### `verify-state.sh` (the gate) Read-only. For each manifest unit, `diff` `/etc/systemd/system/$u.service` against `deploy/systemd/$u.service`. Same for profiles. Exit non-zero on any mismatch. Called from `.github/workflows/deploy.yml` before `deploy-vps.sh` over SSH. On failure, the deploy aborts with a clear "drift detected" message instead of silently clobbering. ### Wire-in In `deploy-vps.sh`, replace the conditional block at lines 86-89 (`if [[ ! -x /usr/local/bin/drizzle-gateway ]]; then ... setup-observability.sh ... fi`) with an unconditional `reconcile-vps.sh` call before the build. Drop the `if` — it's a smell that the reconciler wasn't trusted. ## Why this is the lazy answer The repo unit files have been the source of truth all along. The only missing pieces: 1. Actually copying them on every deploy (today: only 2 of 4 units, conditional). 2. Checking they match before deploy (today: nothing). 3. Enumerating which units are expected so orphans can be detected (today: implicit, scattered). That's ~150 lines of shell. No new deps. No framework. The cleanup loop is ~10 lines and is the entire answer to "how does dead state get removed automatically." ## Suggested commit breakdown (when we tackle this) | # | Commit | Est. lines | |---|--------|------------| | 1 | `feat(deploy): add reconcile-vps.sh declarative state reconciler` — manifest, copy+enable, cleanup loop, drop-in handling, drop-in revert. | ~100 | | 2 | `feat(deploy): wire reconcile into deploy-vps.sh + add verify-state.sh drift gate` — unconditional call, CI step in `deploy.yml`. | ~40 | | 3 | `chore(deploy): prune dead state on VPS` — manual run of reconcile, `systemctl revert` on the 2026-07-15 drop-ins, document what got removed in `.agents/notes/`. | — | ## Optional: goss as the verify step instead of hand-rolled diff The librarian recommended goss for the verify side. It's a 5MB Go binary, YAML specs, auto-generates from live system via `goss autoadd`, exits non-0 on drift. If hand-rolled `diff` in `verify-state.sh` feels too brittle, goss is the upgrade path. Tradeoff: - Hand-rolled `diff` — zero deps, ~30 lines, but limited to "files match." - goss — adds one binary, but can also check ports listening, users exist, file modes, command outputs. Better long-term audit surface. Ponytail call: start with hand-rolled `diff`, switch to goss if/when verify needs grow past file comparison. ## Drift classes the reconciler solves vs. doesn't solve | Class | Solved by reconciler? | Notes | |---|---|---| | Live-vs-repo unit drift | Yes | `verify-state.sh` detects; `reconcile-vps.sh` repairs. | | Drop-in invisibility | Yes | `systemctl revert` + canonical copy. | | Partial reconciler (only 2/4 units) | Yes | Manifest enumerates all expected units. | | Orphan units from decommissions | Yes | Cleanup loop reaps anything not in the manifest. | | Dead config in unit files | Partially | Reconciler enforces the repo version, but catching dead config requires human review of the unit file itself. Lint the unit files separately. | | Destructive `tailscale serve reset` | Worsened (intentionally) | Reconciler makes repo the source of truth — every tailscale serve rule must live in the script. Add a comment: `# ponytail: reset is destructive — every tailscale serve rule must live in this script.` | | Orphan nono profiles | No | Profiles are JSONC files, harmless if no unit references them. Accept as harmless or extend the cleanup loop to `/home/web-services/.config/nono/profiles/` (reap anything not in `deploy/nono/profiles/`). | | Secrets existence/mode | No | Out of scope. Separate `verify-secrets.sh` ticket if needed. | ## Risk flags from the architectural review (ora-1) Things that exist today and need attention when we pick this up: 1. **5+ orphan nono profiles** at `/home/web-services/.config/nono/profiles/` (brioso-api, brioso-cloudflared, brioso-litestream, brioso-refine, plus the otelite one we added). Repo has 2. Decide: reap or accept as harmless. 2. **`brioso-litestream.service.d/logging.conf`** is a tracked drop-in. The reconciler must handle `*.service.d/` directories, not just bare `.service` files — otherwise the logging config gets lost. Easy to miss in commit 1. 3. **No reconciler for secrets** (`web.env`, `litestream.env`, `drizzle-gateway/secrets.env`). A fresh-VPS bootstrap will fail opaquely. Worth a separate `verify-secrets.sh` backlog item that checks existence + mode without reading contents. 4. **`brioso-drizzle-gateway.service` `WorkingDirectory=/opt/brioso`** is conceptually wrong (it's the deploy symlink farm). Harmless today because `STORE_PATH` redirects writes, but worth fixing while we're in there. 5. **`tailscale serve reset`** is destructive and the reconciler makes it more so. Every tailscale serve rule must live in `reconcile-vps.sh`. Document with a `ponytail:` comment. 6. **Confirm `brioso-web.service` `StartLimit*` location.** The 2026-07-15 doc claimed lines 17-18 were under `[Service]` (silently ignored). Current HEAD shows them under `[Unit]` — appears fixed during commit 07d176e. 30-second confirm during commit 3. ## Reference - Source research: - lib-2 (librarian) — industry patterns: goss, testinfra, InSpec, Ansible, NixOS, Packer, migration-style shell. - ora-1 (oracle) — architectural review of current `deploy/` setup, opinionated recommendation. - Related session work: - `.agents/docs/2026-07-15-drizzle-gateway-otelite-tailscale-learnings.md` — the bug debrief that surfaced this pain. - `.agents/plans/2026-07-15-fix-observability-unit-and-deploy-gaps.md` — the fix plan that landed 6 commits but did not address the underlying reconciliation gap. - Tools referenced: - goss: https://github.com/goss-org/goss · https://goss.readthedocs.io - Ansible `systemd_service` module: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/systemd_service_module.html - testinfra: https://testinfra.readthedocs.io - Chef InSpec: https://docs.chef.io/inspec/ ## Status Research complete. Implementation deferred per user direction ("we'll tackle it later"). When picking up: 1. Re-read this doc. 2. Re-verify the risk flags (some may have been addressed by other work). 3. Write the plan in `.agents/plans/-vps-state-reconciler.md`. 4. File a GH issue in `briosoco/brioso` and add to project 1 (likely Phase 7: VPS systemd deploy + CI, or a new Phase 8 sub-item). 5. Implement commits 1-3 above. --- ## [Agentic engineering workflow assessment](https://fitxers.usebrioso.com/internal/agentic-workflow-assessment/) (Internal) # Agentic engineering workflow assessment First self-assessment of the agentic coding loop (2026-08-26). The repo had accumulated substantial agent infrastructure — 19.6K tokens of `AGENTS.md` surface, a 640K-token `.agents/` knowledge corpus, 12 `check-*.ts` gates, 64 skills, 9 MCP servers, four harnesses in active use (Claude Code, Codex, Maki, Crush) — without ever being designed as a system. Source plan: `.agents/plans/2026-08-26-agentic-workflow-assessment.md`. ## F1 — Gate flakiness (fixed) Root cause: `turbo.json` had no `concurrency` cap (default 10), and none of the 27 workspace test scripts capped vitest's worker pool (defaults to `ncpu`). On an 8-core/8GB box already 2.7GB into swap, that's enough concurrent processes to starve memory and fail suites under contention that pass in isolation. | Measurement | Before | After | |---|---|---| | Cold `turbo run typecheck lint test --force` | 2m35s, 353% CPU, **failed** (`@brioso/jobs#test` red under contention, green alone) | **55-58s, 79/79 green**, 3x repeated cold runs, zero flakes | | `mise run validate:ts` (warm) | n/a (validate.ts didn't have a quick tier) | 4.5s | | New `mise run validate:quick` (typecheck+lint+check-\*.ts, no test) | n/a | ~1s warm | | CI (`ci.yml`, 13 runs) | median 2m11s, no timeout | unchanged wall time; now has a 15m `timeout-minutes` safety net | Fix: `turbo.json` `"concurrency": "50%"`; `packages/vitest.shared.mts` (`maxWorkers: 2`) imported by all 27 package configs; `mise run validate:quick` fast tier; `timeout-minutes` on `ci.yml` and `test-slow.yml`. ## F2 — Retrieval was recency-only (fixed) Only retrieval instruction in the repo was "skim the most recent 2-3 notes" — recency over a corpus 100x the context window. Added `scripts/build-agent-index.ts` → `.agents/INDEX.md` (`mise run agents:index`): one grep-friendly line per file across `plans/`, `notes/`, `docs/` (date, path, one-sentence summary). `AGENTS.md`'s Session memory section now points here first, recency-skim second. Open question, not resolved here: whether real hybrid search (BM25 + embeddings + RRF) is worth building on top. The index is the cheap experiment — revisit if grep-the-index demonstrably misses things a hybrid search would have caught. ## F3 — Plan knowledge was scattered (fixed for this repo's store) 173 plan files existed, only 9 tracked (`.gitignore:83` ignored `.agents/plans/`, plus a personal `~/.gitignore-global` rule doing the same independent of the repo). Triaged all 164 untracked files for secrets/scratch content (zero hits — see the commit that un-ignored and added them) and committed them. `AGENTS.md` now says explicitly that `.agents/plans/` is the one store, not any harness-local default (`~/.claude/plans/`, `~/.codex/`, `~/.local/state/maki/plans/`). Not migrated: the 52+36+13 plans still sitting in those harness-local stores. Optional per the source plan — revisit if they keep getting referenced by absolute path from notes (as `.agents/notes/2026-08-25.md` already did once). ## F4 — Durable investment was Claude-locked (partially fixed) - `.mcp.json` added at repo root, declaring the two MCP servers that are genuinely project-owned portable infrastructure (`brioso-sqlite` at `db.usebrioso.com`, `brioso-logs` at `logs.usebrioso.com` — both Tailscale-gated HTTP with per-teammate bearer tokens, no secrets committed). The other 7 servers agents used daily (Stripe, browserOS, posthog, calendar, mindwtr, mobbin, email_ai) run through a personal Bifrost aggregator on `localhost:8317` — not portable, left out by design. - `.claude/settings.json` added: wires `block-dangerous-git.sh` (previously git-tracked but registered nowhere — a dead guardrail reading as a live one) as a `PreToolUse` hook, plus a small read-only Bash allowlist. - 12 of 64 global skills promoted into `.agents/skills/` (symlinked into `.claude/skills/`, following the existing `dogfood`/`qa-studio` pattern): see the promotion commit for the full list and the audit's "leave alone" list (~52 skills judged Claude-Code-generic, personal, or for unrelated projects). - `.agents/skills/skills-lock.json` added, recording each skill's source — the source plan referenced this as an existing precedent; it didn't actually exist yet, so this pass created it. ## F5 — Rot in the always-loaded surface (fixed) - `AGENTS.md`'s duplicated `## Testing conventions` block (lines 233 and 263, byte-identical, verified before deleting) is gone — ~700 tokens saved on every agent turn, every harness. - `.claude/hooks/block-dangerous-git.sh` is now wired (see F4). ## F6 — Retrieval tooling, now documented `AGENTS.md`'s Session memory section now names `ugrep` (and its no-PCRE2-JIT caveat — `-P` patterns are slow), `ast-grep`, `rtk`, `opensrc`, `ghx` as available beyond plain `grep`/`find`. ## Recurring review (weekly agenda) - Re-run the gate benchmarks (warm, cold, peak RSS via `/usr/bin/time -l`); flag any regression against this doc's baseline (55-58s cold, 79/79 green, zero flakes over 3 runs). - `rtk gain` / `rtk discover` deltas. - New notes in `.agents/notes/` worth promoting into `AGENTS.md` or a `.agents/docs/` runbook (the "dreaming"/compaction step), then letting the note age out. - Harness friction encountered that week, per harness (Claude Code, Codex, Maki, Crush). - Instruction-surface token count (`AGENTS.md` + global `CLAUDE.md` + `RTK.md`, currently ~7,030 tokens before this pass) — does it still earn its place? - Whether `.agents/INDEX.md` retrieval is actually closing the gap it was built for, or whether hybrid search is now justified by evidence instead of intuition. ## Ticket No `BRI-*` issue was filed before this work started (per `AGENTS.md`, that's not a blocker). File "Agentic workflow hardening" as an umbrella issue and link this doc, retroactively, in Linear. --- ## [Architecture](https://fitxers.usebrioso.com/internal/architecture/) (Internal) # Architecture Brioso is a distributed system for generating audio: a web app (TanStack Start) handles intake, brand/prompt analysis, approval, job dispatch, and Stripe payment for three product lines — an agency Audio Brand Kit, creator custom generation, and a royalty-free catalog, all at fixed prices (see `prd.md`'s "Where the numbers live"), plus a Pro subscription for volume buyers. A kit or track is a parent record that fans out into one child `job` per asset once approved — each child job flows through the same generation pipeline regardless of which product line created it. Generation runs on worker hardware: a push daemon on a home GPU (pm2-supervised), or an on-demand Modal FunctionCall. Dispatch is push-only (BRI-439): the sweep POSTs each job to the runtime's push endpoint and walks the gateway document's `fallbackChain` on retryable failures. The worker, cloud handler, and wire protocol below are asset-agnostic — they run one job at a time and don't know which product line created it. ## System topology ``` CUSTOMER PATH ───────────── Customer browser │ ▼ ┌──────────────────────────────────────────┐ │ apps/web │ │ TanStack Start: product UI, API, billing │ └───────┬────────────────┬─────────────────┘ │ └──────────────────────► Cloudflare R2 │ uploads and delivery ▼ ┌──────────────────────────────────────────┐ │ Brioso application database │ │ jobs, inference_routes, provider key rings│ └───────┬──────────────────────────────────┘ ▼ ┌──────────────────────────────────────────┐ │ apps/web in-process sweeps │ │ dispatch sweep: every 60 seconds │ │ reaper: every 60 seconds │ │ stripe-doctor: daily │ └───────┬──────────────────────────────────┘ ▼ ┌──────────────────────────────────────────┐ │ packages/dispatch │ │ push dispatch, chain fallback, reaper, │ │ spend guardrails and observability │ └───────┬──────────────────────────────────┘ ▼ ┌──────────────────────────────────────────┐ │ Gateway fallback chain │ │ example: home if healthy else modal │ │ disabled or unconfigured runtimes: skip │ └───┬──────────────────────────┬───────────┘ ▼ ▼ ┌──────────┐ ┌────────┐ │ Home GPU │ │ Modal │ │ daemon │ │ app │ └────┬─────┘ └───┬────┘ └──────────┬─────────────┘ ▼ ┌─────────────────────────┐ │ SA3 / audiocpp handler │ │ generation → presigned │ │ R2 upload → writeback │ └─────────────────────────┘ ADMIN AND OBSERVABILITY PATH ──────────────────────────── Operator ─► tailnet-only Caddy ─┬─► /dash: PocketBase → Admin Console SPA │ │ │ └─► internal web API ├─► db.usebrioso.com: Drizzle / DB tooling └─► ops.usebrioso.com: Arize Phoenix apps/web, dispatch, and workers ─► PostHog + Arize Phoenix lifecycle events, traces, and eval signals ``` apps/web's in-process sweeps are the production periodic triggers (dispatch 60s, reaper 60s, stripe-doctor daily) — Hatchet was removed in BRI-438. PocketBase serves and proxies the tailnet-only admin console; it does not own Brioso business logic or the application database. ## Job lifecycle ``` Agency or creator → apps/web → application database: create parent record + jobs │ ▼ in-process dispatch sweep (60s) │ ▼ reaper → push chain walk → runtime submit │ ▼ POST job → home daemon / Modal function → R2 writeback │ ▼ Stable Audio 3 generation → R2 upload → completion │ ▼ application database: job terminal → parent ready → gated delivery ``` Modal reports results via the FunctionCall the reaper polls (stale-call arm); the home daemon PATCHes `paid` directly. A failed runtime promotes the job through the configured fallback chain without changing the product-level lifecycle. ## Creator subsystems The creator flow (`/studio/creators`) is a secondary, standalone flow — same `jobs` table and dispatch pipeline as kit generation, with a few loop-specific pieces: - **`g` generation namespacing.** Every `jobs.tracks` entry carries a `gen` field (`0` for the first generation, incrementing once per recreate). R2 keys are namespaced `/g/` (`server/jobs/handlers.ts` `issueUploadUrls`) and `idx` stays **globally monotonic** across a job's whole history (`JobsQueue.completeGeneration` offsets incoming indices and appends rather than replacing) — so a purchased `g0` track's assets are never touched by a later recreate's `g1` generation, and `createCheckout` / `downloadTrack` / the Stripe webhook all key off a single flat `idx` with no gen-aware branching. - **Recreate** (`JobsQueue.startRecreate`, `POST /jobs/:slug/recreate`). One atomic `UPDATE ... WHERE` carries every guard (not in flight, 30s cooldown, recreate-count cap — 3 while unpaid, 10 once any track is purchased) so two concurrent recreate calls can't both pass. Clears `sa3_prompt` so the dispatch sweep re-refines the revised brief. - **Email** (`packages/notify`). Provider-neutral (`EmailProvider` interface, Brevo the only implementation today) with `sendOnce(db, slug, kind, gen, build)` as the idempotency layer — an `INSERT ... ON CONFLICT DO NOTHING RETURNING` against `sent_emails` wins the race before `build()` runs, so the webhook and the reaper (both call `onGenerationComplete`) can't double-send the proposal-ready email, and a Stripe webhook retry can't double-send a receipt. - **License PDF** (`packages/licensing`). Rendered on demand (`pdf-lib`, no external dependency) from `docs/track-license.md`'s content assembled as a TS template, uploaded to the same deterministic `/g/license-.pdf` key `downloadTrack` re-derives for a repeat download. - **Feedback** (`job_feedback` table, `POST /jobs/:slug/feedback`). A DB table rather than veena's R2 JSON records, so it's queryable from admin tooling directly. - **Abuse resistance** (`packages/jobs/src/freeGenerations.ts`). A `free_generations` daily counter keyed on normalized email *and* IP independently (both must clear their cap), plus reuse-in-flight (a repeat submission from the same email while a job is still new/composing returns the existing job) — see `prd.md`'s $99 margin-risk note for the cost pressure this guards against. - **Audio protection.** Preview streams are watermarked (Pocket TTS, band-specific ducking, jittered per-track offsets — masters are never watermarked) and served through a proxy (not a redirect) gated on a Fetch-Metadata check plus a short-TTL HMAC playback token (`packages/auth-core/src/playback-token.ts`), so a bare presigned URL never leaks. `getJob` requires either a session that owns the job or a `share_token` (`packages/sharing`) — a stored, non-expiring, non-revocable per-job/per-track credential minted at job creation and threaded through both customer emails and the status page URL, since the product is deliberately link-shareable with no login required. `/s/:token` is a read-only listen page for forwarding a single proposal without exposing the full job. ## Catalog subsystem The royalty-free catalog (`/music`, BRI-125) is pre-generated and browsed rather than commissioned per brief — a materially different shape from the job/kit pipeline above: Catalog tables, bucket split, and storage keys live in [`data-model.md`](data-model.md). - **Taxonomy-driven pages.** `packages/catalog/src/taxonomy.ts` defines four browse axes (genre, mood, instrument, use-case) as path-based routes (`routes/_marketing/music/{genre,mood,instrument}/...`); `apps/web/src/lib/ catalog-seo.ts`'s `shouldIndexTaxonomyPage` keeps a leaf `noindex` until it has both real intro copy and a published track. - **Checkout and license.** `packages/verticals/catalog/src/{track,bundle}-checkout.ts` charge the fixed per-track/bundle price; `packages/licensing` renders `docs/catalog-license.md`'s **non-exclusive** terms for this product type, distinct from the creator flow's exclusive-sale license. ## Billing and entitlement `packages/billing` (`subscription-checkout`, `webhook`, `pro-credits`, `entitlement`, `pro-price-display`, `promo`) backs the Pro subscription. Entitlement state lives in the `subscriptions` DB table, never behind a feature flag — see `.agents/docs/feature-flags.md`'s non-goals. Subscription / entitlement tables, columns, and webhook flow live in [`data-model.md`](data-model.md). ## Account deletion Users can permanently delete their account from Settings -> Account -> Danger Zone. The server function (`apps/web/src/server/account/delete-account-fn.ts`) executes in a single transaction: 1. Cancel the Stripe subscription via API (if active). 2. Hard-delete: `pro_credit_usage`, `grants`, `catalog_favorites`, `catalog_download_usage`, `subscriptions`, `users`. 3. Anonymize: `catalog_purchases.user_id` set to NULL (preserves sales records). Orphan-row policy for email-keyed tables lives in [`data-model.md`](data-model.md). Session invalidation is automatic: the stateless HMAC cookie references a `userId` that no longer exists, so the next request resolves to logged-out. Grace period (soft delete + reaper) and full GDPR erasure (PostHog, analytics, email anonymization) are deferred to a future iteration. ## Repo layout | Path | Role | | -------------------------------- | ---------------------------------------------------------------------------------------- | | `apps/web/` | TanStack Start app — brand/prompt intake, strategy, kit/track orchestration, Stripe, admin | | `apps/worker/` | Home GPU push daemon — generation, presigned-R2 upload, writeback | | `packages/verticals/kits/` | Agency Audio Brand Kit vertical: assembly, brand-schema, checkout, strategy AI workflow, state machine | | `packages/verticals/studio/` | Creator studio vertical: UI components, composer/canvas domain; routes served by apps/web at /studio/* | | `packages/verticals/creator/` | Custom generation vertical: recreate, remix, feedback, free-tier quota, refine AI workflow | | `packages/verticals/catalog/` | Royalty-free catalog vertical: track/bundle checkout, delivery | | `packages/catalog/` | Catalog taxonomy, archetypes, slugs — browse-page data, distinct from `verticals/catalog`'s checkout | | `packages/classification/` | CLAP embeddings + sklearn classifier for catalog QC/enrichment (BRI-142); isolated uv venv, not a `[tool.uv.workspace]` member | | `packages/inference-core/` | Shared protocol types + `SubprocessEngine` (JSONL stdout → Python) | | `packages/musicgen-adapters/` | Domain models, ports (Protocols), storage adapters | | `packages/ai-workflow/` | Stateless LLM workflow (refine, brand analysis) — no DB, no obs, no env | | `packages/jobs/` | Job queue + contract schemas (TS side of the Python contract) | | `packages/entitlements/` | Grant codes + entitlement resolver (allowance, plan, redeem) | | `packages/billing/` | Stripe checkout + webhook handling, promo codes | | `packages/notify/` | Email: provider-neutral message shape, Brevo, templates, at-most-once send | | `packages/licensing/` | License PDF rendering (pure pdf-lib, no I/O) | | `packages/sharing/` | Share-link domain model: crypto-random tokens scoped to a kit or track, no expiry/revocation | | `packages/dispatch/` | Job dispatch sweep, spend ledger, reaper | | `packages/contract/` | TS<->Python wire-protocol source of truth (OpenAPI document) | | `packages/auth-core/` | Auth primitives: OTP, session route guards, rate limiting | | `packages/config/` | Env/config loading + audit, injected not read ambient | | `packages/storage/` | R2/S3 surface + presigned upload URLs, over injected credentials | | `packages/brioso-brand/` | Brand tokens, wordmark, shared React primitives | | `packages/obs-js/` | Shared observability facade (TS) — PostHog errors/events/AI-spans + LogTape, OTel export to Arize Phoenix | | `packages/obs-py/` | Shared observability facade (Python) — same facade, absorbs the old `musicgen-trace` OTel setup | | `packages/stable-audio-3/` | SA3 backend — isolated uv venv, workspace sibling (not a workspace member) | | `packages/db/` | Database schema + migrations | | `deploy/` | Deploy v2: `activate` binary (bun --compile), `pyinfra/v2/` tree, plus systemd units (litestream, Drizzle Gateway), Phoenix podman quadlet, sudoers. The macOS launchd plist is NOT committed — `apps/worker/scripts/install-worker.sh` generates one per-user at `$HOME/Library/LaunchAgents/`. | | `docs/` | End-user / product docs | Synced against `ls packages/` and `ls packages/verticals/` on 2026-08-08. `apps/admin`, `packages/signature`, and `packages/musicgen-trace` may still appear on disk in some checkouts as leftover `.turbo`/`node_modules` build artifacts — they are not git-tracked and not part of the real package set; `apps/admin` was removed per BRI-221 (see Admin/ops surface below). See `packages/stable-audio-3/scripts/README.md` for the standalone generation script and `.agents/docs/gpu-gateway-canary.md` for the push-dispatch operations doc. ## Data model Tables, storage, secrets, and env live in [`data-model.md`](data-model.md) — the canonical reference. This doc only mentions table names inline; consult `data-model.md` for columns, indexes, storage keys, R2 bucket prefixes, Litestream WAL, and secret/env strategy. If `packages/db/src/schema.ts` changes, `data-model.md` is updated in the same PR. ## Wire protocol Workers emit a **discriminated JSONL stream** on stdout. Each line is a single JSON object with a `type` field. The orchestrator (`SubprocessEngine`) parses lines until it sees a terminal `result` event, then returns. ``` {"type": "progress", "stage": "loading_model", "progress": 0.0, "message": ""} {"type": "progress", "stage": "generating", "progress": 0.5, "message": "step 4 of 8"} {"type": "result", "status": "ok", "output_path": "/tmp/out.wav", "sample_rate": 44100, "duration_seconds": 30.0} ``` **Event types:** | `type` | Key fields | Description | | ---------- | ------------------------------------ | ----------------------------------------------------------------------- | | `progress` | `stage`, `progress` (0–1), `message` | Stage progress update | | `event` | `name: "ready" | "unloading"`,` device?` | | `result` | `status: "ok" | "error"`,` output_path?`,` sample_rate?`,` duration_seconds?`,` error?` | Legacy compat: a bare `GenerateResponse` JSON line (no `type` field) is also accepted — if a line has `status` but no `type`, it is treated as a result event. Full types: see `musicgen_adapters.ports.events.WorkerEvent`. ## Deployment modes | Mode | How it works | Guide | | ----------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------ | | **VPS web app** | Deploy v2 (artifact + activate) on a VPS via Tailscale; beta live on v2, prod still v1 until WS5 | [`deploy/README.md`](../../deploy/README.md) | | **Home worker (macOS/Linux)** | pm2-supervised push daemon (`brioso-worker serve`) | [`.agents/docs/worker-deployment.md`](worker-deployment.md) + [`.agents/docs/home-worker-wake.md`](home-worker-wake.md) | | **Modal serverless** | On-demand GPU FunctionCall, push-dispatched | [`.agents/docs/gpu-gateway-beta-runbook.md`](gpu-gateway-beta-runbook.md) | ## Observability | Signal | Source | Backend | Status | | --------------------- | -------------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------ | | **Errors** | `obs-js`/`obs-py` `captureError` (web, worker, runtimes) | **PostHog** (EU) `captureException` | Live. Replaces GlitchTip — GlitchTip DSN/config removed. | | **Lifecycle events** | `obs-js`/`obs-py` `event`, `registerHttpObs`, `JobsQueue` transitions | **PostHog** (EU) `capture()` events | Live. Every observed request (`/jobs`, `/dispatch`, `/kits`, `/uploads`, `/otel` — see `apps/web/plugins/obs.ts`) and job state transition emits one, tagged `runner`. | | **Refine/LLM spans** | `runPrompt` → `WorkflowCtx.onGeneration` → `aiGeneration` | **PostHog** `$ai_generation` | Live. Every `@brioso/ai-workflow` call reports model, latency and token usage; failures too. The package stays vendor-neutral, apps/web maps onto `$ai_*` (`server/ai/workflow-ctx.ts`). | | **Deep OTel spans** | `registerHttpObs` root span + `withSpan`/`with_span` + ai-sdk → OTLP/HTTP | **Arize Phoenix, dev + prod** (`PHOENIX_COLLECTOR_ENDPOINT`) | Live. One trace spans web → runtime: the request's `traceparent` is stamped on the job row (the dispatcher's `leased_by` push is out-of-band, so the row carries the context), and worker writebacks inject it back. Two span processors — OpenInference for ai-sdk spans, a generic one for everything else, since the OpenInference filter drops manual spans. Silent no-op when Phoenix is unreachable or `TRACING_ENABLED=false`. Agent-queryable via Phoenix's native `/mcp` HTTP endpoint (Phoenix 19+) — see `.agents/docs/phoenix-runbook.md`. | | **Health** | `GET /health` checks DB + R2 + stale queue | Edge-triggered PostHog error on `ok→failing` transition | Live. No external prober yet. | | **Product analytics** | `posthog-js` autocapture + session replay | PostHog (EU) | Live. Product-side, unaffected by this migration. | ## Database access | Who | How | Scope | | ------------------ | ---------------------------------------------------------------------------------------------- | -------------------- | | **apps/web** | `@libsql/client` → `DATABASE_URL` (`file:/opt/brioso/shared/data/local.db`) | Read-write | | **apps/worker** | HTTP to `apps/web` job endpoints (`ApiJobStore`); never touches DB directly | Via web | | **Teammate (SQL)** | Drizzle Studio web UI → `db.usebrioso.com` (Caddy) → Drizzle Gateway (:4983) → reads `local.db` | Read-only (passcode) | | **Backup** | Litestream streams WAL from `local.db` → R2 (`brioso-jobs/litestream/`) every 1s | Continuous | Drizzle Gateway is a **viewer, not a DB server** — the web app keeps direct file access, Drizzle Gateway reads the same SQLite file as a second WAL reader. No cutover risk, no Litestream conflict. A future migration to Turso managed is still a single `DATABASE_URL` change for the web app. ## Admin / ops surface `apps/admin` (TanStack Start) is removed (BRI-221); its interim replacement (self-hosted Windmill) was also removed 2026-08-26. There is no dedicated admin UI — see `.agents/notes/2026-08-26-windmill-admin-followup.md`. Do not invest in new admin-specific infra without checking current intent first. ## Where this is headed See [roadmap.md](roadmap.md) for the roadmap — Iteration/Quarter milestones, open issues, and deferred work. --- ## [Audio tooling evaluation](https://fitxers.usebrioso.com/internal/audio-tooling-evaluation/) (Internal) # Audio tooling evaluation Verdict layer over `.agents/notes/own/tools-for-end-users.md` (source note, left as-is). That note is undated, has no verdicts, and one factually wrong claim about the model; this doc is the citable version the PRD and roadmap link to. ## Verdict table | Item | Verdict | Reason | |---|---|---| | Demucs v4 stems | Ship, flag-gated | Vertical layering is the standard FMOD/Wwise pattern; game devs are the primary case, not a secondary one | | Librosa/Essentia BPM + key | Ship, flag-gated | Cheap, benefits all three audiences, no model risk | | EBU R128 normalization | Ship, flag-gated | Already an ffmpeg filter; strictly additive to the existing encode ladder | | Seamless looping (zero-crossing) | Ship, flag-gated | Real gap — SA3 does not write loops. Loop points still cannot ship as `smpl` metadata under FLAC-only; a crossfaded seamless file is the deliverable | | Browser stem mixer | Defer behind stems | Layer-combination previewer, not a DAW. Meaningless before stems exist | | Basic Pitch audio-to-MIDI | Revisit on demand | Real for chiptune/tracker and runtime-sampler work; no current customer asking | | Wavesurfer.js inpainting UI | Ship, flag-gated | SA3 supports mask-based inpainting natively (paper §Editing) — the model side is free | | Brand-specific LoRA training | **Kill (as described)** | See correction below | | JUCE VST3/AU plugin | **Kill** | Game studios work in Unity/Unreal/FMOD/Wwise; agencies work in Premiere/After Effects. A signed cross-platform C++ plugin targets the one host neither vertical uses | | Unity/Unreal editor utilities | **Kill for now** | Downstream of a game-dev vertical that is not scoped. Same trigger as the WAV reopen clause in `prd.md` | | FMOD/Wwise session export | **Kill for now** | Same trigger | | ComfyUI prototyping | Not a product | Internal prototyping convenience; never a customer surface | --- ## [Native audio.cpp home workers](https://fitxers.usebrioso.com/internal/audiocpp-home-worker-runbook/) (Internal) # Native audio.cpp home workers These profiles run the home push daemon (`brioso-worker serve`) with `BRIOSO_WORKER_MODEL=audiocpp`. The M5 uses one resident audio.cpp server for the beta and production worker processes. The NVIDIA profile remains a single worker with the native CLI. Use the exact profile templates as the starting point: - NVIDIA RTX 4070 / CUDA: `apps/worker/.env.audiocpp-nvidia.example` - Apple Silicon / Metal: `apps/worker/.env.audiocpp-apple-silicon.example` Copy one to the machine's untracked `apps/worker/.env`, then set the shared secret, worker id, telemetry endpoint, and absolute binary/model paths. Do not set `AUDIOCPP_BACKEND` alone: `BRIOSO_WORKER_MODEL=audiocpp` is the model selector; `AUDIOCPP_BACKEND=cuda|metal` only selects its accelerator. ## Install and model preparation Pin audio.cpp to **v0.7.3**. On Linux/CUDA, build that tagged source with CUDA enabled and set `AUDIOCPP_BIN` to the `audiocpp_cli` path. On Apple Silicon, use the v0.7.3 Metal `audiocpp_server` binary. For the NVIDIA profile, download the F16 Medium GGUF to a normal local file and record its path in `AUDIOCPP_MODEL_PATH`: ```bash uv tool run --from huggingface_hub hf download \ audio-cpp/audio.cpp-gguf \ Stable-Audio-3-Medium-GGUF/stable-audio-3-medium-f16.gguf \ --local-dir /srv/brioso/models ``` The M5 server uses the q8_0 Medium variant (`stable-audio-3-medium-q8_0.gguf`, ~3.4 GB) at the path in its PM2 JSON config. Download the q8_0 sibling with the same command and copy it there. NVIDIA can also use Q8 by updating `AUDIOCPP_MODEL_PATH`. audio.cpp v0.7.3 uses a file extension to determine tensor format, so an HF cache symlink resolves to a blob without `.gguf` and fails. The copied file above is therefore a deployment requirement, not just a cache optimization. ## NVIDIA RTX 4070 (CUDA) 1. Copy the NVIDIA template to `apps/worker/.env`; keep `BRIOSO_WORKER_MODEL=audiocpp` and `AUDIOCPP_BACKEND=cuda`. 2. Build audio.cpp at v0.7.3 with CUDA, set an absolute `AUDIOCPP_BIN`, and pre-download the F16 Medium GGUF to the absolute `AUDIOCPP_MODEL_PATH`. 3. Run `brioso doctor`; it must pass the binary, model, CUDA, credentials, and backend checks. 4. Generate one F16 Medium WAV directly, then run `brioso start`, `pm2 save`, and `pm2 startup`. 5. Complete an existing API-backed `runner="home"` job and verify it reaches `paid` after upload. ## Apple Silicon (Metal) 1. Install the Metal v0.7.3 `audiocpp_server` and copy the Medium Q8 GGUF to the path in `deploy/pm2/m5-audiocpp-server.json`. 2. Create separate private env files at `~/.config/brioso/m5/beta.env` and `production.env`. Give each its own backend URL, worker secret, and `HOME_WAKE_TOKEN`. Set `BRIOSO_WAKE_REQUIRE_TOKEN=1` in both files. The addon examples use `push.tokenEnv: WORKER_SHARED_SECRET`, which is already available to the backend. Set each daemon's HOME_WAKE_TOKEN to its existing environment specific WORKER_SHARED_SECRET; never commit either value. AUDIOCPP_SERVER_URL explicitly selects the resident transport. Without it, the worker uses the existing CLI. Use `apps/worker/.env.audiocpp-m5.example` for the shared model URL and admission lock settings. 3. Start `deploy/pm2/ecosystem.m5-home.config.cjs`. Its model server listens on `127.0.0.1:7309`; the workers listen on `127.0.0.1:7312` and `:7311`. The supervised SSH reverse tunnel exposes them to the VPS on its own loopback ports `17312` and `17311`. Retain this SSH transport. The workers share one file lock and reject extra work before acknowledging it. 4. From the VPS, check `http://127.0.0.1:17312/health` for beta and `http://127.0.0.1:17311/health` for production. Confirm the runtime IDs. Missing, wrong, and opposite environment bearers must receive `401`. Correctly authenticated empty JSON POSTs to `/jobs` must return `422` without starting inference. Use `push.tokenEnv: WORKER_SHARED_SECRET` for the addon examples. 5. Confirm `/v1/models` says `stable-audio-3-medium` is loaded and both workers are idle before restarting the beta worker for a revision test. Leave the production worker, model, tunnel, and Tailscale policies unchanged. 6. The default gateway files preserve existing routing. Separate `config/gpu-gateway.m5-beta.example.yaml` and `config/gpu-gateway.m5-prod.example.yaml` files describe optional activation. Review and apply individual runtime and chain changes only when activating the addon. Do not import over the live registry. 7. Generate one beta song through `brioso_generate_song`, poll `brioso_get_generation`, and check the download. Production's `/mcp` remains disabled; worker health alone does not prove production inference. ## Checks and recovery For migration from SSH to the dedicated Mac Cloudflare Tunnel, follow the [tunnel rollout and rollback](m5-cloudflare-tunnel.md). Retain SSH until production completes its own HTTPS acceptance checks. `brioso engine` prints `audio.cpp/cuda` or `audio.cpp/metal`; `brioso config` shows the selected `BRIOSO_WORKER_MODEL` and backend. `brioso doctor` requires absolute, existing binary and model paths for audio.cpp profiles. Deploy through the existing guarded workflow when separately authorized. Capture the active release and runtime configuration before activation. On failure, restore the captured beta artifact and worker configuration. Keep strict bearer authentication, the SSH tunnel, and fallback chains. A completed song on Modal is a failed M5 verification. Correlate the job slug, call ID, `home-m5-beta` runner, worker completion, terminal `paid` record, and decoded application download. Revoke any temporary MCP credential. For a bad rollout, run `brioso stop`, correct the profile or restore `BRIOSO_WORKER_MODEL=sa3`, run `brioso doctor`, then `brioso start`. Inspect `brioso logs --lines 100` before rejoining the pool. Do not delete a claimed job or alter its runner while diagnosing; normal lease expiry/fallback remains the recovery mechanism. --- ## [Modal audio.cpp runbook (SA3.0-small, BRI-382)](https://fitxers.usebrioso.com/internal/audiocpp-modal-runbook/) (Internal) # Modal audio.cpp runbook (SA3.0-small, BRI-382) > Superseded by BRI-436/439: `apps/modal-audiocpp` is gone — the audiocpp > image + handler now run inside `apps/modal-worker` (`brioso-worker` Modal > app), push-dispatched via `submit`. Ops live in > `gpu-gateway-beta-runbook.md`; the image-assembly sections below (bake > layer, GGUF pin) still apply to `apps/modal-worker/modal_app.py`. ## Purpose Operational guide for the `brioso-audiocpp` Modal app: how the image is assembled, why the bake layer exists, how to deploy and smoke test, and the failure modes that cost us time to find. Companion to `apps/modal-audiocpp/README.md` (overview) and `.agents/docs/modal-beam-runbook.md` (SA3 medium on Modal, the contrasting pattern). - Modal app: `apps/modal-audiocpp/modal_app.py` - Handler: `apps/runpod/handler_audiocpp.py` - Smoke: `apps/modal-audiocpp/scripts/smoke.py` - CI: `.github/workflows/deploy-modal-audiocpp.yml` ## Current deployment | Artifact | Tag / state | |---|---| | audiocpp image | `usebrioso/brioso-runpod-audiocpp:v9` (built by `.github/workflows/build-runpod.yml`; there are no upstream Linux release prebuilts for audio.cpp, so images come only from our Docker Hub builds) | | Baked GGUF | `stable-audio-3-medium-q8_0.gguf` (`3,582,595,584` bytes, from `audio-cpp/audio.cpp-gguf`) | Lockstep rule: bump `AUDIOCPP_IMAGE_TAG` only together with a new image build, and set `AUDIOCPP_FORCE_BUILD=1` for exactly one deploy after each bump (see Gotchas). ## Image assembly `modal_app.py` builds the image as an ordered chain: 1. `modal.Image.from_registry(f"{IMAGE_NAME}:{IMAGE_TAG}")`, driven by env `AUDIOCPP_IMAGE` / `AUDIOCPP_IMAGE_TAG` (default `:v9`), with secret `brioso-registry` for the private Docker Hub pull. 2. `.entrypoint([])` trivial override (keeps the registry layer cacheable while letting Modal control container startup). 3. `.run_function(_bake_medium_into_image)` as the LAST layer. Layer ordering matters for cache hits: a tag bump invalidates the registry pull, but the entrypoint layer still resolves from cache, and only the GGUF bake layer rebuilds. Ordering the bake last is what makes bumps cheap. The bake function downloads `audio-cpp/audio.cpp-gguf` restricted to `*q8_0*.gguf`, then `rglob("*.gguf")` + `shutil.copyfile` (which follows the HF symlink) into `/app/audiocpp-model/.gguf`. Use `rglob`, not a single-depth glob: the HF snapshot layout nests as `snapshots///.gguf`, so `snapshots/*/*.gguf` finds nothing. Functions `generate` and `load_model` (warmup) both run `gpu="T4"`, `timeout=1200`, with `AUDIOCPP_MODEL_PATH` pointed at the baked `/app/audiocpp-model/stable-audio-3-medium-q8_0.gguf`. ### Warm pool `generate` carries `scaledown_window=300`: after a burst the container stays warm ~5 min, so consecutive jobs within that window reuse it. Scale-to-zero still applies afterward. `min_containers` is the documented-but-not-default lever (a warm T4 is ~$0.59/hr ≈ $425/mo/container); only turn it on if WS-C benchmarks show cold starts dominate. ## Deploy flow 1. Pin the image: `AUDIOCPP_IMAGE_TAG` in `modal_app.py` (default `:v9`). Keep lockstep with the built image (rule above). 2. If the tag changed, export `AUDIOCPP_FORCE_BUILD=1` for this deploy only (see Gotchas / Cache policy). 3. Bake layer: confirm the glob pattern matches only the q8_0 file you intend (no f16 strays). 4. Functions: `gpu="T4"`, `timeout=1200`, `AUDIOCPP_MODEL_PATH` correct. 5. Secrets: `brioso-audiocpp` (jobstore URL + shared secret) attached; `brioso-registry` present for the private pull. 6. Deploy: ```sh uv tool run --from "modal==1.3.5" modal deploy apps/modal-audiocpp/modal_app.py ``` 7. Smoke (below). If you set `AUDIOCPP_FORCE_BUILD=1` in step 2, unset it for the deploy after: the next deploy reuses Modal's cached image. CI deploys the same app via `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` repo secrets in `.github/workflows/deploy-modal-audiocpp.yml`. The `modal==1.3.5` exact pin (D11) exists because the SA3 app (`apps/modal-sa3/modal_app.py`) needs modal 1.3.x (>=1.4 dropped `modal.gpu`). This app is version-tolerant (string `gpu="T4"`), but the pin matches. ## What worked - Bake into image via `Image.run_function`: weights ride with the image, cold containers pay zero download cost. - `AUDIOCPP_MODEL_PATH` short-circuit in `handler_audiocpp._resolve_model_path`: when set, the handler skips the HF cache entirely and uses the baked file directly. - Handler `_materialize_model`: copies HF-cache symlinks to a real `.gguf` file. Required on the RunPod / home worker paths where weights come from the HF cache instead of a baked image. ## Gotchas and why ### HF cache symlinks break audio.cpp v0.7.3 `resource_bundle.cpp:36` calls `std::filesystem::weakly_canonical(path)` BEFORE `tensor_source.cpp:1494-1508` dispatches on `path.extension()`. The HF snapshot symlink resolves to the extensionless `blobs/` path, so `audiocpp_cli` throws `unsupported tensor source format: .../blobs/`. Upstream: 0xShug0/audio.cpp#274, fixed on main, unreleased as of v0.7.3. Workarounds: bake with a copy that dereferences symlinks (Modal, via `shutil.copyfile`), or `_materialize_model` in the handler (RunPod / home). ### HF_HUB_DISABLE_XET=1 must be set before import The Xet backend reads it at import time of `huggingface_hub`. The image sets it as an env var, so imports inside the bake function and handlers see it. Without it, blobs can land as 1 KB LFS pointer files or the download hangs. ### Modal image cache is keyed on the definition string Re-pushing a Docker Hub tag under the same name serves stale cached layers, because `from_registry` caches on the definition string, not the resolved digest. That is why force build exists: `AUDIOCPP_FORCE_BUILD=1` makes `from_registry` pass `force_build=True` for that deploy. Set it only for the deploy immediately after a tag bump, deploy, then deploy once more without it — Modal reuses the cached image. There is no always-on rebuild left in the code (the earlier temporary `force_build=True` leftover is gone). ### Snapshot nesting defeats shallow globs Covered above: `snapshots///.gguf` needs `rglob` or `snapshots/**/*.gguf`. ## Smoke testing `apps/modal-audiocpp/scripts/smoke.py` has three modes: - `preload`: calls `load_model` only. No GPU bill beyond the cold start. - `local`: uploads go to synthetic `127.0.0.1:1` stubs, so generation runs end to end and the upload step fails as expected. No jobstore traffic. - `generate`: full run with real upload; needs a `MODAL_SMOKE_SLUG` present in the jobstore. Prereqs: `MODAL_TOKEN_ID` + `MODAL_TOKEN_SECRET` in the environment, and the `brioso-audiocpp` secret created in Modal. ## Completion webhook + observability (BRI-382 phase 3) At the end of every real job, Modal's `generate` wrapper POSTs the terminal result to `{JOBSTORE_API_URL}/jobs/{slug}/complete?token=` (best-effort; a missed webhook converges via the reaper). The web app then bridges the confirmed transition to a Hatchet `job:completed:` event, which wakes the dispatch workflow's completion waiter, so paid transitions no longer wait out the reaper window. `adhoc-` prefixed slugs skip the webhook. Spans: the Modal wrapper emits `modal.boot` (cold first call only, `cold_start: true`) and `modal.generate` (`cold_start`, `slug`, `modal.container_id`) on the job's existing traceparent. Watch these in Phoenix to read cold vs warm share before deciding on `min_containers`. ## Cost A 10 s / 4-step t2t job on T4 is roughly $0.005 at $0.59/hr; a full `local` smoke run is around $0.01. `preload` costs only the cold start. ## Rollback Repoy to the previous image tag: revert `AUDIOCPP_IMAGE_TAG`, deploy once with `AUDIOCPP_FORCE_BUILD=1` (the old tag's cached layers may be evicted or stale), then deploy without it. No DB or schema coupling exists on the Modal side, so rollback is purely the app deploy. If Modal itself is the problem, the RunPod endpoints remain the falling-back path per `packages/jobs/AGENTS.md`. ## Follow-ups - ~~Revert `force_build=True`~~ Done: force build is now opt-in via `AUDIOCPP_FORCE_BUILD=1`; the code no longer always rebuilds. - No synthetic-GGUF test fixture: a probe against `audiocpp_cli` v0.7.3 showed it validates model resources deeply (requires a real `model_config.json` with keys like `sample_size`, T5 tokenizer sidecar files, and real tensor data), so a few-KB stub cannot get far enough past arg parsing to be useful in tests. The backend-override tests instead assert on argv construction. ## Contrast with the SA3 Modal app `apps/modal-sa3/README.md` uses the registry-image pattern with no bake because PyTorch tolerates HF cache symlinks. audio.cpp cannot (first gotcha above), which is the entire reason this app carries a bake layer. ## Benchmarks (WS-C) `apps/modal-audiocpp/scripts/bench.py` measures wall seconds per job shape (smoke 10s/4-step, medium 30s/6-step, worst 180s/8-step) against `brioso-audiocpp.generate`, for cold (first call after boot) and warm (median of 3 consecutive calls, calls 2-3) containers. ``` # dry run: matrix + T4 cost estimate, writes nothing uv tool run --from "modal==1.3.5" python apps/modal-audiocpp/scripts/bench.py --dry-run # real run (needs MODAL_TOKEN_ID/SECRET; bills real T4 seconds) uv tool run --from "modal==1.3.5" python apps/modal-audiocpp/scripts/bench.py ``` Reports land in `.agents/reports/audiocpp-modal-bench-.json` (same structure as the RunPod bench: `meta` + `results`), with a printed summary table showing cold-start share per shape: `(cold_wall - warm_wall) / cold_wall`. For a clean cold reading, run against a fresh deploy or after `scaledown_window` (300s) has expired, so no leftover container skews the cold number. The cold/warm numbers decide `min_containers`: a large cold-start share justifies keeping one T4 warm (~$425/mo); a small share means scale-to-zero is fine. --- ## [BRI-250 benchmark — latency, GPU/CPU energy, idle states (2026-08-08)](https://fitxers.usebrioso.com/internal/bri-250-benchmark-2026-08-08/) (Internal) # BRI-250 benchmark — latency, GPU/CPU energy, idle states (2026-08-08) Supersedes `bri-250-preliminary-gpu-benchmark-2026-08-07.md` for latency and GPU energy numbers (this run uses 10Hz GPU power sampling vs. the preliminary run's 1Hz, so it resolves short ~2.2s jobs far better — 2-3 samples per job at 1Hz badly undersampled the power ramp). Adds real CPU package energy via Intel RAPL (exact counter delta, no sampling error) and two of the four required operating states. **Still not done, and this doc does not claim otherwise:** State D (sustained/max-throughput, batch_size > 1), the 3-7 day continuous run, and a real wall-power-meter reading. Everything "whole-PC" below is a *software-measured DC total with an explicit, clearly-flagged margin applied* — not a wall measurement. See "Conscious gap" below for exactly what that margin does and doesn't cover, and why a real meter still matters before a final go/no-go call. ## Method `apps/home-gpu-server/scripts/benchmark_phase2.py`, run directly on the box (not over SSH, so latency isn't skewed by network round-trip) against the resident `home-gpu-server` service. 5 untimed warm-up calls, then 100 timed `POST /generate` calls at `duration=180s, steps=8` (the BRI-249/250 reference case), sequential (`max_concurrency=1`). Per call: CPU package energy via `/sys/class/powercap/intel-rapl:0/energy_uj` delta (exact, not sampled). GPU energy via `nvidia-smi --query-gpu=power.draw -lms=100` (10Hz) sampled continuously in the background and trapezoidal-integrated over each call's wall-clock window — this GPU/driver doesn't expose a cumulative energy counter the way RAPL does, so sampling is the only option there. Idle states A and B measured separately with `apps/home-gpu-server/scripts/measure_idle_state.py`: 30s of continuous 10Hz GPU power sampling + RAPL energy delta, with zero in-flight requests. ## Results: latency (100 runs, 180s/8-step, warm) | | min | avg | P50 | P95 | P99 | max | |---|---|---|---|---|---|---| | inference_seconds | 2.178 | 2.183 | 2.184 | 2.186 | 2.188 | 2.188 | | total_seconds | 2.178 | 2.183 | 2.184 | 2.186 | 2.188 | 2.188 | Extremely tight distribution (P99/min ratio ~1.004) — no cold-start, throttling, or GC-style jitter across 100 back-to-back generations. Confirms the preliminary run's 2.18s figure was not a one-off: **~28x faster than Runpod's ~60s/job baseline.** ## Results: energy per song (100 runs, DC-side, software-measured) | | min | avg | P50 | P95 | P99 | max | |---|---|---|---|---|---|---| | GPU (Wh) | 0.1356 | 0.1411 | 0.1391 | 0.1475 | 0.1482 | 0.1486 | | CPU package (Wh) | 0.00738 | 0.00758 | 0.00760 | 0.00773 | 0.00775 | 0.00775 | **Combined DC energy/song: 0.1487 Wh avg** (GPU + CPU package, no PSU loss or board overhead applied yet). Note this revises the preliminary run's GPU-only 0.085 Wh/song estimate upward to ~0.141 Wh/song — the earlier number came from 1Hz sampling against ~2.2s jobs (2-3 samples), which likely missed most of the power ramp. This run's 10Hz sampling (~21-22 samples/job) is materially more trustworthy. ## Results: idle states A and B | State | GPU avg W | CPU package avg W | Combined DC W | |---|---|---|---| | A — idle, model NOT loaded | 16.26 | 10.35 | 26.62 | | B — idle, model resident (warm) | 17.01 | 5.14 | 22.15 | Directly answers epic question #5 ("does keeping the model resident materially increase idle draw?"): **no** — GPU idle draw is nearly identical whether the model is loaded or not (16.3W vs 17.0W), since a resident-but-inactive model just sits in VRAM without drawing meaningful extra power. State B's *lower* CPU reading than State A is more likely measurement noise from a single 30s window right after a service stop/restart than a real effect — the eventual 3-7 day run will resolve this cleanly since it removes the boot-adjacent noise entirely. ## Conscious gap: estimating whole-PC from DC-side numbers Two things happen between "DC power the OS can measure" and "watts the wall meter would show," and neither is visible to software running on the box: 1. **PSU AC→DC conversion loss.** Assumed 80-90% efficiency (a defensible range for a decent-but-unrated/mid-tier PSU; a certified 80+ Gold unit would sit at the high end, an older/unrated unit at the low end — the real unit's rating isn't confirmed). This box's actual PSU has not been inspected for its efficiency certification. Still the single largest open unknown in this whole section — nothing below narrows it. 2. **Motherboard/chipset/VRM-beyond-CPU-rail/fans/storage/NIC ("board tax").** RAPL's package domain covers the CPU die and DRAM controller, but nothing else on the board. Originally a blind 15-35W guess; refined below into a bottom-up estimate grounded in the box's actual identified hardware. ### Board-tax breakdown (bottom-up, not a blind guess) Identified via SSH (`/sys/class/dmi/id/board_name`, `lsblk`): **ASUS MAXIMUS VII HERO (Z97 chipset)**, storage is 2x SATA SSD — Samsung 840 + Kingston SA400S3, no spinning HDD. No fan RPM telemetry available without force-loading a sensor kernel module (`nct6775`), which wasn't done — fan count/speed below is still an assumption, not measured. | Component | Estimate | Basis | |---|---|---| | Z97 PCH chipset | 4.1W (fixed) | [Intel's official Z97 spec](https://www.intel.com/content/www/us/en/products/sku/82012/intel-z97-chipset/specifications.html) — hard number | | Motherboard extras (VRM loss beyond RAPL, onboard audio codec, extra SATA/USB controllers) | 5-10W | Estimated — no direct spec for this board's non-chipset overhead | | Fans (3-4x 120mm, CPU + case) | 4.5-12W | [Industry-typical 1-3W/fan at moderate RPM](https://robots.net/tech/how-much-power-does-a-120mm-case-fan-use/) — count/RPM assumed | | Storage (2x SATA SSD) | 0.3-0.45W | [Kingston SA400 datasheet](https://www.kingston.com/datasheets/SA400S37_us.pdf): 0.195W idle (exact match); Samsung 840 ~0.1W idle (EVO/Pro family proxy) | | **Total board tax** | **13.9 - 26.6W** | vs. the original blind 15-35W guess | Storage turned out to be genuinely negligible — SATA SSDs draw well under 1W idle, far less than the original guess assumed. The remaining uncertainty is concentrated in fan count/RPM and motherboard VRM/controller overhead, neither of which can be pinned down further without either forcing the sensor driver or a real wall meter. Both ranges are explicitly conservative estimates, not measurements. A real wall meter replaces both with one number and removes this whole section — that's still the right next step before a final go/no-go, this is a stopgap so Phase 2 analysis isn't fully blocked on procurement. **Marginal cost/song, GPU+CPU only** (board tax excluded — it's fixed overhead, not incremental to one more generation; only PSU loss applies): | PSU efficiency | AC Wh/song | €/song | |---|---|---| | 80% (conservative) | 0.186 | 0.0000217 | | 90% (optimistic) | 0.165 | 0.0000193 | **Marginal cost/song, all components** (adds board tax's contribution *during the ~2.18s active generation window only* — not the idle-hours amortization below, just what the whole workstation draws while that one song is actually being generated): | Board tax | PSU efficiency | AC Wh/song | €/song | Songs per €1 | |---|---|---|---|---| | 13.9W (low) | 90% (optimistic) | 0.1746 | 0.0000204 | 48,971 | | 13.9W (low) | 80% (conservative) | 0.1964 | 0.0000230 | 43,530 | | 26.6W (high) | 90% (optimistic) | 0.1832 | 0.0000214 | 46,683 | | 26.6W (high) | 80% (conservative) | 0.2061 | 0.0000241 | **41,496 (worst case)** | Board tax barely moves this number — a song only takes ~2.2s, so even the high-end 26.6W board tax only contributes ~0.016 Wh to a ~0.15 Wh song. This is why "worst case" here (41,496 songs/€1) isn't far from the GPU+CPU-only worst case (46,083 songs/€1) — the *real* cost driver for "cost per song" is compute, not board overhead. Board tax matters far more for the *idle/24-7* number below, where it's paid every second regardless of whether a song is generating. **Monthly idle/keep-warm cost** (refined 13.9-26.6W board tax + PSU loss, using the realistic-run weighted idle baseline of 25.1W DC): | Board tax | PSU eff | AC W | €/mo | |---|---|---|---| | 13.9W (low) | 90% | 43.4 | 3.65 | | 13.9W (low) | 80% | 48.8 | 4.11 | | 26.6W (high) | 90% | 57.4 | 4.83 | | 26.6W (high) | 80% | 64.6 | 5.44 | **Working range: €3.65-5.44/month** to keep the box up and model-resident 24/7 — narrower than the original €3.5-6.5 blind-guess range. ## Runpod comparison | | Runpod (baseline) | Home GPU (measured) | |---|---|---| | Latency, 180s/8-step | ≥~60s | 2.18s avg (P99 2.19s) | | Cost/song (marginal, GPU+CPU only) | ~€0.025 | €0.0000193-0.0000217 | | Cost/song (marginal, all components) | ~€0.025 | €0.0000204-0.0000241 | ## Fully-loaded cost/song at volume (refined board-tax range) | Songs/month | Fully-loaded €/song (worst) | Fully-loaded €/song (best) | |---|---|---| | 100 | 0.0544 | 0.0365 | | 1,000 | 0.0055 | 0.0037 | | 5,000 | 0.0011 | 0.0008 | | 10,000 | 0.0006 | 0.0004 | | 50,000 | 0.0001 | 0.0001 | | 100,000 | 0.0001 | 0.0001 | **Crossover volume vs. Runpod's €0.025/song: ~146-218 songs/month** (refined from the original ~140-260 range) — even under the conservative end of the bottom-up assumptions. Below that volume the home GPU is still cheaper than Runpod as long as monthly usage clears roughly 146-218 songs — a bar the epic's own example (~300 generations/user/month for a single user) already clears on its own. This does **not** include Model B (full TCO with hardware depreciation/maintenance) — only Model A (marginal cash cost, existing hardware as sunk cost), per the epic's own framing. ## Realistic mixed-traffic run (24.4 min, 393 jobs) The clean 100-run sweep above is a controlled, uniform-load benchmark. This run instead simulates bursty real traffic against a single continuous power-sampling session (GPU 10Hz + CPU RAPL 1Hz running the whole time, so idle gaps between activity are captured as real idle power, not skipped): a 5-job concurrent burst, 3min idle, 1 single job, 1min idle, another 5-job burst, 5min idle, then 15min of back-to-back sequential load. Script: `apps/home-gpu-server/scripts/realistic_workload_bench.py`. | Phase | Type | Duration | Jobs | Combined DC W | Wh/song | |---|---|---|---|---|---| | burst_1 | 5 concurrent | 11.5s | 5 | 224.46 | 0.142 | | idle_gap_1 | idle | 180s | 0 | 25.58 | — | | single_job | single | 2.4s | 1 | 160.34 | 0.100 | | idle_gap_2 | idle | 60s | 0 | 31.70 | — | | burst_2 | 5 concurrent | 11.4s | 5 | 216.95 | 0.136 | | idle_gap_3 | idle | 300s | 0 | 23.52 | — | | sustained_load | sequential | 901.2s | 382 | 237.64 | 0.156 | **Queueing**: both 5-job bursts hit `max_queue_depth: 4` (1 processing + 4 queued) — the server absorbs simultaneous-request spikes by queueing rather than erroring, as designed (`max_concurrency=1`). **Idle cross-validation**: three independent idle windows averaged 25.1W combined DC (weighted by duration) — consistent with the controlled State A/B measurements above (21-27W) despite following real burst activity, not a clean stop/restart. Idle draw doesn't drift with traffic history. **Margin-adjusted economics from this run** (using the refined bottom-up 13.9-26.6W board tax — see "Board-tax breakdown" above): marginal cost €0.0000202-0.0000227/song (GPU+CPU only), monthly idle cost €3.65-5.44, crossover vs. Runpod's €0.025/song at **~146-218 songs/month** — this run's idle baseline (25.12W) is close enough to the clean benchmark's (25.1W) that, once both use the same refined board-tax range, the two crossover estimates converge almost exactly. Two differently-shaped tests landing on the same crossover band is a meaningfully stronger signal than either alone. ## What's still open before a final go/no-go 1. **Real wall-power-meter reading** — replaces the "conscious gap" section above with one measured number. Board tax is now a refined bottom-up estimate (13.9-26.6W) rather than a blind guess, but **PSU AC→DC conversion efficiency (80-90% assumed) is now the single largest remaining unknown** — nothing short of a real meter (or inspecting the PSU's own efficiency certification label) narrows that further. 2. **State D** — sustained/max-throughput, batch_size > 1. The current API is single-request (`max_concurrency=1`); testing this needs either a server-side batching change or running concurrent `sa3_worker` processes outside the resident-model service. 3. **3-7 day continuous run** — idle draw is now cross-validated across five separate windows (two 30-60s controlled snapshots plus three 1-5min windows from the realistic-traffic run, all landing in the same 21-32W band), which is stronger than a single snapshot but still not the sustained real-world average the epic asks for — also the only way to catch thermal drift, memory leaks, or driver issues a short benchmark can't surface. 4. Full TCO (Model B: GPU/PC depreciation, maintenance, optional UPS). The signal so far is strong enough that none of the above looks likely to flip the go/no-go call — electricity cost clears Runpod's bar by roughly two orders of magnitude even under conservative assumptions — but per the epic's own instruction not to skip Phase 2's rigor, these remain open before formally closing BRI-250. --- ## [BRI-250 preliminary GPU-only benchmark (2026-08-07)](https://fitxers.usebrioso.com/internal/bri-250-preliminary-gpu-benchmark-2026-08-07/) (Internal) # BRI-250 preliminary GPU-only benchmark (2026-08-07) Quick, informal sanity check run on the home GPU box (`gpu-worker`, RTX 4070 Ti Super) right after BRI-249's Phase 1 verification passed. This is **not** the rigorous Phase 2 benchmark BRI-250 asks for — it's GPU-telemetry-only, one machine state, one 10-minute window, no wall power meter. Keep as a reference/sanity baseline; BRI-250's actual DoD still needs the full multi-state, multi-day, whole-PC measurement below. ## Method Sequential `POST /generate` calls (`stable-audio-3-medium`, 8 steps, no sleep between requests — server's `max_concurrency=1` serializes them naturally) cycling through durations `[15, 30, 60, 90, 120, 150, 180]`s, for a fixed 600s wall-clock window. `nvidia-smi --query-gpu=... -l 1` sampled power/utilization/VRAM/temp once a second in parallel. Energy integrated via trapezoidal rule over the power samples. ## Results - 347 jobs completed, 0 failures, over a 599.1s window. - GPU power: avg 177.7W, max 262.4W (min 16.7W between requests). - GPU utilization: avg 60.2%, max 100%. - VRAM: constant 9,658 MiB (model resident throughout — no reload spikes). - GPU temp: avg 60.8°C, max 67°C — no throttling signs at ~60%+ sustained utilization. - Total GPU energy over the window: 29.588 Wh → €0.00346 (at €0.116952/kWh). - **GPU energy per song: 0.085 Wh → €0.00001/song** (GPU electricity only, excludes idle/24-7 keep-warm cost). Inference latency by requested duration (8 steps fixed; latency scales with duration, not step count — decoding/serialization dominates at longer lengths): | Duration | Avg inference time | |---|---| | 15s | 0.73s | | 30s | 0.79s | | 60s | 0.98s | | 90s | 1.12s | | 120s | 1.44s | | 150s | 1.80s | | 180s | 2.18s | 180s/8-step generation is the DoD reference case (BRI-249 #4): 2.18s warm vs. Runpod's ~60s baseline, ~28x faster. ## What this does NOT answer yet Per BRI-250's actual scope, still needed before a go/no-go decision: 1. **Whole-PC wall power** (HIGH PRIORITY per the epic) — GPU telemetry misses CPU, motherboard, RAM, SSD, fans, PSU losses. Needs an external smart plug / wall meter. 2. The 4 required operating states — idle/no-model, idle/model-loaded (warm), active inference, sustained max throughput — each with whole-PC W, not just GPU W. 3. Multi-day (3-7 day) continuous run to capture real 24/7 availability cost, not a 10-minute burst. 4. Marginal vs. fully-loaded cost/song, and the two TCO models (marginal cash cost vs. full economic cost with depreciation). 5. Batch-size > 1 capacity benchmark. 6. The actual Runpod comparison table (this doc only has the home-GPU side filled in). This preliminary run does suggest the GPU-electricity component alone is not going to be the bottleneck in the economics — the open question is almost entirely about idle/24-7 draw, which whole-PC measurement will answer. --- ## [Buy, don't build](https://fitxers.usebrioso.com/internal/buy-dont-build/) (Internal) # Buy, don't build Before writing a script, wrapper or helper, look for: 1. A built-in: OS tool, language stdlib, or an existing dependency. 2. A GitHub-backed tool that is actively maintained and has 50+ stars. Use that. Build only when neither exists or fits, and say why in the PR. Example: max RSS and CPU come from `/usr/bin/time` (`-v` on Linux, `-l` on macOS), not a custom measurement wrapper. For timelines use `psrecord`, for hardware counters `perf stat`. --- ## [CI runner policy](https://fitxers.usebrioso.com/internal/ci-runner-policy/) (Internal) # CI runner policy Goal: GitHub Actions never costs money. Ladder, in order: 1. **Self-hosted tailnet runners** (`RUNNER_TARGET=self-hosted-tailnet`): the default. The OVH Garage VPS adds one slot to the application VPS fleet of four. 2. **GitHub Free included minutes** (`RUNNER_TARGET=ubuntu-latest`): a manual fallback with 2,000 Linux min/mo on the `briosoco` Free-plan org. 3. **Hard stop**: an org Budget with `prevent_further_usage: true` on the actions SKU. Even if the watchdog fails, spend stops at zero. The `blacksmith-4vcpu-ubuntu-2204` runner config stays deleted; `RUNNER_TARGET` must never return to that value. ## Watchdog `.github/workflows/runner-budget.yml` runs every 6h (`scripts/ci-minutes-remaining.ts`): - remaining < LOW_BUFFER (100 min) AND a `self-hosted-tailnet` runner is online → PATCH `RUNNER_TARGET` to `self-hosted-tailnet` after a manual hosted fallback - while on `self-hosted-tailnet` → keep it there across billing cycles - billing API unreadable → the workflow FAILS LOUD (never assumes minutes remain). RUNNER_TARGET is left untouched. The thresholds are minutes, not a boolean: `LOW_BUFFER` = `LAST_FULL_PIPELINE_MINUTES` (20, measured 2026-10-10) x `PIPELINES_PER_CHECK_WINDOW` (5), so hosted stays in use only while at least 5 full pipelines of minutes remain. Re-measure and adjust both constants in `scripts/ci-minutes-remaining.ts` when CI cost or push rate changes. Manual check (dry-run, never PATCHes): `mise run ci:minutes`. Secret/variable requirements: - `MINUTES_PAT`: fine-grained PAT with org **Administration: read** (billing) and **Variables: RW** (the RUNNER_TARGET PATCH; `GITHUB_TOKEN` cannot do either). - `RUNNER_TARGET`: repo **variable** (not secret), value `ubuntu-latest` or `self-hosted-tailnet`. All workflows read `${{ vars.RUNNER_TARGET || 'ubuntu-latest' }}`. One-time hard-stop setup (run once, verify after billing-platform changes): ``` gh api -X POST /orgs/briosoco/settings/billing/budgets \ -f budget_product_sku=actions -F prevent_further_usage=true ... gh api /orgs/briosoco/settings/billing/budgets # verify ``` If no card is on file, GitHub Free also blocks Actions at 0 min by itself; the budget is defense in depth for when that changes. ## Application VPS self-hosted runners (historical) Superseded by [brioso-vps runners](#brioso-vps-runners-scale-to-zero-bri-433): the `ovhcloud-ghrunner-*` fleet is decommissioned and CI runs on the Garage VPS. The recipe below is kept for the sizing rationale and security model. See the [infrastructure map](infrastructure.md) for host placement, accounts, and the dated registration snapshot. The application VPS (`ovhcloud`) runs 24/7 in the tailnet with bun/uv/mise and the turbo cache on localhost. The following runner model and setup recipe apply to that host. Runner facts: - Outbound-only: the runner long-polls github.com; no inbound ports needed. - The runner container runs as root with sudo available; no deploy keys and no secrets in env. Private-repo self-hosted runners execute arbitrary workflow code, so the container boundary IS the security boundary (labels are not). - **4 registrations** on the 6-core, 11.7 GB box (label `self-hosted-tailnet` on all) so a push's ~12 jobs don't queue behind 2 slots. Not 5: with the VPS's other services at about 2.1 GB RSS, a fifth unconditional slot's worst case exceeds the available ~8.9 GB. Add a 5th only if the `CI_MEASURE=1` numbers show light jobs under ~1.2 GB and `bun scripts/ci-queue-wait.ts` p90 is still above 2 min. - **Fresh container per job** (BRI-449): rootless Quadlet units `deploy/ci-runner/ghrunner@.container` (template, instances 1 to 4) run `ghcr.io/briosoco/brioso-ci-runner` (built and published by `.github/workflows/ci-runner-image.yml` from `deploy/ci-runner/Containerfile`: `ubuntu:24.04` (same glibc as `ubuntu-latest`) + the official actions/runner tarball + mise-baked bun/node/uv/python, nothing else — no ffmpeg, docker, pwsh, awscli or gcc) with `EPHEMERAL=true`, `--rm` and `Restart=always`. The runner exits after one job, the container is destroyed and systemd recreates it with a new registration. - The PAT comes from a podman secret (`gh-runner-pat`), mounted at `/run/secrets/gh-runner-pat`; the image entrypoint mints registration tokens from it — the PAT never sits in the unit or on disk in the clear. - **Shared bun cache:** the ONLY volume is `ci-bun-cache`, mounted at `/root/.bun/install/cache` and shared by every job on every runner. One volume, not per-job volumes: per-job caches would re-download ~500 MB of dependencies every run, and the poisoning risk of a shared cache is accepted because same-repo PRs are opened only by the owner — poisoning would need a compromised dependency or account, and those already reach the build directly. - **Slice limits** (`deploy/ci-runner/user-997.slice (was user-995.slice)`): `MemoryHigh=6.5G` (throttles, never kills), `MemoryMax=8G`, `MemorySwapMax=1G`, `CPUWeight=40`, `IOWeight=40` on the `ghrunner` user's logind slice, covering all containers. Each container also has a hard `--memory=5g` so a runaway job is OOM-killed alone. - **Heavy lock:** the vite build and the boot smoke (~4 GB each) run under `flock -w 900 /root/.bun/install/cache/.heavy-build` on self-hosted runners, so only one is in flight fleet-wide (lock file on the shared volume, never a GitHub concurrency group, which cancels queued deploy builds). - **Per-slot memory env:** the entrypoint writes the runner's `.env` from `CI_SLOTS` and `CI_SLICE_MAX_MB` (set in the Quadlet template): `GOMEMLIMIT`, `GOMAXPROCS`, `TURBO_CONCURRENCY`, `RAYON_NUM_THREADS`, `VITEST_MAX_WORKERS` and a `NODE_OPTIONS` heap cap, so tsc, tsgolint, turbo, oxlint and vitest size themselves to slice-max / slots. Tune the values from the max-RSS table that `mise run validate` prints per step (TS and Python, via `/usr/bin/time -v`; `CI_MEASURE=0` opts out). The tool is OS-level, so it works on any process, not just these gates: `/usr/bin/time -v ` on Linux (max RSS in KB, `Percent of CPU`), `/usr/bin/time -l ` on macOS (bytes). For a timeline use `psrecord`, for hardware counters `perf stat`. Hosted runners never see the `.env`. - **Queue watcher:** `ci-queue-watch.{sh,service,timer}` (every 2 min, outside GitHub) emits a JSON status line, reaps stale offline registrations and pings Healthchecks.io `ci-runner-queue` (success, or `/fail` when the queue is non-empty with no runner and no healthy orchestrator, or a run has been queued over 6 h). It reads `HC_PING_KEY` from `~ghrunner/.config/ci-queue-watch.env`. Details in the brioso-vps section below. - Two timers (`deploy/ci-runner/`): `ci-runner-image-pull.timer` pulls the published `:latest` daily (the workflow rebuilds it weekly to keep the actions/runner version inside GitHub's support window; a pull only retags, so running jobs are never interrupted); `ci-bun-cache-wipe.timer` wipes the cache volume monthly (safe while bun runs; replaces the old nightly prune). - Tailscale ACL: tag the node `tag:ci`, egress to github.com + turbo cache only. Setup recipe (Quadlet form — the `@` template needs podman >= 5.2; the `Secret=` unit key needs >= 4.8). Cutover: start instances 1 and 2, then `podman stop` (not rm) the old myoung34 pair, add 3 and 4 while watching `free -m` and the slice's `memory.peak`; remove the old containers after a green week: ``` # on the VPS, as the ghrunner user (uid 995) podman pull ghcr.io/briosoco/brioso-ci-runner:latest podman secret create gh-runner-pat - # PAT on stdin, never on disk cp deploy/ci-runner/ghrunner@.container \ ~/.config/containers/systemd/ cp deploy/ci-runner/*.{service,timer} ~/.config/systemd/user/ install -D deploy/ci-runner/ci-queue-watch.sh ~/.local/lib/ci-runner/ci-queue-watch.sh systemctl --user daemon-reload systemctl --user start ghrunner@{1,2,3,4}.service systemctl --user enable --now \ ci-runner-image-pull.timer ci-bun-cache-wipe.timer ci-queue-watch.timer # as root: install user-997.slice as the drop-in, then systemctl daemon-reload ``` The retired recipe — `docker run --restart=unless-stopped myoung34/github-runner` — combined `EPHEMERAL=true` with a same-container restart, which was the documented root cause of both the ~16 GB writable-layer growth and the tamper gap: state (mise/bun binaries) persisted across jobs, so a root PR job could poison a later secret-bearing build. `--rm` + `Restart=always` is the fix. Watchdog behavior if the runner is down: it will NOT flip `RUNNER_TARGET` (it checks `GET /repos/.../actions/runners` for an online runner carrying the label first). If minutes are exhausted AND no runner is online, jobs fail rather than bill — acceptable, that is what the budget hard stop would do anyway. ## brioso-vps runners (scale to zero, BRI-433) `brioso-vps-ovh` (the Garage VPS, 8 vCPUs, 23 GiB RAM) is the CI host. The four `ovhcloud-ghrunner-*` runners and the old single slot are gone; nothing queues on them. The `ghrunner` user is UID 997 here. **Model.** Nothing runs when idle. The orchestrator (`deploy/ci-runner/ci-orchestrator`) starts one fresh ephemeral container per queued `self-hosted-tailnet` job, at most 4 at once, and each container is removed when its job ends. A `workflow_job` webhook (HMAC verified by adnanh/webhook, only `/hooks/github` exposed through the Cloudflare tunnel) cuts latency; the 60 s `ci-reconcile.timer` polls GitHub and is the source of truth, so a lost delivery, a dead tunnel or a reboot only delays a job by up to a minute. Measured idle cost of four always-on runners was about 100 MB RAM and 0.2 core, so the orchestrator is justified by clean per-job state and no accumulated cache or layers, not by idle savings. K3s with ARC was rejected: its server alone costs more than the idle fleet, `dind` needs privileged pods, and it adds a second orchestrator. **Off switch and fallback.** `ci-runners.target` is the always-on fleet (`ghrunner@1..4`), installed but disabled. Disable `ci-reconcile.timer` and `ci-webhook.service`, enable the target, and the orchestrator stays idle while it is active. Stopping the target drains every runner; never mask. Crash loops stop after 5 failures in 15 min and are reported by the watcher. Job containers get `--device /dev/fuse` (when the host has it) so the runner image self-build can use buildah on `fuse-overlayfs` instead of `vfs`, which needed about 3x the image size on the shared `/srv/ci` (BRI-449 follow-up, `.agents/plans/2026-10-17-ci-runner-image-disk.md`). **Guard.** Before each start: `/srv/ci` mounted, over 3 GiB free on root, over 5 GiB free and 200000 inodes on `/srv/ci`, over 2 GiB MemAvailable. A failing guard leaves the job queued and the watcher alerts. `user-997.slice` bounds the fleet (`MemoryHigh=13G`, `MemoryMax=16G`); each container has a 5 GiB ceiling (a limit, not a reservation) and `RuntimeMaxSec` (2 h) caps a hung job. **Credentials.** The long lived fine-grained PAT (repo Administration RW, Actions read) lives at `/home/ghrunner/.config/brioso/github-runner-pat`, mode 0600. The wrapper mints a short lived registration token per container; the PAT is never mounted into a job (workflow code runs as root in the container and could read it). The webhook secret is in `~ghrunner/.config/ci-orchestrator.env`, mode 0600. **Visibility.** `ci-fleet-status` prints units, containers, registrations, queue, disk and memory in one table. `ci-queue-watch.sh` writes a JSON line per run to journald and `/srv/ci/state/ci-fleet.jsonl`, and alerts on a non-empty queue with no runner and a stale orchestrator, or any run queued over 6 h. `runner-budget` treats a healthy orchestrator (Healthchecks badge `CI_ORCHESTRATOR_BADGE_URL`) as runner capacity, since zero runners online is normal. See `deploy/ci-runner/README.md` for install and recovery. ## Monitor placement (BRI-401) Monitors (`runner-budget`, `vps-health`, `pyinfra-drift`) do NOT ride `vars.RUNNER_TARGET` — a monitor that shares the fleet's runner dies with the very outage it exists to report (zero minutes → hosted job never queues; dead VPS runner → no run at all). All three are pinned to `self-hosted-tailnet`, on `schedule`/`workflow_dispatch` triggers only — never `pull_request`, because a PR-triggered run on the box holding the passwordless-sudo pyinfra key would be an exfiltration path. `runner-budget` is self-hosted outright (it needs only the billing API, so it runs even at zero hosted minutes); `vps-health`/`pyinfra-drift` reach secrets only via the GitHub `environment:` (deployment branch policies gate the refs that can resolve `VPS_PYINFRA_SSH_KEY` etc.). Dead-man heartbeat: each monitor ends with `curl -fsS -m 10 "https://hc-ping.com/$HC_PING_KEY/$JOB"` on success only (`HC_PING_KEY` is the bare Healthchecks.io ping key; the same key feeds `ci-queue-watch` on the VPS, so there is no URL secret) — no ping, and a Healthchecks.io check (outside GitHub) fires the alert. `JOB` is `runner-budget` / `vps-health` / `drift-beta` / `drift-prod`. Steps are skipped while the `HC_PING_KEY` secret is unset; the owner creates the checks and stores the secret. The tailscale step in `vps-health`/`pyinfra-drift` is `if: runner.environment == 'github-hosted'` (the self-hosted box is already on the tailnet), followed by a `tailscale status` diagnostic on hosted runs — the "Connect to tailnet" failures' root cause (OAuth client, `tag:ci` ACL, or the v4 `ping` input) is still open; owner must check the TS OAuth client. ## Trigger volume (reduced 2026-09, BRI-356) - Deploy beta: dispatch-only (legacy push path removed during deploy-v2). - Security Gates: PR-only (push:main duplicate dropped). - CI validate: `paths-ignore` for `**/*.md`, `docs/**`, `.agents/**`, `**/*.svg`, `videos/**`. - Slow tests: nightly cron disabled (commented in `test-slow.yml` with a re-enable note); PR paths-filtered trigger remains. - pyinfra drift: daily at 06:17 UTC (was hourly); post-deploy drift still runs per deploy in `pyinfra-deploy-beta.yml`. To re-enable a disabled schedule, uncomment the cron in the workflow file; to check what a change will cost, `gh run list` before/after a week. ## macOS/Linux parity Root `pyproject.toml` declares `[tool.uv] environments` for both `darwin-arm64` and `linux-x86_64`, making the lockfile universal. `scripts/validate.py` runs `uv lock --check` on every invocation: it resolves the Linux dependency graph on the laptop in ~1s with no wheel downloads, so a platform marker that breaks Linux resolution fails locally instead of only surfacing after a push to CI. The residual platform-only set that parity can't (and shouldn't) close — these are real, intentional differences, not gaps: | Package | Platform | Where | | -------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------- | | `mlx` | darwin only | `packages/stable-audio-3`, `packages/musicgen-adapters` | | `rumps`, `wakepy` | darwin only | `apps/worker` | | `mflux-image` | darwin arm64 only, excluded from the workspace entirely | `packages/mflux-image` | | `torch`/`torchaudio` → `pytorch-cu128` index | linux (and win32) only | `packages/stable-audio-3` — CPU wheels substituted in CI, see `test-slow.yml` | `uv tree` on macOS vs. the VPS for the same commit should only ever diverge on this list; anything new appearing in that diff is a regression. ## Agent escalation order for platform-sensitive changes Never push to observe behavior — push only to land batched work. For anything platform-sensitive (Python deps, torch, native modules, Nitro output): 1. `mise run validate:quick` locally, `mise run dev:preview` for build/SSR shapes. `uv lock --check` (now part of `validate:quick:py`) already catches the most common "works locally, fails in CI" class: dependency resolution divergence. 2. Still platform-sensitive after (1)? Run it on the VPS over the tailnet (`ssh ovhcloud`, same `mise run `) — same OS/arch/uv resolution, warm caches, seconds instead of a push-build-deploy round trip. 3. VPS-only runtime bugs: `mise run deploy:beta -- ` on the box, then `mise run ops:health`. 4. Push to land work, batched — never to observe it. ## Agent conventions Batch pushes and use `[skip ci]` on docs-only intermediate commits; see AGENTS.md. --- ## [CI runner profile](https://fitxers.usebrioso.com/internal/ci-runner-profile/) (Internal) # CI runner profile Baseline resource profile for the shared-VPS CI path — Phase 0 of [`.agents/plans/2026-10-08-ci-performance.md`](../plans/2026-10-08-ci-performance.md) (BRI-449). These numbers are indicators, not gates; per-run values come from the resource-report steps and `scripts/ci-step-timings.ts`. ## Baseline (measured 2026-10-08, last 3 green build-artifact runs on PRs) | Job | Wall time | Step breakdown | |---|---|---| | `build` | ~1m40s | mise-action 22-26s · `bun install` ~5s · "Build apps/web" 56-60s · compile activate ~1s · dbtools bundle 0-4s · pack ~11s · release publish ~0s | | `boot-smoke` | ~70s | mise ~22s · artifact download 6-14s · netns boot smoke 20-29s | | `smoke-test-artifact` | ~65s | artifact download ~8s · activate-subcommands drill ~29s | | `deploy-pyinfra` (push) | ~5m total | build + drift-gate + deploy serially | ## Reading the memory numbers `memory.peak` (cgroup) counts page cache, so it overstates real usage — that is why the report prints the `memory.stat` split. Treat **anon + shmem** as the process footprint; `file` is mostly reclaimable page cache (source files, `.output` trees, the bun cache) and only matters when PSI says the kernel is stalling on reclaim. ## Top consumers per axis - **Time**: the `apps/web` vite/rolldown build (the single biggest step), the mise tool restore (paid once per job, and the smoke jobs serialize after `build`), and the serialized smoke jobs themselves (each re-downloads the artifact and re-does tool setup). - **Memory**: the vite/oxc build and the bun install/cache population. - **Disk**: docker writable layers on the runner containers, `.output` trees, android/dotnet leftovers on hosted images. - **Network**: `bun install`, the ffmpeg tool install, and apt fetches. ## Soft thresholds | Signal | Warn | Hard max | |---|---|---| | `memory.stat` anon + shmem | `::warning::` above 4.5 GB | 6 GB (target for `MemoryMax` in Phase 2) | | memory PSI `some` avg60 | `::warning::` above 10 | — | ## How the scripts are invoked In workflows (after `jdx/mise-action`, which is what provides bun): ```yaml - name: Resource report (start) # add --sample for the 2s sampler if: always() run: bun scripts/ci-resource-report.ts start # ...job steps... - name: Resource report (end) if: always() run: bun scripts/ci-resource-report.ts end ``` Per-step timings for before/after tables (read-only, needs `gh` auth): ```sh bun scripts/ci-step-timings.ts --workflow build-artifact.yml --runs 10 bun scripts/ci-step-timings.ts --workflow build-artifact.yml --job build ``` --- ## [Client/server boundary](https://fitxers.usebrioso.com/internal/client-server-boundary/) (Internal) # Client/server boundary How Brioso keeps Node-only code out of the browser bundle and browser-only code out of the server. Enforcement lives in `@brioso/env-guards` (sentinel), the oxlint client-glob restrictions, the slimmed `client-server-boundary` bundle-marker tripwire, and `bunx vite build` in CI. ## Incident history Two production incidents motivated this architecture. On 2026-08-24 a beta deploy broke: `@aws-sdk/lib-dynamodb`'s `fromNodeProviderChain` (pulled in via `@brioso/db` -> `@brioso/config` -> `@infisical/sdk` through a static import chain reachable from client components) failed to resolve in the browser bundle, hard-failing the build. On 2026-09-11 a posthog-node import reached the client graph and crashed at hydration. Both were the same class of bug: a static import of server-only code from a module that TanStack Start compiled into the browser bundle. ## The three zones Every module is exactly one of: - `/shared` — browser and server safe. No Node builtins, no browser-only globals. - `/client` — browser only. Reached from server code = leak. - `/server` — Node only. Reached from the client graph = leak (the common case). ### Per-package semantics | Package | Zones | Notes | | ------------------ | ----------------------------- | ----- | | `@brioso/db` | server-only + `/onboarding` split | root `.` and `/client` are server (sentinel on the barrel); `/onboarding` is pure browser-safe reference data (BRI-238 country/enum lists); `/testDb` is test-graph only | | `@brioso/storage` | server-only (whole package) | `/types` and `/zip` are pure; `/r2` etc. pull aws-sdk; all wired entries carry the sentinel | | `@brioso/dispatch` | server-only (whole package) | fully sentinel-wired | | `@brioso/jobs` | server-only + pure subpaths | sentinel on queue, lease-queue, completion, creation, delivery, lookup, notify, receipt, recovery, thread, thread-access, thread-history, tracks, fallback; browser-safe exempt subpaths: `/constants`, `/disposableEmailDomains`, `/duration`, `/customMusicRequest`, `/editing-params`, `/errors`, `/email`, `/state`, `/validation`, `/retention`, `/quota-read`, `/thread-id`, `/testing` | | `@brioso/billing` | server-only (whole package) | stripe SDK; webhook/subscription-checkout/portal sentinel'd (Phase 1) | | `@brioso/entitlements` | server-only (whole package) | sentinel on promo-codes/redeem/resolve; redeem-fn is *Fn-stub reachable so no bare sentinel (see exemptions below) | | `@brioso/ai-workflow` | server-only (whole package) | sentinel on run/model/run-agent/refine-config/preferences; `/types`, `/errors`, `/refine-schema`, `/preferences-schema` are pure | | `@brioso/obs-js` | split | `/client` (posthog-js), `/shared`, `/analytics-identify`, `/useFeatureFlag` browser-safe; `/server`, `/nitro`, `/logtape` server | | `@brioso/auth-core` | split | `/otp`, `/rate-limit`, `/session-check`, `/route-guard` etc. shared; `/session-server`, `/session-config-bound`, `/worker-secret` sentinel'd | | `@brioso/sharing` | split | `/token` and `/types` pure; `/store` sentinel'd with lazy table resolution (its old static `shareLinks` import bridged drizzle into the client graph) | | `@brioso/contract` | isomorphic (model only) | imported directly by web + worker; never served over HTTP | Lint layer (Phase 2): a client-glob oxlint override bans value-imports of `@brioso/db`, `@brioso/obs-js`, `@brioso/storage`, `@brioso/billing`, `@brioso/dispatch`, `@infisical/sdk`, `stripe`, `posthog-node`, `@aws-sdk/*`, `@brioso/db/client`, `@brioso/db/testDb`, `@brioso/obs-js/{server,nitro,logtape}` and `@brioso/auth-core/session-server` from `apps/web/src/{components,routes}` and `packages/verticals/*/src/{components,routes}`. Type-only imports are erased and always allowed. ### Known exemptions (documented, reviewed) - **`*Fn` files and package fn modules** (`@brioso/billing/billing-fn`, `@brioso/entitlements/redeem-fn`, studio `server/session.ts`, ...) are legitimately graph-reachable: the compiler replaces handler bodies with client stubs. They must keep every server import dynamic inside handler bodies and must NOT carry a bare `@brioso/env-guards/server` side-effect import. The browser-condition throw stub would otherwise be merged into a client chunk by rolldown and evaluated at page load (found during the Phase 2 sweep). Server-only-ness for these files is enforced by the lint layer + boundary plugin instead. - **The dynamic `@brioso/db` chunk** still exists in `.output` client assets (onboarding-fn's server-only helpers import tables inside functions). It is reachable only behind `*Fn` stub call sites that never execute in the browser; the boundary plugin's generateBundle marker assert keeps that chunk free of posthog-node/node builtins. New packages default to one zone; split only when a real consumer needs both. ## The sentinel: `@brioso/env-guards` `packages/env-guards` exports two sentinel subpaths using package.json export conditions: - `@brioso/env-guards/server`: `default` resolves `src/empty.ts` (a no-op, free on the server); `browser` resolves `src/throw-server.ts`, which throws with the fix if it is ever reached from a client graph. - `@brioso/env-guards/client`: inverse (`node` throws, `default` empty). Declaring server-only-ness is now one line (`import "@brioso/env-guards/server";` as the first import) plus a doc-table row. It is vendored rather than the npm `server-only` package because TanStack Start does not set the `react-server` condition that package keys on; the `browser` condition is the one vite reliably resolves for the client environment. Recipe for a new server-only entrypoint: 1. First import in the file: `import "@brioso/env-guards/server";` 2. Add `@brioso/env-guards` as a `workspace:*` dependency of the package. 3. Add a row to the table above. ## Rules for apps/web - `createServerFn()` splits **per handler body, not per module**. Only the `.handler(fn)` closure becomes a client stub; the rest of the file ships to the browser. - Server imports go **inside the handler body** as `await import(...)`, or in a non-exported helper referenced only from handler bodies. - Files whose value is a `createServerFn` call end in `*Fn` (`billing-fn.ts`, `refine-fn.ts`). Their module-level code still runs in the browser, so they must not import server modules statically either. - Server-only app code lives under `apps/web/src/server/**`; the boundary plugin blocks client resolution of anything climbing into that tree. ## Enforcement map (final, Phase 3 — 2026-09-11) Layered, cheapest first: 1. **Sentinel** (`@brioso/env-guards`): runtime/build-time throw if a server module is resolved in the wrong environment. Wired into ~38 server entrypoints across db, storage, obs-js, auth-core, billing, jobs, sharing, entitlements, dispatch, ai-workflow. 2. **oxlint `no-restricted-imports`**: static-grep-level ban on forbidden specifiers in client-reachable files (Phase 2). 3. **Bundle-marker tripwire** (`apps/web/vite-plugins/client-server-boundary.ts`, slimmed in Phase 3): post-build `generateBundle` assert that fails the build if an emitted client chunk contains a server-only marker (`posthog-node`, `POSTHOG_PERSONAL_API_KEY`, `node:async_hooks`). Its old resolveId package denylist was removed — the sentinel owns that. 4. **CI `vite build`**: the whole pipeline runs on every web change. Retired enforcement layers: - `scripts/check-client-server-leak.ts` (static grep) — deleted Phase 3 (2026-09-11); superseded by the lint layer (2). - `apps/web/vite-plugins/server-only-module-stub.ts` denylist (@infisical/sdk, @aws-sdk/* stubs) — slimmed Phase 3 (2026-09-11) to only `@tanstack/start-server-core`, which still value-reaches the client graph via `apps/web/src/lib/taxonomy-cache.ts` (root specifier re-exports `request-response.js`, which imports `node:async_hooks`). Delete the plugin when taxonomy-cache stops importing the root specifier. Escalation triggers for adding new bundler-level checks — each requires a plan doc: - a leak class the sentinel cannot see (e.g. dynamic-import chains entered via non-condition paths); - a third environment (worker, edge runtime) with different conditions; - two or more sentinel false-positive incidents in a quarter. Hard rule: **denylists must never accrete in plugin code.** New server-only-ness is one sentinel line plus a doc-table row. The Phase 3 retired the last plugin denylist (`isServerOnlySpec`); only the marker list remains, and it must stay at the three node-identity markers above. ## False positives and exemptions Known-safe patterns that must not trip the guards: - **Type-only imports** (`import type`) are erased at compile time. - **`vi.mock`** in tests never reaches a bundler. - **`createTestDb()`** (`@brioso/db/testDb`) and **`@brioso/jobs/testing`** are test-graph only, never imported from app code. - **`*Fn` files** under `apps/web/src/server/**` are legitimately graph-reachable — the compiler replaces handler bodies with client stubs. --- ## [Code search tool policy](https://fitxers.usebrioso.com/internal/code-search-tools/) (Internal) # Code search tool policy Which search tool to use when working in this repo. All three are installed: | Tool | Binary | Use for | | --- | --- | --- | | ripgrep | `rg` | Default. Token/regex search across the repo. Fast startup, respects .gitignore. | | ugrep | `ug` / `ugrep` | Only when you need features rg lacks: fuzzy/boolean search, searching inside archives (zip, tar, pdf, docx via `-z`), interactive TUI. Do not reach for it by default. | | ast-grep | `ast-grep` / `sg` | Structural code search: patterns with metavariables, AST-level find/replace, lint rules. Skip regex guessing when you need syntax. | ## ast-grep quick reference Patterns use `$METAVAR` placeholders. Always pass `-l ` (ts, tsx, py). ```sh # structural search: every try/finally cleanup site ast-grep run -p 'try { $A } finally { $B }' -l ts apps packages # every fan-out site ast-grep run -p 'Promise.all($A)' -l ts apps packages # rewrite (dry-run prints diff; add --update to apply) ast-grep run -p 'obj dispose()' -r 'obj[Symbol.dispose]()' -l ts packages/jobs # project-configured rules and their severities ast-grep scan apps packages ``` Rules live in `sgconfig.yml` (repo root) and can be wired into `mise run validate` like any other linter. To add a new structural lint rule, add it to `sgconfig.yml` first and only write ad-hoc `ast-grep run` commands for one-off queries, not for rules that should be enforced on every push. ## Choosing fast - Searching for a symbol name or error string → `rg`. - Searching for a code shape (call with a specific argument count, nested try/finally, unawaited promise) → `ast-grep`. - Searching inside a compressed artifact or wanting boolean/fuzzy matching → `ugrep -z` / `ug -Q`. --- ## [Component-driven workflow](https://fitxers.usebrioso.com/internal/component-driven-workflow/) (Internal) # Component-driven workflow React Cosmos 7.4.1 is the component sandbox for `packages/ui` primitives. Decision rationale: [`.agents/notes/2026-09-10-component-driven-research.md`](../notes/2026-09-10-component-driven-research.md). Plan: [`.agents/plans/2026-09-10-component-driven-cosmos.md`](../plans/2026-09-10-component-driven-cosmos.md). ## The rule A primitive is developed in **Cosmos first**, then composed into a screen. The fixture is the contract; the screen is the consumer. **Verified means**: the primitive author has seen and exercised the base state AND every other state (variants, sizes, error/disabled/loading) in Cosmos BEFORE screen code lands. A PR that adds a primitive + screen usage in the same commit but no fixture exercise is a defect — bounce it. The CVA config is the variant source of truth. Map over its keys in the fixture; do not copy-paste `variant="…"` strings. Kitchen-sink research established that a stale matrix is worse than no matrix; the Cosmos feedback loop catches the gap on the next run. ## Primitive-author vs screen-author - **Primitive-author** (`packages/ui/AGENTS.md`): write the primitive, update or create the fixture, run Cosmos, exercise base + each variant + dark mode. Only then commit. - **Screen-author** (`apps/web/AGENTS.md`): import from `@brioso/ui/ui/`. If the primitive does not exist (or needs a new variant/size), STOP and switch to the primitive-author side. ## Cosmos setup - `cd apps/web && bunx cosmos` — Vite dev server on `localhost:5000` (or 5001 if taken). The navigation tree lists every fixture under `src/cosmos/`; the controls panel exposes props so you exercise the matrix without editing the fixture. - New fixture: `apps/web/src/cosmos/.fixture.tsx`, Multi-Fixtures format (`export default { Primary: <…/>, ... }`). Import the primitive from `@brioso/ui/ui/` (the package exports subpath) so the sandbox exercises the built primitive, not a local copy. - Directory-tree decorator (`apps/web/src/cosmos/cosmos.decorator.tsx`) wraps every fixture in the providers it needs (`TooltipProvider`, theme via `options.theme`). For per-fixture locale, export `options: { locale, messages }` from the fixture (Cosmos Fixture Options, PR #1649). ## `/cosmos/` route Dev-only, gated three ways: 1. `sensitive-route-guard` (`/cosmos` is in `SENSITIVE_PREFIXES` in `packages/auth-core/src/route-guard.ts`) — request-time gate, requires `WORKER_SHARED_SECRET` Bearer. 2. `process.env.NODE_ENV === "production"` check in the route's server `GET` handler (mirrors `apps/web/src/routes/dev/login.ts:104`). 3. `import.meta.env.DEV` check at component render time (tree-shakes the body out of production bundles). The route is in iframe mode — `bunx cosmos` must be running alongside `vite dev`. Excluded from `apps/web/scripts/sync-contract.ts` (no public OpenAPI leak). ## Adding a primitive 1. `packages/ui/src/ui/.tsx` — CVA convention per `button.tsx`. 2. `package.json#exports` — add `"./ui/": "./src/ui/.tsx"`. 3. `apps/web/src/cosmos/.fixture.tsx` — Multi-Fixtures, matrix derived from CVA config. 4. `cd apps/web && bunx cosmos` — exercise every variant + size + dark mode. 5. Atomic commit per root `AGENTS.md` rule (one logical change). --- ## [Data model](https://fitxers.usebrioso.com/internal/data-model/) (Internal) # Data model > Source of truth for tables, R2 buckets, secrets, and env. `architecture.md` > only mentions table names inline; this doc carries columns, indexes, and > storage key conventions. If `packages/db/src/schema.ts` changes, update > this doc in the same PR. ## Storage surfaces ### Database (libSQL) * Local SQLite file at `/opt/brioso/shared/data/local.db`, opened via `@libsql/client` against `DATABASE_URL=file:/opt/brioso/shared/data/local.db`. * Single node, single file. Drizzle Gateway is a read-only viewer (second WAL reader). **No Turso today** — the earlier `infisical-env-guide.md` line "prod uses Turso" is inaccurate; correct or remove it. Migrating to Turso managed would still be a single `DATABASE_URL` swap (web keeps the same `libsql://...` URL); not planned. ### Object storage — compute path * **Push-only (BRI-439)** — two runtime kinds: the home GPU daemon (`apps/worker/`, pm2-supervised, `POST /jobs` push) and the Modal app (`apps/modal-worker`, FunctionCall `generate` writes straight to R2 via the payload's presigned URLs). Both upload to R2 (`brioso-jobs` prod bucket) and report completion via the writeback path. See `apps/worker/AGENTS.md` and `.agents/docs/gpu-gateway-canary.md`. ### Object storage — dev stand-in (local-only) * Garage S3 container at `:3900` (`brioso-jobs` bucket, `garage` region), reached via `R2_ENDPOINT`/`R2_FORCE_PATH_STYLE=1`/`R2_REGION=garage`. * Drop-in for R2 in `dev:fake` / `dev:preview`; never used in production. See `.agents/docs/home-worker-local-run.md` for setup. ### Object storage — catalog (separate bucket) * Catalog has its own R2 bucket: `brioso-catalog`, deliberately separate from `brioso-jobs` per `docs/royalty-free-music-decisions.md`'s "R2 bucket split". Uses `packages/storage/src/catalog-config-bound.ts`. ## R2 bucket map | Bucket | Purpose | Key prefixes | Lifecycle | |---|---|---|---| | `brioso-jobs` | jobs, kit artifacts, litestream WAL, cloud-runner uploads, home-worker uploads | `jobs//`, `kits//`, `litestream/` (protected) | Per `.agents/docs/r2-lifecycle-rules.md` | | `brioso-catalog` | catalog masters, previews, embed/verdict payloads | `masters/`, `previews/`, `embeds/`, `verdicts/` | Per `.agents/docs/r2-lifecycle-rules.md` | ## Drizzle schema 39 tables in `packages/db/src/schema.ts`, grouped by domain. Status is `active` unless a vestigial/`DROP TABLE` ponytail marks it deprecated. ### Jobs #### `jobs` (active) * **PK**: `id` (autoincrement integer) * **FKs**: `kitId` → `kits.id` * **Indexes**: `idx_jobs_slug` (unique), `idx_jobs_state`, `idx_jobs_runner_home` (partial: `runner='home'`), `idx_jobs_leased` (partial: `leasedUntil IS NOT NULL`), `idx_jobs_stripe` (unique, partial: `stripeSessionId != ''`), `idx_jobs_message` (partial), `idx_jobs_kit` (partial), `idx_jobs_thread`, `idx_jobs_parent`, `idx_jobs_email` * **Notable columns**: `slug`, `email` (no `user_id` FK; orphan-on-delete), `state` (job lifecycle), `runner` (runtime id: `"home"` built-in or a gateway-config id; insert default via `defaultRunner`), `paramsJson`, `tracks` (JSON blob of `Track[]` — preview + 3 masters per take), `threadId`, `parentJobSlug`, `traceId` / `traceparent` (W3C context; bridges web→worker spans), `costCents`, `artifactsPurgedAt` (BRI-372 tombstone), `createdAt`/`updatedAt` (`nowIso`) #### `jobEvents` (active) * **PK**: `id` (autoincrement integer) * **FKs**: none (audit log keyed by `slug`, not by id) * **Indexes**: `idx_job_events_slug` * **Notable columns**: `slug`, `fromState`, `toState`, `actor`, `fieldsTouched` (JSON list), `reason`, `traceId`, `createdAt` (`nowIso`) * Append-only audit log; one row per applied state transition. #### `jobFeedback` (deprecated) * **PK**: `id` (autoincrement integer) * **FKs**: none * **Indexes**: `idx_job_feedback_slug` * **Notable columns**: `slug`, `trackIdx`, `text`, `createdAt` (`nowIso`) * Vestigial — no longer written; Fillout is the feedback system of record. DROP TABLE pending a coordinated export to Quino. ### Artifacts #### `kits` (active) * **PK**: `id` (autoincrement integer) * **FKs**: none * **Indexes**: `idx_kits_slug` (unique), `idx_kits_state`, `idx_kits_stripe` (unique, partial: `stripeSessionId != ''`) * **Notable columns**: `slug`, `agencyEmail`, `agencyName`, `clientBrandName`, `briefJson` / `analysisJson` / `strategyJson` (3-stage JSON blobs), `state` (`analyzing` → `strategy_ready` → `approved` → `generating` → `ready`), `zipKey` (R2 key for deliverable), `visionCostCents`, `stripeSessionId`, `paid` (boolean), `paidAt`, `createdAt`/`updatedAt` #### `kitCheckoutSessions` (active) * **PK**: `id` (autoincrement integer) * **FKs**: `kitId` → `kits.id` * **Indexes**: `idx_kit_checkout_sessions_session` (unique on `stripeSessionId`), `idx_kit_checkout_sessions_kit` * **Notable columns**: `stripeSessionId`, `createdAt` (`nowIso`) * One row per Stripe Checkout session ever minted for a kit — full reconciliation trail, not just the paying session. ### Workers #### `workers` (active) * **PK**: `id` (text) * **FKs**: none * **Indexes**: none * **Notable columns**: `lastHeartbeat`, `model`, `state` (default `idle`) #### `evalRuns` (active) * **PK**: `id` (autoincrement integer) * **FKs**: `jobId` → `jobs.id` (nullable); `promptId` → `prompts.id` (loose ref) * **Indexes**: none * **Notable columns**: `traceId`, `runpodWorkerId`, `score` (real), `notes`, `createdAt` (`nowIso`) #### `prompts` (active) * **PK**: `id` (autoincrement integer) * **FKs**: none * **Indexes**: none * **Notable columns**: `version`, `body`, `status` (default `draft`), `createdAt` (`nowIso`) ### Ops / alerts #### `alerts` (active) * **PK**: `id` (autoincrement integer) * **FKs**: none * **Indexes**: `idx_alerts_dedupe_key_unresolved` (unique partial: `resolved = 0 AND dedupeKey IS NOT NULL`) * **Notable columns**: `level` (default `info`), `message`, `resolved` (int 0/1), `dedupeKey` (nullable upsert key), `createdAt` (`nowIso`) * Stopped the BRI-57 watchdog from inserting a fresh row every tick. #### `handledMessages` (active) * **PK**: `messageId` (text) * **FKs**: none * **Indexes**: none * **Notable columns**: `createdAt` (`nowIso`) * At-most-once dedup ledger for inbound webhook messages. ### Users / auth #### `users` (active) * **PK**: `id` (text UUID, `$defaultFn(() => crypto.randomUUID())`) * **FKs**: none * **Indexes**: `idx_users_email` (unique) * **Notable columns**: `email`, `name`, `country`, `aiExperience`, `useCase`, `discoveryChannel`, `discoveryChannelOther`, `onboardingCompletedAt`, `createdAt`/`lastSeenAt` (`timestamp` mode, `$defaultFn(() => new Date())`) #### `emailVerifications` (active) * **PK**: `id` (text UUID) * **FKs**: none (keyed on `email`) * **Indexes**: `idx_email_verifications_email` * **Notable columns**: `email`, `otpHash` (bcrypt; plaintext never stored), `expiresAt`, `attempts`, `consumedAt`, `createdAt` (timestamp mode) #### `sentEmails` (active) * **PK**: `id` (autoincrement integer) * **FKs**: none * **Indexes**: `idx_sent_emails_slug_kind_gen` (unique on `slug`,`kind`,`gen`) * **Notable columns**: `slug`, `kind` (`"proposal_ready" | "receipt" | …`), `gen`, `createdAt` (`nowIso`) * At-most-once send ledger; `sendOnce()` does INSERT ON CONFLICT DO NOTHING. #### `shareLinks` (active) * **PK**: `id` (autoincrement integer) * **FKs**: none * **Indexes**: `idx_share_links_token` (unique) * **Notable columns**: `token`, `jobSlug`, `scope` (`"kit" | "track"`), `trackIdx`, `viewCount`, `createdAt` (`nowIso`) * Scoped, non-revocable share links. No `expires_at` by design. #### `userPreferences` (active) * **PK**: `userId` (text) * **FKs**: `userId` → `users.id` * **Indexes**: none (PK is userId) * **Notable columns**: `preferencesJson` (JSON `string[]` of inspectable prefs), `summary`, `source` (`"summary" | "manual"`), `createdAt`/`updatedAt` * Tier-2 memory — explicit, user-visible preference rows. ### Billing #### `subscriptions` (active) * **PK**: `userId` (text) * **FKs**: `userId` → `users.id` * **Indexes**: `idx_subscriptions_user` (unique), `idx_subscriptions_stripe_subscription` (unique on `stripeSubscriptionId`) * **Notable columns**: `stripeCustomerId`, `stripeSubscriptionId`, `plan` (`"monthly" | "annual"`), `status` (mirrors Stripe lifecycle string), `priceCentsLocked` (early-adopter price lock), `currentPeriodEnd`, `cancelAtPeriodEnd` (boolean), `createdAt`/`updatedAt` * One row per Pro subscription, upserted on (re)subscribe. #### `proCreditUsage` (active) * **PK**: composite (`userId`, `month`) * **FKs**: `userId` → `users.id` * **Indexes**: `idx_pro_credit_usage_user_month` (unique) * **Notable columns**: `creditsUsed`, `updatedAt` (`nowIso`) * Month-bucketed counter (`month` = `"YYYY-MM"`). #### `creatorRefinementUsage` (active) * **PK**: composite (`userId`, `periodKey`) * **FKs**: `userId` → `users.id` * **Indexes**: `idx_creator_refinement_usage_user_period` (unique) * **Notable columns**: `count`, `updatedAt` (`nowIso`) * OpenRouter refinement allowance; period-keyed (typically month). #### `processedStripeEvents` (active) * **PK**: `eventId` (text) * **FKs**: none * **Indexes**: none * **Notable columns**: `createdAt` (`nowIso`) * Webhook idempotency ledger for `customer.subscription.*` lifecycle events. #### `promoCodes` (active) * **PK**: `id` (autoincrement integer) * **FKs**: none * **Indexes**: `idx_promo_codes_code` (unique), `idx_promo_codes_active` * **Notable columns**: `code`, `kind` (`"pilot" | "topup" | "trial"`), `generations`, `durationDays` (trial only), `active` (boolean), `maxRedemptions` (null = unlimited), `redemptionCount` (atomic counter), `notes`, `createdAt`/`updatedAt` #### `grants` (active) * **PK**: `id` (autoincrement integer) * **FKs**: `userId` → `users.id` * **Indexes**: `idx_grants_user`, `idx_grants_user_expires`, `idx_grants_user_code_unique` (unique on `userId`,`code`) * **Notable columns**: `code` (redeemed promo or `"trial:cohort-a"` synthetic; not FK), `generations`, `planOverride` (`"pro" | null`), `expiresAt`, `createdAt` (`nowIso`) * One row per redeemed/direct grant; entitlement resolves by stacking. #### `freeGenerations` (active) * **PK**: composite (`keyType`, `keyValue`, `day`) * **FKs**: none * **Indexes**: `idx_free_generations_key_day` (unique) * **Notable columns**: `count`, `updatedAt` (`nowIso`) * Daily free-preview counter keyed on normalized email OR client IP (`keyType` discriminates; both caps enforced independently). ### Catalog #### `catalogTracks` (active) * **PK**: `id` (text UUID) * **FKs**: none * **Indexes**: `idx_catalog_tracks_slug` (unique), `idx_catalog_tracks_genre`, `idx_catalog_tracks_mood`, `idx_catalog_tracks_published`, `idx_catalog_tracks_free_tier`, `idx_catalog_tracks_seed_axis`, `idx_catalog_tracks_classification_status`, `idx_catalog_tracks_watermark_code` (unique) * **Notable columns**: `slug`, `title`, `genreSlug`/`subgenreSlug`/`moodSlug` (BRI-126 taxonomy axes, slugs not display names), `instrumentSlugsJson` (JSON `string[]`), `bpm`, `key`, `durationSeconds`, `peaksJson` (BRI-193 128 uint8 waveform peaks), `previewKey`, `masterFlacKey`, `masterMp3320Key`, `masterMp3128Key` (4-tier keys mirror `jobs.tracks`), `featured` / `published` / `freeTier` (booleans), `classificationStatus` (`"pending" | "ok" | "failed"`), `classifiedArtifactsVersion`, `classifiedMovement`, `classifiedCharacter`, `watermarkCode` (BRI-129 AudioSeal), `generationSeed` / `generationHardware` / `generationPrompt` (reproducibility), `seedAxis` (`"genre" | "subgenre" | "mood" | "instrument"`), `createdAt`/`updatedAt` * Public royalty-free catalog track; distinct from `jobs.tracks`. #### `catalogPurchases` (active) * **PK**: `id` (autoincrement integer) * **FKs**: `trackId` → `catalogTracks.id`; `userId` → `users.id` (nullable) * **Indexes**: `idx_catalog_purchases_stripe` (unique on `stripeSessionId`), `idx_catalog_purchases_track`, `idx_catalog_purchases_user` * **Notable columns**: `email`, `amountCents`, `purchasedAt` (`nowIso`) * One row per PWYW sale (BRI-132). Catalog is non-exclusive (BRI-89). #### `catalogDownloadUsage` (active) * **PK**: `id` (autoincrement integer) * **FKs**: `userId` → `users.id`; `trackId` → `catalogTracks.id` * **Indexes**: `idx_catalog_download_usage_unique` (unique on `userId`,`day`,`trackId`,`format`), `idx_catalog_download_usage_user_day` * **Notable columns**: `day`, `format`, `createdAt` (`nowIso`) * BRI-284: one consumed catalog-master download per user/day/track/format. #### `catalogFavorites` (active) * **PK**: composite (`userId`, `trackId`) * **FKs**: `userId` → `users.id`; `trackId` → `catalogTracks.id` * **Indexes**: `idx_catalog_favorites_user_track` (unique), `idx_catalog_favorites_track` * **Notable columns**: `createdAt` (`nowIso`) * Verified-user × track likes (BRI-138); like-twice is a no-op. #### `catalogQcFailures` (active) * **PK**: `id` (autoincrement integer) * **FKs**: none (standalone log; failed generations never become `catalogTracks`) * **Indexes**: `idx_catalog_qc_failures_created` * **Notable columns**: `leafAxis`, `leafSlug`, `title`, `reason`, `detail`, `seed`, `createdAt` (`nowIso`) * BRI-145 QC review queue log; one row per failed seed attempt. #### `catalogCopyrightClaims` (active) * **PK**: `id` (autoincrement integer) * **FKs**: none (`trackUrl` is free-text, not FK) * **Indexes**: `idx_catalog_copyright_claims_created`, `idx_catalog_copyright_claims_status` * **Notable columns**: `kind` (`"takedown" | "counter_notice"`), `claimantName`, `claimantEmail`, `trackUrl`, `description`, `signature`, `status` (default `"open"`), `createdAt` (`nowIso`) * BRI-147 DMCA intake — single table for both directions, discriminated by `kind`. #### `catalogTrackEmbeddings` (active) * **PK**: `trackId` (text) * **FKs**: `trackId` → `catalogTracks.id` * **Indexes**: none (PK is trackId) * **Notable columns**: `artifactsVersion`, `embedding` (raw float32 blob, 512-d CLAP, 2048 bytes/row), `createdAt` (`nowIso`) * BRI-142; upserted on trackId so only the latest embedding survives. #### `catalogTrackClassificationVerdicts` (active) * **PK**: `id` (autoincrement integer) * **FKs**: `trackId` → `catalogTracks.id` * **Indexes**: `idx_catalog_track_classification_verdicts_track_axis_slug` (unique on `trackId`,`taxonomyAxis`,`promptedSlug`), `idx_catalog_track_classification_verdicts_verdict` * **Notable columns**: `taxonomyAxis` (`"genre" | "mood" | "instrument"`), `promptedSlug`, `verdict` (`"match" | "mismatch" | "unexpressible"`), `probability` (null iff unexpressible), `matchedLabel`, `viaTop1Fallback` (boolean), `artifactsVersion`, `createdAt` (`nowIso`) * BRI-142 classifier QC; one row per (track, axis, prompted slug). Never gates publish. #### `catalogTrackVideos` (active) * **PK**: `id` (autoincrement integer) * **FKs**: `trackId` → `catalogTracks.id` * **Indexes**: `idx_catalog_track_videos_youtube_video_id` (unique), `idx_catalog_track_videos_track`, `idx_catalog_track_videos_track_current` (unique partial: `current = 1`) * **Notable columns**: `youtubeVideoId`, `title`, `description`, `current` (boolean; at most one current row per track — history trail via `current = false`), `publishedAt` (`nowIso`) * BRI-75; one row per uploaded YouTube video for a catalog track. #### `catalogPromptReuses` (active) * **PK**: `id` (autoincrement integer) * **FKs**: `userId` → `users.id`; `trackId` → `catalogTracks.id` * **Indexes**: `idx_catalog_prompt_reuses_user`, `idx_catalog_prompt_reuses_track` * **Notable columns**: `promptSnapshot`, `createdAt` (`nowIso`) * Tier-1 relational memory: "user reused catalog track X's prompt" — one row per reuse event, history not latest-state. ### Inference / kill switch / spend #### `inferenceRoutes` (active) * **PK**: `provider` (text) * **FKs**: none * **Indexes**: none (PK is provider) * **Notable columns**: `weight` (0–100, normalized at read time), `enabled` (boolean; forced 0 when false), `fallbackAfter` (optional chain hint), `updatedBy` (audit), `updatedAt` (`nowIso`) * BRI-359 GPU gateway v2 routing; BRI-439 retired weight routing — the gateway document's `fallbackChain` is the dispatch order and `weight` / `fallbackAfter` are vestigial columns kept for backfill compatibility. #### `inferenceProviderKeys` (active) * **PK**: composite (`provider`, `position`) * **FKs**: none * **Indexes**: `idx_inference_provider_keys_position` (unique) * **Notable columns**: `keyEnv` (env-var name, never a key value), `secretEnv` (Modal paired secret, else null), `label` (admin hint), `updatedBy`, `updatedAt` (`nowIso`) * BRI-359 key ring; resolver looks up env by name. Empty table = fixed env-name fallbacks, a no-op of pre-gateway behavior. #### `workerIdleStreak` (vestigial — watchdog removed in BRI-439) * **PK**: composite (`endpointId`, `streakKind`) * **FKs**: none * **Indexes**: `idx_worker_idle_streak_endpoint` * **Notable columns**: `streakKind` (`"idle" | "queued_without_workers"`), `count`, `updatedAt` (`nowIso`) * BRI-338 Phase 2: durable watchdog streak state (replaces in-memory Maps). #### `killSwitchState` (vestigial — kill-switch removed in BRI-439) * **PK**: `id` (always `1` — singleton state row) * **FKs**: none * **Indexes**: none * **Notable columns**: `lastFiredAtMs`, `updatedAt` (`nowIso`) * BRI-338 Phase 2: durable kill-switch re-fire cooldown. #### `runpodSpend` (vestigial — RunPod runtime removed in BRI-439) * **PK**: `date` (text, `"YYYY-MM-DD"`) * **FKs**: none * **Indexes**: none * **Notable columns**: `cents`, `updatedAt` (`nowIso`) * Daily RunPod spend counter (UTC date). Cents to avoid float drift. #### `runpodSpendCache` (vestigial — RunPod billing removed in BRI-439) * **PK**: `id` (always `1` — singleton state row) * **FKs**: none * **Indexes**: none * **Notable columns**: `cents`, `fetchedAtMs`, `updatedAt` (`nowIso`) * BRI-338 Phase 2: durable 60s cache of RunPod real-billing API read. ### Spaces / canvas #### `spacesCanvasDocuments` (active) * **PK**: `userId` (text) * **FKs**: `userId` → `users.id` (cascade delete) * **Indexes**: `idx_spaces_canvas_documents_updated` * **Notable columns**: `documentJson` (JSON layout/annotations), `revision`, `updatedAt` (`nowIso`) * One owner-scoped Spaces document. Layout + annotation state only; never preview credentials. ## Cross-cutting fields * `createdAt`/`updatedAt`: written via `nowIso()` from `packages/db/src/constants.ts` — `strftime('%Y-%m-%dT%H:%M:%SZ', 'now')` evaluated by SQLite. Note: `users`, `emailVerifications` use Drizzle `timestamp` mode with `$defaultFn(() => new Date())` instead; the `nowIso()` pattern is the rest of the schema. * `paid`: boolean; catalog tracks, purchases, downloads gate on this. * Email-keyed tables: `jobs`, `kits`, `emailVerifications`, `freeGenerations` keyed on the email string (no `user_id` FK; v1 retains rows on account deletion as orphans — see `architecture.md` §Account deletion). * Opaque branded IDs: `JobId`, `KitId`, `UserId`, etc. — see `packages/db/src/brands.ts` (zod-branded string types, no runtime cost). Where the underlying column is an autoincrement integer (e.g. `jobs.id`), the brand wraps the integer; text UUID columns (`users.id`) use `crypto.randomUUID()` as the default. * JSON-blob columns: `peaks`/`peaksJson` (audio fingerprint), `traceparent` (OpenTelemetry context), `paramsJson` (job input JSON), `instrumentSlugsJson`/`tracks` (catalog tag set / job tracks array), `verdict`-typed blobs (classification JSON), `documentJson` (Spaces canvas). ## Litestream * WAL streamed from local SQLite → R2 at `brioso-jobs/litestream/`. * Prefix is **protected** from lifecycle rules (see `.agents/docs/r2-lifecycle-rules.md` §"Litestream prefix"). ## Secrets / env * **Infisical**: project-wide env per `.agents/docs/infisical-env-guide.md`. Per-environment (dev/preview/prod) scopes. * **nono profile**: per-secret disk file at `/etc/brioso/` for systemd- managed processes that can't reach Infisical at boot. * **Drizzle Gateway**: read-only viewer; reads the same SQLite file as a second WAL reader. ## Migration policy * Schema-as-TS in `packages/db/src/schema.ts`. * `drizzle-kit generate` produces SQL under `packages/db/src/migrations/`. * Migrations are embedded and applied at boot via `migrate()` from `drizzle-orm/libsql/migrator`. * Each migration is **one ticket**: one PR, one schema change, one reversible migration. --- ## [Deploy circuit breaker + fail-first](https://fitxers.usebrioso.com/internal/deploy-policy/) (Internal) # Deploy circuit breaker + fail-first Two consecutive deploy failures to the same env for the same change means **stop retrying and stop hand-patching hosts** — write `.agents/plans/YYYY-MM-DD-deploy-simplify-.md` and remove or fix the faulty layer before a third attempt. Hand patches must land as repo changes in the same session or be reverted. Deploy problems must surface pre-merge (`--dry`, PR check, `mise run deploy:rehearse`), never first after merge. **Pull dispatch is forbidden** for every runtime — push-only (`submit(job) -> call_id` + `status(call_id)`, `packages/jobs/AGENTS.md`). Rationale: `.agents/notes/2026-10-07-deploy-feedback-loop.md`. --- ## [deploy:rehearse — local systemd-container cutover rehearsal](https://fitxers.usebrioso.com/internal/deploy-rehearse/) (Internal) # deploy:rehearse — local systemd-container cutover rehearsal `mise run deploy:rehearse` runs the **real pyinfra deploy path** — the same `deploy/pyinfra/v2/operations.py` that cutovers `beta` on the VPS — against a local systemd-enabled Docker container. BRI-428 / audit Phase B items 6–8: the deploy path failing at seams nobody could exercise locally. ## Run it ```sh mise run deploy:rehearse # full run; leaves the container up mise run deploy:rehearse -- --fresh # rebuild + recreate from scratch mise run deploy:rehearse -- --teardown # remove the container after a green run mise run deploy:rehearse -- --down # just remove the container mise run deploy:rehearse -- --report .agents/reports/x.md # write the stage table to a file ``` Requires `docker`, `mise` (bun + node installs get bind-mounted in), `uv` (pyinfra venv), `zstd`. Everything else is local: no Infisical, no VPS, no network side-effects beyond the base-image pull and `bun install` inside the pack stage. ## What the stages prove | stage | what it exercises | |--------------|-------------------| | container | image `ubuntu:24.04` **pinned by digest** (same recipe as the `pyinfra-docker` CI job: install systemd, boot PID 1) + the rehearsal extras (`deploy/rehearse/Dockerfile`: caddy, jq, `brioso`/`web-services` users, r2 stub) | | runtimes | host mise's bun/node are available inside the container (also the `activate` binary inside the pack) | | pack | `scripts/pack-artifact.ts` — the ONE packing implementation; build-artifact.yml calls it in `--pack-only` mode. Builds the same `brioso-web-.tar.zst` contract it ships (`.output`, migrations, seed-data, `activate`, dbtools bundles + libsql, `env-keys.json`) + a `.sha256` sidecar | | seed | container gets the on-VPS preconditions: env-file placeholder, brioso-caddy + upstream seed, r2 stub | | converge-1 | `PYINFRA_TARGET=deploy` full converge: `web` desired state + env-render **write** of `deploy/rehearse/beta.env.fixture` | | converge-2 | converge again → every op `No changes` (idempotency) | | dry-empty | `PYINFRA_TARGET=drift --dry --json` reports zero drifted ops | | deploy-1 | full cutover: flock, fetch `file://` artifact, sha256 verify, extract, contract pin, squatter pre-flight, **dbtools backup + drizzle migrate + seed-ingest**, `activate` (atomic slot flip + warm gate + caddy upstream rewrite + reload), `last-result.json` readback | | deploy-2 | same-SHA redeploy — the real code flips to the other slot and re-warms (no-op-or-recutover semantics asserted) | | health-gate | `/ready` + `/health` return 200 **sha-pinned** through caddy (`Host: beta.usebrioso.com`) and on the active slot port | | smoke | `smoke.py` origin checks + the slot unit and brioso-caddy are `active` | ## deploy:status / deploy:logs One code path (`deploy/rehearse/diag.py`), backend selected by `--target`: ```sh mise run deploy:status # beta over SSH (default, tailnet) mise run deploy:status -- --target prod mise run deploy:status -- --target local # the rehearsal container mise run deploy:logs -- --target local --unit brioso-beta-web-b -n 50 ``` `status` prints the operator view: slot units, `last-result.json` fields, `/health` + `/ready` via the edge, failed units, disk. `logs` tails journalctl. ## What it does NOT prove - **Env render ≠ Infisical.** Rehearsal uses `PYINFRA_ENV_MODE=write` with the checked-in fixture; beta/prod CI renders `.env` via Infisical + `check` mode (names-only diff). Secrets-sync is BRI-427. - **CF edge / DNS / TLS.** Caddy serves plain HTTP on 127.0.0.1:80; the real Cloudflare path isn't exercised. - **SSH transport.** `web_local` connects via `@docker`; SSH-specific failures (tailscaled, auth, host keys) stay CI-side. - **GPU/Modal/RunPod** — out of scope entirely. - **Pack parity is by construction**: BRI-449 #5 collapsed both packers into `scripts/pack-artifact.ts`; the workflow calls it in `--pack-only` mode (CI stages the env-dependent inputs) and rehearsal runs it end-to-end. ## Debugging a failed stage The container is kept running on failure (and after green runs without `--teardown`): ```sh docker exec -it brioso-deploy-rehearse bash journalctl -u brioso-beta-web-b.service -n 50 # inside the container cat /opt/brioso/beta/deploy/last-result.json # what activate reported mise run deploy:status -- --target local # the operator view ``` Known host quirk: kernels with `fs.protected_regular=2` (e.g. this VM) refuse `O_TRUNC` on a sticky-dir file owned by another uid — even for root. The driver `rm -f`s stale `/tmp/brioso-web-*.tar.zst` before each deploy for this reason. --- ## [Web redesign — concept (2026-07-21)](https://fitxers.usebrioso.com/internal/design/) (Internal) # Web redesign — concept (2026-07-21) Concept direction for `apps/web/`. Pairs with `prd.md` (already updated m0011). Final implementation will be drafted into Stitch after sign-off. ## What this is A redesign of `apps/web/` so the product looks like a **tool** a branding agency refines, not a service they hire. Same backend, same flow, new visual + information design. Minimal surface area, maximal restraint. ## What this is NOT - Not a brand-marketing refresh. The wordmark changes; the design system changes; the IA changes. Copy on `prd.md` already updated. - Not bespoke illustration. No custom motion library. No 3D. No Lottie. - Not a 6-step wizard. Intake collapses to 3 steps with optional context tucked behind a single "Tell us more" affordance. ## Brand foundation - **Voice**: tool-first. The agency operates it; we don't consult. - **Moat**: white-label, copyright-free, platform-agnostic generated audio. Agency ships it on client site / product / app / ads / social / retail / phone / decks. Suno / ElevenLabs / Epidemic Sound can't. - **Output metaphor**: "your kit." Not "your strategy." The strategy is the artifact; the kit is the deliverable. - **Mechanism**: a tool the agency refines. UI should feel like a studio tool — knobs, previews, regenerate — not a form. ## Design tokens ### Color Monochrome ground, maroon pulse, no warm earth palette. | Token | Value | Use | |---|---|---| | `--ink` | `#0e0a08` | Type, primary | | `--ink-soft` | `#3a3530` | Body text, secondary | | `--paper` | `#faf7f2` | Page background (warm off-white, near-white) | | `--paper-2` | `#f0ebe1` | Subtle elevation (cards, hover) | | `--rule` | `#e3dccf` | Hairlines, dividers | | `--maroon` | `#5a1a1f` | Pulse highlight, primary CTA, focus | | `--maroon-ink` | `#3a0f12` | Maroon text on paper | The maroon is the *only* chroma in the system. Everything else is monochrome. Background is near-white paper, not the cream that read as artisan bakery. Type is near-black, not warm brown. ### Type | Role | Family | Weights | |---|---|---| | Display (wordmark, H1, hero) | **Fraunces** | Black (900), Bold (700), Regular (400) | | Body / UI / microcopy | **Satoshi** | Regular, Medium, Bold | - Wordmark: "brioso." — lowercase, Fraunces Black (pinned at wght=900, opsz=144 in the variable source), trailing period. - Type scale (Satoshi): 12 / 14 / 16 / 18 / 20 / 24 / 32 / 44. - Display (Fraunces): only 44 / 64 / 96 for hero + section openers. - Line height: 1.4 body, 1.05 display. - Letter-spacing: -0.02em display, -0.01em H2, 0 body, +0.04em micro. - Never set body copy in Fraunces; it is a wordmark/display face. Fraunces is OFL (Undercase Type). Source .ttf files live under `apps/web/public/fonts/source/` (gitignored); the bundled woff2 outputs in `apps/web/public/fonts/` are generated by `scripts/build-fraunces-woff2.py` and re-run via `bun run build:assets`. Satoshi is OFL/FLF and ships with the full family in `Downloads/Satoshi_Complete.zip`. ### Spacing 8px base. Scale: 4 / 8 / 12 / 16 / 24 / 32 / 48 / 64 / 96 / 128. ### Radius Default 12px. Cards 16px. Pills 999px. Inputs 10px. ### Motion | Element | Motion | Duration | Easing | |---|---|---|---| | Pulse (radial burst) | breathe + slow rotate | 6s breathe, 60s rotate | sine in-out | | Page enter | fade + 8px up | 400ms | cubic-bezier(.2,.8,.2,1) | | Hover (buttons, links) | opacity / underline | 150ms | ease-out | | Modal / dialog | fade + scale 0.98→1 | 200ms | ease-out | The pulse is the only "ambient" motion. Everything else is reactive. ## Information architecture Single product, two entry paths surfaced equally in the top nav. ``` Home / Landing — both audiences, with a tab toggle Strategy / Free Audio Strategy (logo + URL → PDF) Studio / The kit generator ($99 paid intake → kit delivery) For Musicians / B2C variant of Studio (same engine, simpler copy) Pricing / One card: $99 per kit. Free strategy is the lead magnet. Docs / (later) — public-facing guides for white-label usage. ``` Top nav (5 items): Studio · For Musicians · Pricing · Docs · Sign in. **No** "Strategy" in the nav — it's a marketing surface, linked from hero, not a destination in IA. ## Page-by-page ### Landing (`/`) - Hero: wordmark left-aligned, Fraunces 96px (text-hero), single line. Below it, one sentence: the moat — "White-label audio for every place your brand lives." (Iterate.) Right of hero: the **pulse**. Breathing radial burst, ~360px, centered, hollow core. Maroon highlights active spokes. No headline adjective soup. No carousel. - Below hero: a single quiet section. Three columns of one line each. - "Generate" (logo + URL) - "Refine" (3-step intake, regen) - "Ship" (white-label ZIP + guidelines, zero Brioso branding) - Mid: comparison strip. Suno / ElevenLabs / Epidemic Sound / Brioso. Columns: where it can ship. Brioso column wins by being the only one with all rows green. No editorializing, the table makes the case. - Bottom CTA: "Make your kit." → `/studio`. - Footer: minimal. Brand wordmark, two columns (Product / Company), social row. No newsletter. ### Intake (`/studio`) Three steps, one screen tall each. Right rail: live preview of the emerging kit (logo uploaded → first waveform renders as soon as the strategy step completes). 1. **Brand** — logo upload, URL paste, brand name. Required. 2. **Vibe** — one question: "Pick three words your brand should *sound* like." Chip-grid. Twenty-eight words total, four columns on desktop, three on mobile. Multi-select to three. The chosen words drive archetype + emotional-goals inference server-side. 3. **Context** *(optional)* — single collapsible panel. "Tell us more." Inside: long-form prompt, brand voice notes, references. Default closed; agent feels short by default. Submit → strategy doc lands in `~/library` (B2C) or `~/clients/:slug` (B2B) → payment → generation → kit delivery. ### Strategy doc (`/strategy/:slug`) The free deliverable. Public read-only URL. - Hero: brand name (Fraunces 64, text-display), one-line summary (Satoshi 18). - Body: Audio Strategy document. Five sections: Brand in sound / Personality / Archetype / Emotional goals / Sonic palette. - Each section renders with a small inline waveform glyph next to the header — 24px, monochrome. Subtle. - Bottom CTA: "Generate your kit — $99" → `/studio?prefill=:slug`. ### Kit delivery (`/studio/:id/kit`) The product. The artifact. - Top: brand wordmark left, "kit." right (lowercase, Fraunces, period). - Audio blocks (3–5). Each block has: - Title (Fraunces 32): e.g., "Hero loop" / "Loading state" / "Hold music" / "Notification chime" / "Son outro". - Live waveform (wavesurfer.js, maroon playhead). - Duration, BPM, key, mood chips. - Action row: Play · Regenerate · Download (`.flac` + `.mp3`) · Mark for client. - Right rail: **package export**. White-label ZIP preview. - Files: `logo/` folder (SVG + PNG), `audio/` (flac + mp3), `guidelines.pdf`. - Branding toggle: "Strip Brioso from export" — on by default. - Download button: maroon, primary. - Below: usage table. Where this kit can ship (web / product / app / social / retail / phone / decks). Each row: green check. This is the moat made visible at the deliverable. ### B2C variant (`/for-musicians`) Same `/studio` engine, different surface copy. Top of the page replaces the agency framing with personal-music framing. Same 3-step intake minus the brand-logo step (replaced with track title). Kit delivery renders the same. **No** white-label toggle — it's their own music. ## Components Build small. Reuse shadcn primitives in `apps/web/components/ui/`. Custom-built (in `apps/web/components/brioso/`): - `Pulse` — the radial burst. SVG, ~80 lines. Props: `size`, `phase`, `highlightSpokes`. Animates via CSS custom properties + `requestAnimationFrame`. ~30 lines of motion logic. - `Wordmark` — Fraunces, "brioso.", trailing period. Single SVG (auto-generated paths from the variable source), no JS. - `KitCard` — one audio block in the kit. wavesurfer.js inside, action row beneath. - `ChipGrid` — multi-select to N, used in intake vibe step. - `MaroonButton` — primary CTA, single component, used everywhere a CTA appears. No "variants." One visual treatment, monochrome hover. Reused as-is (no rebuild): - shadcn `Button`, `Input`, `Dialog`, `Tabs`, `Tooltip`, `Sheet`. - `react-hook-form` + zod for intake validation (already in app). - `wavesurfer.js` for playback. - Stripe Checkout (already wired). ## Motion details — the pulse - 36 spokes around a hollow core. - Per-spoke base length, base phase offset, base amplitude. - Animation: amplitude breathes from 0.6 → 1.0 → 0.6 over 6s (sine). - Phase offsets per spoke give the breathing a "heartbeat" — not a metronome. - Rotation: 360° over 60s, linear, infinite. - Maroon highlight: a small subset of spokes (e.g., 6) ride the peak of the breathe — when amplitude is highest, they fire to maroon. The rest stay ink. - Implementation: SVG, `` per spoke, `transform="rotate(...)"`, `stroke-dasharray` modulated by amplitude via CSS variables. JS only sets `transform: rotate(...)` and the amplitude CSS var per frame. - Respect `prefers-reduced-motion`: skip rotation, slower breathe, no maroon cycling. ## Out of scope (this pass) - Account dashboard polish (deferred — kit delivery is the primary surface). - Agency multi-client management UI (future — for v2). - Public docs site (linked from kit delivery, built later). - Animations beyond the pulse + standard hover/enter. - A "blog" or any content marketing surface. - A/B testing infrastructure. ## Open after sign-off - Display-face migration is resolved — Fraunces (OFL) ships six static 72pt cuts + the variable source under `apps/web/public/fonts/source/`, rebuild via `bun run build:assets`. - Final hero copy + comparison table copy. - Implementation plan broken into Stitch-shaped chunks. - GitHub Project issue: file under project briosoco/projects/1 with the Epic: "Web redesign v2." --- ## [Dev login guide (\`/dev/login\`)](https://fitxers.usebrioso.com/internal/dev-login-guide/) (Internal) # Dev login guide (`/dev/login`) To dogfood or test a workflow against a running web server without walking the email/OTP flow each time, hit `/dev/login` — a single dev-only route that upserts the user, applies any persona state, mints a real session cookie, and redirects, all in one request. No seeding step, no JSON artifact, no manual cookie injection. It mints via the same `createSessionToken()`/`resolveSecret()` path `handleVerifyOtp` uses after a real OTP verify, so there is no bypass path or test-only auth surface — it just skips the email round-trip, and only for `@brioso.test` addresses. ``` GET /dev/login?persona=agency&redirect=/studio GET /dev/login?persona=pro-new&redirect=/onboarding GET /dev/login?email=whatever@brioso.test&redirect=/studio ``` `persona` is one of `agency`, `creator`, `free-new`, `pro-new`, `free-returning`, `pro-returning` — see `PERSONAS` in `src/routes/dev/login.ts` for the exact email/state each maps to. The four `*-new`/`*-returning` personas apply the plan (free/Pro) x completion (first-time/returning) 2x2 that `/onboarding` QA needs, including the Pro-just-subscribed welcome screen (reachable by appending `?session_id=anything` to the URL against `pro-returning`/`free-returning` — no persona needed, `session_id` just flips which component renders). `email` still works for any other `@brioso.test` address; `redirect` defaults to `/studio` and must be a single-leading-slash relative path. Works identically from a browser, `curl`, or a Playwright fixture — just navigate/request the URL, no separate script invocation or cookie plumbing: ```ts await page.goto("http://localhost:3000/dev/login?persona=agency&redirect=/studio/agencies"); ``` ```bash curl -i "http://localhost:3000/dev/login?persona=creator" # Set-Cookie in the 302 ``` Dev-only by construction: request-time 404 under `NODE_ENV=production`, plus `import.meta.env.DEV` so the handler is stripped from the production bundle. To reset every seeded `*@brioso.test` row (e.g. between scenario runs), use `scripts/dev-reset-users.ts` (`mise run db:reset-users`). For a real end-to-end login smoke test, still walk `/login` and read the OTP from the server log (dev captures it when `EMAIL_PROVIDER` is unset); do not rely solely on `/dev/login`. For driving the app interactively against a live local stack, use the `qa-studio` skill (checking one specific fix/state) or the `dogfood` skill (an open-ended sweep across every journey) — both under `.agents/skills/`. --- ## [Dev-seed fixture states](https://fitxers.usebrioso.com/internal/dev-seed-states/) (Internal) # Dev-seed fixture states The studio dev-seed scenarios seed deterministic rows under fixed slugs so a QA pass (Playwright layout suite, dogfood walks, ad-hoc browser checks) can land on a known state without burning a real generation. This page is the single source of truth for which scenario exercises which state, which `@brioso.test` account owns the rows, and how to reseed cleanly. ## Persona: who to log in as The canonical studio QA persona is **`creator`** (`/dev/login?persona=creator&redirect=/studio`). Every `studio:*` fixture is seeded under `creator@brioso.test`. `persona=agency` is kept as a thin alias that mints the same session (older runbooks and curl examples still say "agency"). Use `creator` in any new docs. Other personas (`free-new`, `pro-new`, `free-returning`, `pro-returning`) own the onboarding state matrix — they only have `subscriptions` / `users.onboardingCompletedAt` rows, no jobs. `billing:pro` / `billing:free` cover subscription-only states. ## States Two state machines live on top of the same `jobs` row: ### `JobState` (DB column) Authoritative list: `packages/contract/src/jobStates.ts`. All seven values are valid at the DB level (the column is plain `text`, no CHECK constraint). Studio fixtures only seed a subset; runtime code can move a job through the rest. | State | Meaning | | --- | --- | | `unverified` | OTP requested, not yet verified — never visible in the studio | | `new` | Verified, queued for a runner | | `composing` | Claimed by a runner, generation in progress | | `paid` | Complete, tracks delivered — "success" | | `failed` | Generation failed (terminal) | | `paused` | User-stopped mid-generation (resumable) | | `cancelled` | User-deleted (soft-delete; excluded from `/jobs/mine`) | ### `JobLifecycle` (UI render bucket) Authoritative list: `packages/verticals/studio/src/components/canvas/lineage.ts` `toLifecycleState()`. Three values — what the canvas node actually renders: | Lifecycle | Renders as | Triggers | | --- | --- | --- | | `complete` | TrackList + waveform + Remix/Use prompt actions | `state === "paid"` | | `generating` | GeneratingProposals placeholder | anything not in the `stopped` set | | `stopped` | Inline alert (`"This generation stopped before the tracks were ready…"`) | `failed` / `paused` / `cancelled` | `composing` and `new` both render as `generating`; the difference is visible only to the polling loop (`isTerminalJob` in `hooks/useJobPolling.ts`), which keeps polling `composing` jobs and stops on terminal states. ## Scenarios All under `creator@brioso.test` unless noted. Slugs are stable so a Playwright spec can deep-link to them. Reseed with `mise run db:seed -- `. | Scenario | Slug | JobState | Lifecycle | Notes | | --- | --- | --- | --- | --- | | `studio:empty` | (none) | — | — | Account has zero jobs. New-brief landing screen. | | `studio:thread-short` | `dev-feedback-mock-0001` | `paid` | `complete` | Single resolved job, 1 track. The `/studio` feedback icon row. | | `studio:thread-long` | `dev-seed-thread-long` (plus 21 turns) | `paid` x 22 | `complete` x 22 | Forces feed overflow + the fade mask. | | `studio:thread-longword` | `dev-seed-thread-longword` + `-t1` | `paid` x 2 | `complete` x 2 | Thread of 2 takes; the second has a 200-char unbroken token for the mobile width-overflow bug. | | `studio:thread-composing` | `dev-seed-thread-composing` | `composing` | `generating` | Single-turn composing state — the loading state. | | `studio:thread-failed` | `dev-seed-thread-failed` | `failed` | `stopped` | Single-turn failed state — the `reduceConversationMessages` failed branch. | | `studio:thread-paused` | `dev-seed-thread-paused` | `paused` | `stopped` | Worker-stopped mid-generation; renders the same stopped node as `failed` but a distinct DB state. | | `studio:thread-composing-mid` | `dev-seed-thread-composing-mid` + `-t1` | `paid` then `composing` | `complete` + `generating` | Mid-thread generation: paid take plus an in-progress follow-up. Exercises the live-polling node next to a completed one. | | `studio:thread-failed-mid` | `dev-seed-thread-failed-mid` + `-t1` | `paid` then `failed` | `complete` + `stopped` | Failure-after-progress: paid take plus a failed follow-up. Exercises the stopped node next to a completed one. | | `studio:at-limit` | `dev-seed-at-limit-{0,1,2}` under `onboarding-free-returning@brioso.test` | `paid` x 3 | `complete` x 3 | Free-plan quota exhausted (`FREE_THREAD_CAP=3`) — renders `AtLimitCta` instead of the composer. | | `sharing:link` | `dev-seed-sharing-link` | `paid` | `complete` | Resolved job plus a share-link token (`/s/dev-seed-share-token`). | | `billing:pro` / `billing:free` | (no jobs) | — | — | `subscriptions` rows only. | ## Reseed ```sh # One-shot, idempotent (each scenario upserts by slug): mise run db:seed -- studio:thread-long studio:thread-failed \ studio:thread-composing-mid studio:thread-failed-mid # Reset every dev-seed-* row (jobs and share links; user accounts are # separate — use `mise run db:reset-users` if those need clearing too): mise run db:seed -- --reset # List all scenarios: mise run db:seed -- --list ``` The `--reset` flag deletes every `jobs.slug LIKE 'dev-seed-%'` row and every `share_links.jobSlug LIKE 'dev-seed-%'` row; it intentionally does not touch `dev-feedback-mock-0001` (no `dev-seed-` prefix) or the `dev-seed-at-limit-*` set under `onboarding-free-returning@`. ## Why this doc exists The previous dev-seed comment header listed scenarios but said nothing about the state each one exercises. The 2026-08-23 session saw three generations show "This generation stopped before the tracks were ready…" under `agency@brioso.test` — those were stale rows under a persona that owned no fixtures, not a UI regression. Adding the state mapping here and pointing all skills/docs at `creator` makes the next person stop chasing a phantom bug. --- ## [Diag Access Runbook](https://fitxers.usebrioso.com/internal/diag-access-runbook/) (Internal) # Diag Access Runbook Locked-down `diag` Linux user on the VPS for teammates' AI agents to run diagnostics over Tailscale SSH: read journald, check unit health, and restart exactly the two web slots. Replaces the need for any per-agent service or token: `ssh diag@ "journalctl ..."` is the whole interface. > **Scope for the diag user:** read all journald units (via the > `systemd-journal` group), `systemctl status` / `is-active` on > `brioso-web*`, and `sudo systemctl restart` on `brioso-web-a` and > `brioso-web-b` only. No root, no shell as root, no file reads outside > world-readable paths, no writes anywhere. ## What diag can do | Capability | How | Boundary | |---|---|---| | Read journald, all units | member of `systemd-journal` group | read only; covers web slots, Caddy, Litestream, deploy failures | | Read the deploy result | `cat /opt/brioso/{,beta/}deploy/last-result.json` | file is root-owned 0644 inside a `root:diag` 0750 dir (lockstep rule below); no sudo | | Check web slot health | `systemctl status brioso-web*`, `is-active brioso-web*` | brioso-web units only | | Restart a web slot | `sudo -n systemctl restart brioso-web-a` (or `brioso-web-b`) | exact commands, NOPASSWD, nothing else | | Disk / general read-only diagnostics | `df -h`, `systemctl --failed`, etc. | unprivileged commands only | Everything else is denied by normal file permissions (`/home/*` is `700`) and by sudoers: `sudo -n systemctl restart caddy`, `sudo -n /bin/bash`, any other sudo invocation fails. ## Provisioning (pyinfra) The user, group membership, permissions, and sudoers are provisioned by the v2 `diag_user` pyinfra subsystem (`deploy/pyinfra/v2/state/diag_user.py`, all envs). The sudoers drop-in `/etc/sudoers.d/diag` is rendered from `deploy/sudoers/diag` (kept in exact parity by `v2/tests/test_v2_sudoers_parity.py`): ``` diag ALL=(ALL) NOPASSWD: /usr/bin/systemctl status brioso-web* diag ALL=(ALL) NOPASSWD: /usr/bin/systemctl is-active brioso-web* diag ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart brioso-web-a.service diag ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart brioso-web-b.service ``` Validate after deploy with `sudo visudo -c`. (Both files exist on the branch and are being implemented in parallel; this runbook covers the access model either way.) ## last-result.json readability (lockstep rule) The deploy result file `/last-result.json` is rewritten by the root-run `activate` binary on every deploy, so its permissions must be part of the desired state, not a one-off chmod (which regresses at the next deploy). The lockstep rule across three files: - `deploy/activate/src/commands.ts` (`writeResult`): writes the file mode `0644`. - `deploy/pyinfra/v2/state/diag_user.py`: keeps both deploy dirs - This runbook: documents the capability (table above). Group path traversal (`diag` in the dir's group, dir g+x) makes the root-owned file readable without sudo — plus a traverse-only (`--x`) ACL for diag on `/opt/brioso` itself, which is `web-services:brioso 0750` and would otherwise block diag from entering the tree at all (same module pins it; `--x` grants no listing, so `shared/` env files stay unreachable). If any of these moves, move all. ## Blue-green restart caveat Restarts are **not** free: which slot is active matters. - Restarting the **inactive** slot is always safe: Caddy proxies to the other slot, and deploys use this exact mechanism. - Restarting the **active** slot briefly drops traffic (seconds, user-visible) until the unit comes back. Check which slot is active before restarting: read `/opt/brioso/deploy/state.env` on the VPS (`ACTIVE_SLOT` / the active slot's port is recorded there). When in doubt, don't restart; report instead. ## Common agent one-liners ```bash journalctl -u brioso-web-a -n 200 --no-pager # recent logs for a slot systemctl --failed # any failed units df -h # disk systemctl status brioso-web-a # unit health sudo -n systemctl restart brioso-web-a # restart (see caveat above) ``` ## Manual Tailscale steps (one-time, out-of-repo) These are done in the Tailscale admin console, not by pyinfra. **Export the live policy first** and edit on top of it — never paste `.agents/docs/tailscale-acls.jsonl` over live (it would drop CI's SSH rule and break every deploy). 1. **Network grant on `:22`**: Tailscale permits an SSH connection only when the policy has **both** a network grant and an SSH rule. Add to `grants`: ```jsonc { "src": ["autogroup:member"], "dst": ["tag:server"], "ip": ["tcp:22"] } ``` Members otherwise only have `tcp:443` to `tag:server`, so without this they are blocked at the network layer before the `ssh` section is consulted — which is why the VPS looks "online and reachable" (ICMP/443 work) while SSH does not. 2. **SSH ACL rule**: add to the policy's `ssh` section: ```jsonc { "action": "accept", "src": ["autogroup:member"], "dst": ["tag:server"], "users": ["diag"] } ``` `accept`, not `check`: teammates point non-interactive AI agents at this and `check` stalls the first connect on a browser re-auth URL. Attribution still comes from tailnet identity in the connection logs, and the account is fenced by sudoers + file permissions. The existing baseline `check` rule grants members SSH to `autogroup:self` (their own devices) as any nonroot user; it does **not** cover `tag:server`, so the two rules don't conflict. 3. **Session recording for diag (follow-up, not yet done)**: `record` is *not* a valid `ssh` action — the only actions are `accept` and `check`. Recording is a `recorder` field (plus optional `enforceRecorder`) on an `accept`/`check` rule and needs a **tsrecorder node** provisioned and tagged in the tailnet. Tracked as BRI-376; do not add a `record` rule. 4. **Verify the final policy** with the ACL editor's rules preview: - members are allowed to `tag:server` as `diag` and as **no other user**; - `tag:ci` -> `deploy` (the CI deploy rule) is still allowed. After saving, overwrite `.agents/docs/tailscale-acls.jsonl` with the **live** post-change export (not a hand-edit) and keep doing that after every console change — the file is a read-only mirror, not the source of truth. ## Audit - **Tailscale admin console**: connection logs tie every SSH session to the tailnet identity that opened it, before mapping to the shared `diag` Linux user. - **journald sudo logs**: `journalctl -t sudo` on the VPS shows every sudo command diag ran (command line, tty, exit status). - **Session recording is NOT active** (no tsrecorder node yet — see Manual Tailscale steps #3). Until it is, sudo journald logs plus Tailscale connection logs are the only attribution for the shared `diag` account. - Shared-user audit story: multiple teammates share `diag`, but attribution comes from Tailscale identity at connect time; sudo logs add command-level confirmation. ## Revocation (per teammate) There are no per-user credentials on the VPS to rotate. To revoke a teammate: remove them from the tailnet. If finer grain is wanted later, switch the ACL rule's `src` from `autogroup:member` to a dedicated `group:diag` and manage membership in the admin console. ## Verification After provisioning and the manual Tailscale steps: 1. **Positive** (as a teammate, from their tailnet machine, non-interactive): `ssh -o BatchMode=yes diag@ 'journalctl -u brioso-web-a -n 5 --no-pager'` succeeds; `ssh -o BatchMode=yes diag@ 'sudo -n systemctl status brioso-web-a'` succeeds with no browser prompt and no password prompt. 2. **Negative**: `ssh root@` and `ssh deploy@` as a member are denied; `sudo -n systemctl restart caddy` as diag fails with "not allowed"; `cat /home//.bashrc` gets permission denied; `sudo -n /bin/bash` is refused. (Verified 2026-08-26 on prod via `sudo -u diag` — all pass.) 3. **CI not broken** (the real regression risk): re-run `deploy.yml` (or push a trivial commit to `beta`) in the same session as the policy change and confirm the `Deploy over Tailscale SSH` step still connects as `deploy@`. 4. **Recording**: not active — no tsrecorder node yet (Manual Tailscale steps #3). BRI-376 tracks provisioning it. 5. ~~**LogMCP no-regression**~~ — LogMCP was removed repo-side on 2026-08-26 (WS4 of the deploy v2 restructure); the VPS-side decommission steps are in [`.agents/docs/windmill-logmcp-decommission-runbook.md`](windmill-logmcp-decommission-runbook.md). Once executed, `systemctl status brioso-logmcp` should report the unit as not found/inactive. 6. `mise run validate` green for the repo change (pyinfra + docs). ## LogMCP removal LogMCP was decommissioned 2026-08-26 (repo-side removal in the WS4 branch; host-side steps pending in [`windmill-logmcp-decommission-runbook.md`](windmill-logmcp-decommission-runbook.md)). It was loopback-only in practice — the `logs.usebrioso.com` Caddy block was never provisioned. **Diag SSH is now the sole logs/journald surface** and the restart surface lives in diag's sudoers (the LogMCP clitool plugin was never provisioned with a write token). ## Teammate handoff **No SSH key is needed.** With Tailscale SSH enabled, tailscaled terminates the connection itself and authenticates the tailnet identity over WireGuard node keys (SSH auth type `none`) — there is no `authorized_keys` file for `diag`, and the pyinfra module deliberately never creates one (`deploy/pyinfra/v2/state/diag_user.py`). Any key your local `ssh` offers is ignored. Nothing to generate, nothing to share. Prerequisites: 1. Be a member of the tailnet (`autogroup:member`) with Tailscale running and logged in on the machine you connect from. 2. `ssh diag@`. No tokens to store, no MCP client config, no per-user credentials. Access follows tailnet membership. The `ssh` rule uses `accept`, so there is no device-approval or browser prompt on first connect. Two gotchas that look like auth failures but are not: - **`REMOTE HOST IDENTIFICATION HAS CHANGED`** — Tailscale SSH presents its own host key, distinct from the VPS's sshd host key. If you ever SSHed to that host directly, drop the stale entry: `ssh-keygen -R `. - **`Permission denied (publickey)`** — the connection reached the real sshd, not Tailscale SSH, i.e. the policy change has not taken effect for you yet. It is not a missing-key problem; do not "fix" it by asking for a key. --- ## [Docs site runbook (docmd → Cloudflare Pages)](https://fitxers.usebrioso.com/internal/docs-site-runbook/) (Internal) # Docs site runbook (docmd → Cloudflare Pages) The internal docs site publishes repo-root `docs/` + `.agents/docs/` as a static docmd site at **https://fitxers.usebrioso.com**, gated behind Cloudflare Zero Trust Access. This file covers how it works and the manual setup that the repo cannot do for you. ## How it works - Workspace: `apps/docs` (`@brioso/docs`). - `pnpm --dir apps/docs build` (wired as `mise run docs:build`): 1. `src/stage-content.ts` cleans `apps/docs/site-docs/` and copies repo-root `docs/` (markdown/assets only) to the tree root and `.agents/docs/**` to `site-docs/internal/` (everything except `.DS_Store` — the seo CSVs and papers PDFs ship too). 2. `docmd build` (`docmd.config.json`) renders the static site to `apps/docs/dist`, including `llms.txt`, `llms-full.txt`, `llms.json`. 3. `src/copy-functions.ts` copies `apps/docs/pages-functions/` into `dist/functions/` so the deploy dir carries the Pages Function. - Deploy: `.github/workflows/deploy-docs.yml` — push to `beta` touching `.agents/docs/**`, `docs/**`, `apps/docs/**`, or the workflow itself. Cheap ubuntu-latest job: checkout → deploy-runtime → `turbo run build --filter=@brioso/docs` → wrangler `pages deploy apps/docs/dist --project-name=docs-brioso`. Secrets: `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` from the `beta` GitHub Environment (asserted by the secrets-check job in deploy-pr-checks.yml, names only). ## pages.dev bypass docmd output is pure static, so the block ships as a Cloudflare Pages Function (`apps/docs/pages-functions/_middleware.ts`): any request whose hostname ends in `.pages.dev` gets a bare `404 Not found`; everything else passes through. Because Access sits in front of the custom domain only, this middleware is what keeps `.pages.dev` from exposing the docs unauthenticated. **Don't remove it** without replacing the guard. ## URLs - Site: https://fitxers.usebrioso.com - llms index: https://fitxers.usebrioso.com/llms.txt - Full corpus (agents): https://fitxers.usebrioso.com/llms-full.txt - Machine-readable index: https://fitxers.usebrioso.com/llms.json Agents fetch the corpus through Access with a service token. The token lives in **Infisical** (`prod` environment, `/infra` folder, keys `CF_ACCESS_CLIENT_ID` / `CF_ACCESS_CLIENT_SECRET`) — same source of truth as the rest of the infra secrets (`infisical-env-guide.md`). Any machine with an `infisical login` session pulls it the standard way: ```bash # fetch the corpus with the token exported from Infisical infisical run --env=prod --path=/infra -- \ curl -sS -o /tmp/llms-full.txt \ -H "CF-Access-Client-Id: $CF_ACCESS_CLIENT_ID" \ -H "CF-Access-Client-Secret: $CF_ACCESS_CLIENT_SECRET" \ https://fitxers.usebrioso.com/llms-full.txt ``` GitHub Actions uses the copies in the `beta` environment secrets (same key names); those are write-only and not an agent retrieval path. ## Setup state (as of 2026-10-17) All steps below are **done**; this section records the concrete resources so future maintenance doesn't have to rediscover them: - Pages project: `docs-brioso` (account `usebrioso`, id `ea4dcf9f4d3179bfead02c492d8f59e3`, production branch `beta`). - Custom domain: `fitxers.usebrioso.com` attached (zone `usebrioso.com`), DNS CNAME `fitxers → docs-brioso.pages.dev` (proxied — Access only runs on proxied traffic). - Access app: `Brioso Docs (fitxers.usebrioso.com)`, id `e67ab30c-cb84-4cdd-88c5-bdd6a53bdd22`, session 12h. Two policies: - `team emails` (allow): carloshpiedra@gmail.com, josocjoq+dev@pm.me, quino@usebrioso.com, carlos@usebrioso.com. - `agent service token` (**bypass**): include `{"service_token": {"token_id": ""}}`. The API rejects `{"service_token": {}}` and `{"service_token": {"id": ...}}` — it must be `token_id`. Bypass (not allow) is what lets the token through without an interactive login. Free tier caps at **50 users**. - Agent service token: `docs-agents` (client id `67eb0874715f4e2cba520cb1b3640369.access`). Values in Infisical `prod` `/infra` and GitHub `beta` env secrets. Rotated twice on 2026-10-17 (learn: don't clean up the only copy of a secret before it's in the store). - GitHub `beta` environment secrets: `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`, `CF_ACCESS_CLIENT_ID`, `CF_ACCESS_CLIENT_SECRET`. ### Recreate-from-scratch checklist (if ever rebuilding) 1. **Cloudflare Pages project**: `wrangler pages project create docs-brioso --production-branch=beta`. The API token needs `Cloudflare Pages: Edit`. 2. **Custom domain**: attach `fitxers.usebrioso.com` to the project; create the DNS CNAME (proxied) — the API token needs `DNS: Edit`. 3. **Access app**: Zero Trust → Access → Applications → self-hosted app for `fitxers.usebrioso.com`, allow policy = the team emails. 4. **Access service token for agents**: Access → Service credentials → Service Tokens → create. Authorize via an app policy with decision **bypass** and include `{"service_token": {"token_id": ""}}`. Store in Infisical `prod` `/infra` (`CF_ACCESS_CLIENT_ID` / `CF_ACCESS_CLIENT_SECRET`) and as GitHub `beta` env secrets. 5. **GitHub Environment secrets** (`beta`): `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID`. Until both exist, the deploy-docs workflow fails and the secrets-check job on PRs is red. ## Adding / rotating - **New docs page**: drop the `.md` anywhere in `docs/` (end-user) or `.agents/docs/` (internal, lands under `/internal/`). Push to `beta`; the paths-filtered workflow deploys it. No navigation config needed — docmd titles from the H1 and builds the tree automatically. - **New human user**: add them to the Access app policy (50-user cap). - **Rotate the agent service token**: create a new token, repoint the bypass policy's `token_id`, update Infisical + GitHub `beta` secrets, then delete the old token (deletion fails while it's still referenced by the policy — update the app first). - **Rotate the deploy API token**: rotate in Cloudflare (My Profile → API Tokens), update the `CLOUDFLARE_API_TOKEN` secret in the `beta` GitHub Environment + Infisical. --- ## [Dogfood Runbook — apps/web](https://fitxers.usebrioso.com/internal/dogfood-runbook/) (Internal) # Dogfood Runbook — apps/web Superseded by the `dogfood` skill (`.agents/skills/dogfood/SKILL.md`, symlinked into `.claude/skills/dogfood/`) — invoke it directly rather than following steps here. This file is kept only as a redirect so old links don't 404. For checking one specific fix or state instead of doing a full exploratory sweep, use the `qa-studio` skill (`.agents/skills/qa-studio/SKILL.md`) instead. Source of truth for screen-sync policy: `apps/web/AGENTS.md` ("Keep `.agents/screens/` in sync"). --- ## [Evals Fundamentals](https://fitxers.usebrioso.com/internal/evals-fundamentals/) (Internal) # Evals Fundamentals Team reference for LLM and audio-generation evaluation. Distills the research that informed the [BRI evals upgrade plan](../../.agents/notes/) so the next session does not re-derive it. Concrete, not theoretical: every section ends in "what we do at Brioso." ## 1. Eval anatomy Every eval has four parts. Anthropic frames them in their [develop-tests guide](https://docs.anthropic.com/en/docs/build-with-claude/develop-tests) and the [Anthropic eval cookbook](https://github.com/anthropics/eval-cookbook): 1. **Input**: the prompt + context the system receives. 2. **Output**: what the system produced. 3. **Reference / golden**: the expected output, or a set of properties the output must satisfy. 4. **Score**: a function from (output, reference) to a number or label. **Success criteria come before evals.** You cannot grade what you have not specified. Before writing a test, write the success criteria: what does "this refined prompt is good" actually mean? Behaviors, not vibes. Examples: - "Prompt contains no vocal terms" (regex-checkable, easy). - "Refined prompt mentions the same three constraints as the user input" (judge-checkable, needs anchors). - "Generated audio scores >= 3.0 on Audiobox content-enjoyment on the FMA-Pop prompt subset" (metric-checkable, needs a numeric floor). If you cannot write the criteria as a checklist, you cannot grade them. The Anthropic eval cookbook ([evaluate-with-human-review](https://github.com/anthropics/eval-cookbook/blob/main/misc/buildkite/eval_recipe/README.md)) recommends writing criteria before touching the model. **At Brioso**: every new `.eval.ts` file in `packages/verticals/*/src/ai-workflow/` opens with a comment block naming the four parts and the success criteria. Without that block, the PR is incomplete. ## 2. Grading hierarchy Use the cheapest reliable grader. The order, from best to worst when "best" means "cheapest AND most reliable": 1. **Code / regex** (always first). Schema validity, length bounds, forbidden-term absence, JSON shape, cost ceiling. 2. **Human** (for ground truth and calibration). Slow, expensive, used sparingly. 3. **LLM-as-judge** (for what code cannot see). Cheaper than human, but has failure modes that code does not. Code graders fail loud and deterministic. Human graders are slow and expensive but set the ceiling. LLM judges sit between them and inherit the LLM's quirks, so we constrain them tightly. ### LLM-as-judge rules Anthropic's [claude-as-judge guide](https://docs.anthropic.com/en/docs/build-with-claude/claude-as-judge) and OpenAI's [evals guidance](https://platform.openai.com/docs/guides/evals) converge on the same recipe: - **Rubric with behavioral anchors.** Not "is this good?" but "does output contain X? does it avoid Y? is Z preserved?" Each anchor is a binary or 1-3 score, not a Likert "rate from 1 to 5." - **CoT-then-verdict structured output.** Two sections in the response: `reasoning` (free-form, separate field) and `verdict` (the score). Mixing them lets a confident-but-wrong rationale contaminate the grade. Use JSON schema or tool-use to enforce the split. - **Pairwise over absolute.** "Is A better than B?" is much easier for a judge than "rate A from 1 to 5." Anchor on the original-vs-refined pair, not a free-floating score. - **Cross-family model.** The judge must be a different model family than the model being graded. If the refine workflow uses Claude, the judge is not Claude. Same-family judges share the same blind spots. - **Temperature 0.** Deterministic verdicts. Sampling temperature above 0 in a judge adds noise without adding signal. - **TPR / TNR calibration before gating.** Before a judge becomes a CI gate, hand-label 50 cases (positive + negative). Report true-positive rate, true-negative rate, and inter-rater agreement (Cohen's kappa against the human labels) in the PR that introduces the judge. A judge with TPR < 0.8 is not a gate; it is a triage signal. **At Brioso**: judge rules are encoded in a shared judge template under `packages/ai-workflow/src/evals/judge-template.ts`. Every judge imports from it, including rubric structure, temperature setting, and the schema for `reasoning` + `verdict`. Cross-family rule means a judge is never the same model as the task. Calibration numbers live in the PR description, not in a wiki. ## 3. Nondeterminism LLM outputs are stochastic. A single shot tells you nothing. - **N-trial pass-rate.** Run each case N times (default 5, per the plan). The case passes if `passRate >= 0.8`. Single-shot assertions hide variance and produce flaky CI. - **Paired deltas, not absolute scores.** When comparing model A vs model B on the same case set, compute per-case pass-rate deltas (A - B) and report the mean + standard error across cases, not the raw pass-rate of each model in isolation. The pairing kills case-difficulty confounders. - **MDE before trusting small deltas.** Minimum detectable effect at alpha=0.05, power=0.8, depends on case count and per-case variance. Compute it from the first 30 cases, then add it to every eval report. A 2-point pass-rate delta on a 30-case suite with high variance is noise, not signal. The Anthropic eval cookbook ([statistical-analysis](https://github.com/anthropics/eval-cookbook/blob/main/misc/evaluation/stat_analysis_guide.ipynb)) walks through the calculation. **At Brioso**: the runner reports `(meanDelta, stdError, mde)` per comparison. Sub-MDE deltas are reported as "no signal detected," not as "no improvement." A red CI on a sub-MDE delta is a flake, not a regression. ## 4. Dataset hygiene The dataset is the most important artifact in any eval system. It is also the easiest to quietly corrupt. - **Provenance tags.** Every case carries a `provenance` field: `{source: "prod" | "synthetic", prodDate?, ticketRef?, authorId}`. Production cases get a ticket ref so a regression has a story. Synthetic cases get an author so we can audit who invented which adversarial input. - **Production-failure promotion loop.** When a user-reported failure hits the refine/strategy workflow, the on-call runs semantic dedup against the current dataset (CLAP embedding on the prompt, cosine threshold 0.92) and adds the case with `ticketRef: "BRI-NNN"`. This is the BRI-148-style incident-cited regression rule applied to evals. - **Anti-overfit (fresh-author rounds, quarterly).** Every quarter, a different team member authors 5-10 fresh cases from real usage without seeing the current dataset. If their cases have a pass-rate-statistically-different from the legacy set, the legacy set was overfit to its authors. - **Provenance-weighted scoring.** Weight cases by recency: a case from the last 30 days of production gets weight 2, older cases weight 1. This keeps the eval honest about drift in real usage. - **Version pinning in CI.** The dataset version is a hash in the runner config. CI fails if the hash in the runner does not match the hash in the versioned JSONL file. A runner silently loading a mutated dataset is the worst failure mode in this whole stack. **At Brioso**: datasets live as versioned JSONL in `packages/verticals//src/evals/datasets/.vN.jsonl`. The runner reads the file path, hashes it, and refuses to start if the hash drifts mid-run. ## 5. Audio-generation metrics Audio eval is harder than text eval because there is no free "did it contain X" check. We triangulate four signals: ### FAD (Frechet Audio Distance) Use [Microsoft fadtk](https://github.com/microsoft/fadtk) (Python, published with Gui et al. ICASSP 2024, [arXiv:2311.01616](https://arxiv.org/abs/2311.01616)). FAD measures distributional distance between a candidate's audio embeddings and a reference set's embeddings in a frozen embedding space (VGGish, CLAP, or MERT). Lower is better. Best practices from fadtk's own README: - Pick a reference set that matches the task. For generative music, fadtk recommends [FMA-Pop](https://github.com/mdeff/fma) (4839 tracks, the subset Gui et al. propose). Do not default to MusicCaps for open-ended generation. - Report the embedding model, reference set, sample count, and duration alongside every FAD number. Without all four, the number is not interpretable. - Run per-song FAD on outliers (`fadtk --indiv`) to listen to what "low" and "high" actually mean in your domain. **At Brioso**: FAD runs inside the `packages/classification` venv (Python, isolated). The runner calls `fadtk` as a subprocess, parses `scores.csv`, and persists the result to the Phoenix experiment. ### CLAP-to-original-intent (weak proxy) CLAP cosine similarity between the generated audio and the *original user prompt* (not the refined prompt) measures whether the generator preserved user intent. It is a **weak proxy**, with measured correlation to human preference: - The original CLAP score has Pearson r ≈ 0.28-0.30 with human relevance judgments ([Takano et al., Human-CLAP, 2025](https://arxiv.org/abs/2506.23553): "CLAPScore has a low correlation with subjective evaluation scores"). - For text-audio *alignment* specifically, music-trained LAION-CLAP checkpoints (LAION-MA, LAION-MS, LAION-MSA) reach the highest correlations with human ratings in the [Benchmarking Music Generation Models survey (arXiv:2506.19085)](https://arxiv.org/abs/2506.19085). This means CLAP is fine as a smoke signal (does the model ignore the prompt entirely?) but not as a quality gate. A 5-point swing in CLAP score does not predict a 5-point swing in human preference. ### Audiobox Aesthetics (MOS proxy) [Meta Audiobox Aesthetics](https://arxiv.org/abs/2502.05139) (Tjandra et al., 2025) is a no-reference per-item predictor trained on four orthogonal axes: - **PQ** (Production Quality): clarity, fidelity, absence of artifacts. - **PC** (Production Complexity): sophistication of the production. - **CE** (Content Enjoyment): subjective appeal. - **CU** (Content Usefulness): fitness for downstream use. Per the paper, the AES predictors show comparable or superior correlation with human MOS while being more robust across audio types than DNSMOS / NISQA / UTMOS. We already have `facebook/audiobox-aesthetics` on HuggingFace and the `audiobox-aesthetics` PyPI package, so this drops into `packages/classification` next to fadtk. Use it as the scalable MOS proxy: every audio-gen experiment run records the four AES scores per case. Aesthetics is per-item, fast, and reference-free, so it scales to nightly full-tier runs. ### Periodic human pairwise Before shipping any model swap, run a small human pairwise study: >= 5 ratings per (case, pair) cell, counterbalanced (every rater hears A and B in both orders). Blind to model identity. Record: - Per-case winner. - Bradley-Terry score per model across the case set. - Win rate with 95% Wilson interval. This is the ground-truth anchor for FAD + CLAP + AES triangulation. If the three automated metrics agree with human pairwise, ship it. If they disagree, debug the metrics, not the model. **At Brioso**: the audio testbed pins prompt set, seeds, and SA3 variant. Candidate-vs-incumbent reports always show all four metrics (FAD, CLAP-to-intent, AES-CE, AES-PQ) plus the human pairwise result when one was run. A model change without at least one human pairwise study goes to the bottom of the queue. ## 6. Phoenix architecture decision Synthesis of the "Phoenix vs vitest" section of the upgrade plan. We keep both, with a hard split. ### What Phoenix owns (system of record) [Phoenix](https://arize.com/docs/phoenix) is the system of record for eval data. It holds: - **Versioned datasets** ([datasets concepts](https://arize.com/docs/phoenix/datasets-and-experiments/concepts-datasets), [how-to](https://arize.com/docs/phoenix/datasets-and-experiments/how-to-datasets)). Every example carries its provenance tags. - **Experiments** ([how-to-experiments](https://arize.com/docs/phoenix/datasets-and-experiments/how-to-experiments)): one experiment = one run of a task + evaluators over a pinned dataset version. Outputs, scores, traces all persisted together. - **UI side-by-side**: prompt variants, model candidates, and strategy-change experiments compare on the same dataset version with paired deltas visible per case. - **Annotations** ([annotating in the UI](https://arize.com/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/annotating-in-the-ui), [cookbook](https://arize.com/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/cookbook-human-in-the-loop-workflows-annotations)). Human-in-the-loop review of judge outputs and trace spans. - **Online evals on spans** ([annotations-and-evaluations tutorial](https://arize.com/docs/phoenix/tracing/tutorial/annotations-and-evaluations)): sample 2-5% of live production traces, run the same rubric, promote failures to the dataset. The TS client reference ([phoenix-client annotations](https://arize.com/docs/phoenix/sdk-api-reference/typescript/packages/phoenix-client/annotations)) lives in the obs facade only. No app code imports `@arizeai/phoenix-client` directly, per the `scripts/check-obs-imports.ts` rule. ### What vitest/pytest owns (thin CI gate) The test file's job becomes: 1. Load the pinned Phoenix dataset version. 2. Run the task under test. 3. Log the experiment (task output + evaluator scores + traces). 4. Assert locally: `passRate >= 0.8` and no regression vs the last green baseline. Phoenix's [pytest integration](https://arize.com/docs/phoenix/datasets-and-experiments/eval-ci-with-pytest) (`@pytest.mark.phoenix`, suite -> dataset, case -> example, run -> experiment, exit code = gate) does this natively. The [Arize blog](https://arize.com/blog/evals-in-llm-evals-as-tests-how-to-write-llm-evals-as-tests/) expands the pattern. TS uses the [vitest integration](https://arize.com/docs/phoenix/sdk-api-reference/typescript/packages/phoenix-client/integrations-vitest) at >=6.11.1. Set `PHOENIX_TEST_TRACKING=0` to run the gate fully offline (no Phoenix server round-trip), useful for air-gapped CI and local debugging. So the rule is: **Phoenix = datasets, experiments, scores, judges, traces. vitest/pytest = gating.** The test runner is a thin shell around the Phoenix experiment. ### PostHog boundary PostHog stays product analytics only: funnels, retention, feature flags, session replay. It has no eval role. Eval data does not flow through PostHog, and eval dashboards are not PostHog dashboards. Core AI tooling consolidates on Phoenix. **At Brioso**: `packages/obs-js` and `packages/obs-py` are the only importers of `@arizeai/phoenix-client` and `arize-phoenix-client`. All other code routes through the `obs` facade. ## 7. Onboarding links Canonical references, grouped by topic. The [Phoenix evals skill](.agents/skills/phoenix-evals/) (Arize-published, v1.0.0) is the in-repo manual; it covers setup, evaluators, experiments, and CI integration step by step. ### Phoenix eval docs - [Datasets & experiments quickstart](https://arize.com/docs/phoenix/get-started/get-started-datasets-and-experiments) - [Datasets concepts](https://arize.com/docs/phoenix/datasets-and-experiments/concepts-datasets) - [Datasets how-to](https://arize.com/docs/phoenix/datasets-and-experiments/how-to-datasets) - [Experiments how-to](https://arize.com/docs/phoenix/datasets-and-experiments/how-to-experiments) - [Dataset evaluators](https://arize.com/docs/phoenix/datasets-and-experiments/how-to-dataset-evaluators) - [Eval CI with pytest](https://arize.com/docs/phoenix/datasets-and-experiments/eval-ci-with-pytest) - [Blog: LLMs as tests](https://arize.com/blog/evals-in-llm-evals-as-tests-how-to-write-llm-evals-as-tests/) - [LLM evals overview](https://arize.com/docs/phoenix/evaluation/llm-evals) - [Pre-built metrics](https://arize.com/docs/phoenix/evaluation/pre-built-metrics) - [phoenix-evals PyPI docs](https://arize-phoenix.readthedocs.io/projects/evals/) - [Annotating in the UI](https://arize.com/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/annotating-in-the-ui) - [Cookbook: human-in-the-loop annotations](https://arize.com/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/cookbook-human-in-the-loop-workflows-annotations) - [Online evals / annotations on spans](https://arize.com/docs/phoenix/tracing/tutorial/annotations-and-evaluations) - [TS phoenix-client reference](https://arize.com/docs/phoenix/sdk-api-reference/typescript/packages/phoenix-client/annotations) ### Anthropic eval guidance - [Develop tests guide](https://docs.anthropic.com/en/docs/build-with-claude/develop-tests) - [Claude as judge](https://docs.anthropic.com/en/docs/build-with-claude/claude-as-judge) - [Eval cookbook](https://github.com/anthropics/eval-cookbook) - [Stat analysis notebook](https://github.com/anthropics/eval-cookbook/blob/main/misc/evaluation/stat_analysis_guide.ipynb) ### Audio metrics - [Microsoft fadtk](https://github.com/microsoft/fadtk) (FAD library, ICASSP 2024) - [Audiobox Aesthetics paper](https://arxiv.org/abs/2502.05139) (Tjandra et al., 2025) - [audiobox-aesthetics PyPI](https://pypi.org/project/audiobox-aesthetics/) - [FMA dataset](https://github.com/mdeff/fma) (the FAD reference set) - [Human-CLAP (Takano et al., 2025)](https://arxiv.org/abs/2506.23553) (CLAPScore correlation with humans) - [Benchmarking music generation metrics (arXiv:2506.19085)](https://arxiv.org/abs/2506.19085) (LAION-CLAP checkpoints, music-trained variants) ## 8. Promotion loop When a user reports a failure in a refine/strategy/analyze workflow, the on-call engineer runs the production case through the promotion loop to add it to the golden dataset. This is the incident-cited regression rule applied to evals (BRI-148-style). The goal is to make every real failure a permanent test that cannot silently regress again. ### Step 1: Flag A user or on-call flags a production failure. The case is captured as a structured payload: - `brief` or `captureDoc`: the input that triggered the failure. - `observedFailure`: what the model did wrong (free text, used in `notes`). - `spanLink`: optional Phoenix span URL for the trace. No PII is required in the payload. The promotion tool runs a redaction pass before writing to the dataset. ### Step 2: Export The engineer exports the failure as a JSON payload (file or stdin) and notes the Linear ticket number (e.g. BRI-320). The ticket number is the provenance anchor: it ties the eval case back to the incident so future readers can find the context. ### Step 3: Redact `bun eval-promote.ts` runs a PII redaction pass over the brief before writing. Redacted types: email addresses, phone numbers, bearer tokens/JWTs, long hex blobs (40+ chars), long base64 data URIs (image/audio payloads). A redaction report is printed in the output so the engineer can audit what was changed. ### Step 4: Promote ```sh bun eval-promote.ts --ticket BRI-320 --suite refine < failure.json ``` `eval-promote.ts` does the following: 1. Parses the JSON payload and validates it has a content field. 2. Derives a deterministic `caseId` from the ticket number plus a slug derived from the brief. The format is `{ticketSlug}-{briefSlug}`, e.g. `refine-bri-320-refuses-short-brief`. Determinism means re-running the same promote is idempotent: the second run finds the duplicate and refuses to append. 3. Loads existing `caseId`s from the target JSONL and refuses to overwrite. Append-only, never destructive. 4. Validates the generated row through the matching schema parser (`parseRefineCase`, `strategyCaseSchema`, or `analyzeCaseSchema`). 5. Sets `provenance.kind = "prod"`, `provenance.ticket = "BRI-NNN"`, `provenance.date = YYYY-MM-DD`. 6. Appends the row to the correct JSONL file (`refine.jsonl`, `strategy.jsonl`, or `analyze.jsonl`) via `Bun.write` with `append: true`. ### Step 5: PR The row is appended as a single new line in the JSONL. A PR is required to merge it into the golden dataset. The PR must include the Linear ticket so reviewers can see the incident context. PR review checks: - The `brief`/`captureDoc` is accurate to the production failure. - Tags, budgets, and other fields are correct. - PII redaction was sufficient. - The `notes` field captures the failure mode in plain language. **What we do at Brioso**: the promotion loop is gated behind the on-call runbook. The `eval-promote.ts` script is the single entry point; do not hand-edit JSONL files. caseId naming follows the `refine-bri-NNN-slug` convention so tickets and cases are cross-referenceable. ## 9. Judge calibration cadence Before a judge can gate a merge, two conditions must be met: calibration (accuracy vs. human labels) and cadence (regular refresh to prevent drift). ### Calibration gate A judge is not a CI gate until it has been calibrated against 50 hand-labeled cases (positive and negative). Report in the PR that introduces the judge: - **TPR / TNR**: true-positive rate, true-negative rate against human labels. TPR < 0.8 disqualifies the judge as a gate; it becomes a triage signal. - **Inter-rater agreement**: Cohen's kappa against human labels. Agreement >= 0.6 is acceptable; >= 0.75 is strong. - **Two-rater agreement** (online evals): before judge verdicts gate merges in the online sampling flow, two raters annotate the same sample independently. Agreement must be >= 80-85% before the judge's verdicts are trusted as authoritative. Below that threshold, judge verdicts are advisory only. The rubric stays binary or decomposed checks, never a Likert scale: "does output contain X?" (pass/fail), "does it avoid Y?" (pass/fail). Each check is independently readable and auditable. ### Quarterly refresh The judge rubric, prompt, and any behavioral anchors must be refreshed at least once per quarter. The trigger is calendar, not degradation signals: judges degrade slowly and quietly. A quarterly refresh keeps the rubric anchored to current product behavior rather than the behavior that existed when the judge was written. Refresh steps: 1. A different engineer (not the original judge author) hand-labels 20 cases from the most recent quarter of production traces. 2. Run the judge against those 20 cases. 3. Compare TPR/TNR against the baseline from the original calibration run. Any regression > 5 points is a signal to rewrite anchors. 4. If anchors are rewritten, re-calibrate against a fresh 50-case set before re-enabling the gate. **What we do at Brioso**: judge calibration numbers live in the PR description of the PR that introduced the judge. The quarterly refresh is tracked as a recurring Linear task. Phoenix annotations are the source of truth for both calibration cases and the human-in-the-loop review that drives online eval sampling. --- ## [Feature flags](https://fitxers.usebrioso.com/internal/feature-flags/) (Internal) # Feature flags ## Verdict Flags are adopted, but narrower than a generic "adopt feature flags" answer, because the repo already solves most of what flags usually buy. | Need | Already solved by | Where | | ----------------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------- | | Env differences | Infisical + `cfg.get()` typed schema | `packages/config/src/config.ts:10-30` | | Change a setting without redeploy | `initConfig()` + `onConfigReload` + SIGHUP reload | `packages/config/src/bootstrap.ts:69`, `config.ts:123-152` | | Operational kill switch | RunPod kill switch + spend caps | `packages/dispatch/src/runpodKillSwitch.ts:20-23`, `sweep.ts:98-110` | | Feature on/off | presence-based toggles (feature is on iff its secret exists) | `billing/src/webhook.ts:101-108`, `notify/src/config-bound.ts:24-29` | | Dev-only scaffolding gate | `isFakeFlow()` | `packages/verticals/kits/src/fakeFlow.ts:21-33` | | Use-case divergence | routes + separate `packages/verticals/{kits,creator}` | `routes/_marketing/agencies/`, `routes/studio/creators.tsx` | The "flags let you flip things without a redeploy" argument is largely already true here via Infisical + SIGHUP. `cfg.get()` genuinely cannot do three things, and those are the only justification for adding flags: 1. **Percentage rollout** — ramp a risky backend change 5% → 25% → 100% keyed on a stable bucket. `cfg.get()` is all-or-nothing. 2. **Per-person / per-property targeting** — beta cohort by email, variant by person property. 3. **Experiment/analytics join** — flag exposure lands in the same PostHog project as the conversion events already captured, so a variant can be measured. A config value cannot. 4. **Staged feature exposure** — a built-but-unproven capability opened to a named cohort (design partners, a persona, a percentage) before general release. This is genuinely per-person targeting, already justification #2; naming it separately makes the *feature-launch* case explicit so it isn't read as forbidden by the "No flags for use-case selection" non-goal below. ## Non-goals Writing these down is half the value; unbounded flags are debt. - **No entitlements/plans via flags.** The Pro subscription (`packages/billing` exports `./subscription-checkout`, `./pro-credits`, `./entitlement`, `./pro-price-display` alongside `./promo` and `./webhook`) is a real plan/credit concept now, but it lives entirely in the `subscriptions` DB table and `entitlement.ts`'s `isActiveSubscription`, never behind a flag — the policy this non-goal states held even as the entitlement itself shipped. This is the distinction most likely to get lost once staged exposure is in scope: stems-behind-a-paid-tier (as `pricing-analysis.md` once recommended) is an entitlement, not exposure — a flag controls whether the code path exists for a person, and must never be the billing boundary. - **No flags for use-case selection.** Routes already do it. A flag for *whether a capability is available to this person* is in scope (justification #4 above); a flag for *which product they are using* is not. - **No flags for env or secrets.** `cfg.get()` owns that. - **No OpenFeature abstraction layer, no Flipt.** PostHog is already in the stack (`posthog-node` in `@brioso/obs-js`, `posthog-js` + `@posthog/react` in `apps/web`). Zero new infra, zero new deps, no catalog change. ## Decision rule Reach for `cfg.get()` first. Reach for a flag only when the need is percentage rollout, per-person targeting, or an experiment that must join PostHog analytics. If none of those apply, it's config, not a flag. ## Architecture Flags are evaluated **at the edge** and the resolved value is injected inward — same seam as the existing `onKitPaid` port (`packages/billing/src/webhook.ts:24-28`). A domain package receives a resolved value (e.g. `backend: "sa3" | "musicgen"`), never a flag key. This preserves the observability invariant and the per-package import bans in `.oxlintrc.json` without adding a single exemption. ``` Infisical ─► packages/config (POSTHOG_PERSONAL_API_KEY) │ cfg.get ▼ packages/obs-js/src/posthog.ts ← only place posthog-node is imported PostHog client + personalApiKey + featureFlagsPollingInterval: 300_000 (local eval) │ ▼ packages/obs-js/src/index.ts ← facade adds: flag(), flags(), FlagSubject │ ┌──────────┴───────────┐ ▼ ▼ apps/web route handlers / apps/worker, dispatch sweep server fns (edge) (bucket on job.slug) │ └─► resolved value passed as a plain param into verticals/* and ai-workflow apps/web browser: posthog-js already evaluates flags after init. useFeatureFlagEnabled / useFeatureFlagVariantKey from @posthog/react. Same distinct id as the server ⇒ server rollout and client experiment agree. ``` ## Identity Percentage rollout is only meaningful with a stable bucket key, and server and client must agree or the experiment is noise. - **Logged in**: `user.id`. - **Anonymous browser traffic**: `apps/web/src/server/obs/flagSubject.ts` reads PostHog's own cookie `ph__posthog` server-side and pulls `distinct_id` out of it — the same id posthog-js is already using client-side, so a server-evaluated flag and a client-evaluated flag land in the same bucket. This lives in `apps/web`, not `obs-js`, because it is HTTP/cookie-bound. - **No browser at all** (worker, dispatch sweep): bucket on `job.slug`, already the obs distinct id (`packages/obs-js/src/context.ts`). - On login, call `posthog.identify(user.id)` client-side so pre/post-auth flag values do not flicker. `FlagSubject` (`packages/obs-js/src/context.ts`) is a distinct type from `ObsContext`, deliberately: `ObsContext`'s distinct-id helper falls back to `ctx.component`, which would put every anonymous request in one bucket and silently break every percentage rollout. ## Failure behaviour Flags must never break a request. `flag()` returns the caller's default on any failure: - No `POSTHOG_API_KEY` → `getClient()` returns `null` → default. Local dev never touches the network. - No `POSTHOG_PERSONAL_API_KEY` → client still constructed, local eval off, flags fall back to a remote HTTP call per evaluation. Degraded, not broken. - Network/timeout → `featureFlagsRequestTimeoutMs: 2000` plus try/catch, resolves to the default. Local-evaluation caveats when authoring a flag: stick to person-property and percentage-rollout operators. Experience continuity, static cohorts, and `is_not_set` force a remote round trip per evaluation. ## Lifecycle - Naming: `-`, kebab-case. - Every flag's PostHog description carries its `BRI-*` issue and a target removal date. A flag past its date is a cleanup ticket. - Ceiling of 10 live flags. Raised from 5 because the Workstream 4 audio-tooling survivors are 5 candidate flags on their own, plus `agencies-hero-copy` already live. Flag keys are a typed union in `obs-js` (`packages/obs-js/src/index.ts`), promoted at the same time the ceiling was raised rather than deferred until the old 5-key limit was hit. - The `posthog:cleaning-up-stale-feature-flags` skill runs the sweep. ## Current flags - `agencies-hero-copy` (client, multivariate) — headline/CTA copy variant on `_marketing/agencies/index.tsx`. Copy only; not a price experiment. The kit's live price is set by `KIT_PRICE_CENTS` in `kits/src/kit-checkout.ts` and `SONIC_KIT_PRICE` in `sonicBranding.ts` — a copy flag driving only the headline would advertise a different amount than it charges. A real price test needs a flag payload driving both constants, and is its own ticket. - `agencies-site` (client + server, boolean) — gates the whole agencies site (routes, nav/footer links, homepage CTAs, sitemap/robots exposure) behind membership of the `pilot-agencies` PostHog cohort during a private pilot. BRI-203. Target removal/graduation: 2026-09-15. - `studio-attach-upload` (client, boolean) — hides the composer's "+" attach-reference button (`apps/web/src/components/studio/CreatorPromptInput.tsx`) while the upload flow behind it is still a placeholder dialog with no working upload logic. Default off. BRI-243. Graduates (flag removed, button always shown) when BRI-243 ships; no due date set on that ticket yet, so there's no committed removal date — revisit at the next stale-flag sweep if BRI-243 is still unscheduled. - `audio-stems` (server, boolean) — staged exposure for Demucs v4 stem separation (vocals/drums/bass/other, FLAC-only) on Pro-plan tracks. Pro membership is the entitlement (`packages/jobs/src/completion.ts`); this flag only controls rollout — us-only → 25% of Pro → 100% → remove. Dispatched from the single completion-time trigger in `apps/web/src/server/jobs/handlers.ts` (`dispatchPendingStems`). BRI-157. Target removal: once at 100% of Pro for a full billing cycle with no regressions — revisit at the next stale-flag sweep if still open past 2026-11-01. `dispatch-sa3-rollout` was considered but deliberately **not** shipped: the SA3 torch.compile rollout (`.agents/plans/2026-07-27-sa3-torch-compile-rollout.md`) is a per-endpoint `SA3_TORCH_COMPILE` env var flipped in the RunPod dashboard, not a code-level branch — there is nothing for a flag to gate. Do not add one until an actual code-level backend selection exists to resolve into. `creator-studio-cta` (optional, `/studio/creators`) was also **not** shipped. Its only stated purpose was to exercise the anonymous-identity path end to end on real unauthenticated traffic, which `agencies-hero-copy` already does — a second call site adds no new coverage of the mechanism. Add it if `/studio/creators` itself ever needs a real copy/CTA experiment; don't add it just to have a second flag. ## Proposed — audio tooling staged exposure (BRI-122) Names reserved ahead of implementation; no call site exists yet, so there is nothing to poll for a stale-flag sweep yet. Register the `BRI-*` issue and a target removal date (justification #4, staged exposure) when each capability actually ships. All default **off**. - `audio-analysis` — Librosa/Essentia BPM + key detection - `audio-loudness` — EBU R128 loudness normalization - `audio-seamless-loop` — zero-crossing seamless looping - `audio-inpaint-ui` — Wavesurfer.js mask-based inpainting UI **Blocker to note, not to fix here.** The server-side flag path was built but had zero production call sites — `flag()`, `flags()` (`packages/obs-js/src/index.ts:75-95`) and `flagSubjectFromRequest()` (`apps/web/src/server/obs/flagSubject.ts:31-42`) were referenced only by tests until `agencies-site` (BRI-203) became the first production caller. `posthog.identify(user.id)` — required by the Identity section above so a user's flags don't flicker across login — landed in `apps/web/src/lib/analytics-identify.ts` (BRI-197, `b0ab993`), called from `_authed.tsx`, `login.tsx`, and `components/music/OtpStep.tsx`. Both costs are paid; this note stays only as history for the next of these five flags to ship. --- ## [GPU gateway beta rollout runbook (BRI-420 / plan 3 v2 §5)](https://fitxers.usebrioso.com/internal/gpu-gateway-beta-runbook/) (Internal) # GPU gateway beta rollout runbook (BRI-420 / plan 3 v2 §5) The human-executed checklist that takes the beta GPU gateway from "stack merged" to "Modal runtime registered and importable". Post-BRI-439 the gateway is push-only — no weights or ramp stages; dispatch order is the `fallbackChain`. Day-2 operations live in [gpu-gateway-canary.md](gpu-gateway-canary.md). **Who runs this:** a human holding Infisical + Modal + GitHub credentials. No VPS shell is needed beyond what the deploy pipeline already does; every command below runs from your own shell/browser. Nothing in this doc is executed by CI or an agent session. **Safety:** `set +o history; umask 077` before touching secrets, `set -o history` when done. Secrets go straight into Infisical/GitHub — never into files, issues, or chat. ## CLI verification status (vs pinned `modal==1.3.5`, D11 exact pin) | Spelling | Status | |---|---| | `modal environment create ` | verified — positional `name`, no `--env` flag (creates in the current workspace) | | `modal secret create -e/--env --force K=V…` | verified — `-e`/`--env` and `--force` both exist | | `modal deploy -e/--env ` | verified — already used by `deploy-modal-worker.yml` | | `modal app stop ` | verified (`modal app` group) | | `modal token {set,new,info}` | exists, but only manages *account* tokens — **no proxy-token subcommand**. Proxy-auth `wk-…`/`ws-…` pairs stay a dashboard step | | `infisical secrets get/set … --env=` | spelling unverified here — the Infisical env *slug* for beta (`staging` below, per the plan) must be confirmed; `infisical environments` / the dashboard lists them | Anything not on this list is "verify before use" — check `modal --help` on the pinned version (`uv tool run --from "modal==1.3.5" modal …`) before scripting it. ## Prerequisites - [ ] U1–U4, U6, U7a and U9 are merged to `beta` (BRI-407, runtime ids, BRI-413 dispatch→wake, BRI-411 modal-worker, BRI-412 spend ledger, BRI-417 admin plumbing, BRI-418 local runtime) and deployed. - [ ] Plan-2 gates green: the last `deploy-pyinfra` beta run passed drift-gate → validate → deploy → origin `/health` smoke → `/internal/config-audit`, and `/ready` reports ready. - [ ] GitHub secrets exist: `MODAL_ESPETRO_TOKEN_ID` / `MODAL_ESPETRO_TOKEN_SECRET` (deploy token for the `espetro` account — the existing `MODAL_TOKEN_ID`/`MODAL_TOKEN_SECRET` pair serves account 1 *if* it holds a deploy token; see §5.1 of the plan). - [ ] The home GPU box runs the BRI-437 push daemon (`brioso-worker serve` on :7311 — see [home-worker-wake.md](home-worker-wake.md)) and its tailnet hostname is known: `config/gpu-gateway.beta.yaml` ships `push.url` as `http://REPLACE_WITH_HOME_TAILNET_HOST:7311/jobs`, a placeholder you fill in before importing. `HOME_WAKE_TOKEN` is optional — uncomment `push.tokenEnv` only if a non-tailnet caller must submit; tailnet peers (`tailscale whois`) and loopback are authorized without it. `BRIOSO_WAKE_ALLOW_NODES` narrows which nodes may submit. ## Step 1 — sanity: beta still works today ```bash curl -fsS https://beta.usebrioso.com/health curl -fsS https://beta.usebrioso.com/ready ``` **Expected:** both 200; `/ready`'s `gateway` block shows the effective source and a valid config. **If it fails:** stop — fix beta first; the gateway rollout rides on the deploy pipeline. Triage: hop 1–2 of the §7 table in [gpu-gateway-canary.md](gpu-gateway-canary.md#observability). ## Step 2 — per-Modal-account environment + secrets Repeat for each Modal account runtime (today: `espetro` → `modal-espetro`). Export that account's **deploy** token first so the CLI acts on the right workspace: ```bash export ACCT=espetro RUNTIME=modal-espetro export MODAL_TOKEN_ID=… MODAL_TOKEN_SECRET=… # this account's deploy token modal environment create beta || true # verified: 1.3.5; "exists" is fine modal secret create --env beta --force brioso-worker \ BRIOSO_WORKER_MODEL=audiocpp modal secret create --env beta --force brioso-registry \ REGISTRY_USERNAME="$REGISTRY_USERNAME" \ REGISTRY_PASSWORD="$REGISTRY_PASSWORD" ``` Post-BRI-439 the Modal app never calls the backend — `generate` writes to R2 via the payload's presigned URLs, so the pull-era keys (`JOBSTORE_API_URL`, `WORKER_SHARED_SECRET`, `BRIOSO_WORKER_RUNTIME`) are gone from `brioso-worker`; `config/secrets-sync.yaml` is the source of truth for what the app still needs. Then create the **proxy-auth token pair** (`wk-…`/`ws-…`): Modal dashboard → Settings → Proxy Auth Tokens (the secret is shown once). Store it straight into Infisical — new keys, not a rotation: ```bash infisical secrets set MODAL_ESPETRO_PROXY_KEY= \ MODAL_ESPETRO_PROXY_SECRET= --env=staging ``` **Expected:** `modal secret create` prints the created/updated secret; the dashboard shows the new proxy token; Infisical lists both new keys. **If it fails:** `environment create` error other than "already exists" → check `MODAL_TOKEN_*` is that account's deploy token (`modal token info`, `modal profile current`). Secret-create failure on `--env beta` → the environment wasn't created; rerun the secret-create above. Wrong-workspace surprise → `modal token info` names the workspace you're actually authed as. ## Step 3 — deploy the Modal worker (per account/env) Actions → **deploy-modal-worker** → `account=espetro`, `environment=beta`, `confirm=yes`. Human action — the workflow spends third-party compute. **Expected:** deploy log ends with `Created web function submit => https://espetro-beta--brioso-worker-submit.modal.run` (plus `status` and `cancel`); the step summary prints them. **Record the printed submit URL** — it must match the `{workspace}-{env}--{app}-submit.modal.run` shape that the beta YAML's `modal: {workspace, environment, app}` block produces (`espetro` + `beta` → `espetro-beta--brioso-worker-submit.modal.run`). If the real URL differs, fix the YAML's `modal` block before importing. **If it fails:** missing `MODAL__TOKEN_*` secrets → add them under repo/org secrets and re-run; no `submit =>` line → the app file didn't expose the endpoint — check the run log against the workflow's URL-extraction step. ## Step 4 — redeploy beta through the plan-2 pipeline So CI re-renders the env (Infisical hash check) with the new `MODAL_ESPETRO_PROXY_*` names present: ```bash bun run release:beta # cuts vYYYY.WW.N-beta.M → deploy-pyinfra.yml # or: Actions → "Deploy web (pyinfra)" → env=beta, ref= ``` (`deploy-beta.yml` is dead audit-trail only — do not dispatch it.) **Expected:** deploy-pyinfra run is green end-to-end: drift-gate → validate → deploy → origin `/health` smoke → `/internal/config-audit`. **If it fails:** drift-gate red → real VPS drift, resolve before deploying; config-audit 503 → a REQUIRED key is missing from the rendered env — its output names the key; smoke sha-pin mismatch → the wrong artifact deployed. ## Step 5 — zero-GPU probe checks ```bash curl -s -o /dev/null -w '%{http_code}\n' -X POST \ "https://espetro-beta--brioso-worker-submit.modal.run" -d '{}' # 401 curl -s -o /dev/null -w '%{http_code}\n' -X POST \ -H "Modal-Key: " -H "Modal-Secret: " \ "https://espetro-beta--brioso-worker-submit.modal.run" -d '{}' # 400 (validated, not spawned) ``` An empty body with the proxy pair returns 400 (`missing …` from `validate_push_job`) — it exercises auth + validation without spawning a GPU call. A real end-to-end submit is covered by the admin canary job below, not curled by hand. **Expected:** 401 without the pair; 400 with it. **If it fails:** 401 *with* the pair → the keys are deploy tokens, not proxy tokens — create a proxy-auth pair in the dashboard and update Infisical. 404 → wrong URL or app name (compare the recorded deploy URL from step 3). Timeout → Modal control plane — retry once, then hop 3–4 of the ops table in [gpu-gateway-canary.md](gpu-gateway-canary.md). Also verify in the admin console (tailnet): beta → GPU Gateway → `modal-espetro` shows health `cold`/`ready` and budget $10. ## Step 6 — import the beta config 1. Export first — import is **replace semantics** (rows absent from the document are deleted with their key rings): `GET /api/admin/beta/gpu-gateway/export` (admin console → export, or the hook route) and save the YAML as the rollback file. 2. Import `config/gpu-gateway.beta.yaml` from this repo (`POST /api/admin/beta/gpu-gateway/import` with `{ "document": "" }`, or the UI's import affordance — pass `ifMatchVersion` from the export when posting by hand). 3. Confirm the registry shows `home` then `modal-espetro`, chain `[home, modal-espetro]` — both enabled. The chain order IS the policy: home serves when healthy, modal-espetro is the fallback. There are no weights to park. **Expected:** `applied: true`, the two rows in `inference_routes` with `chain_position` 0/1. **If it fails:** 400 → schema rejection (the error names the field — the checked-in file is CI-verified, so a 400 means the doc was hand-edited or a *deployed* schema predates the file's fields); 412 → someone edited since your export — re-export and re-import. A runtime row landing but `health: unreachable` → `MODAL_ESPETRO_PROXY_*` names don't resolve in beta env — back to step 4's config-audit output. ## Done-state Registry imported with chain `[home, modal-espetro]`, both enabled, the `submit` endpoint auth-verified, and the home daemon's real tailnet host filled into `push.url`. Smoke it with an admin job pinned to `runner: modal-espetro` (admin console), then watch day-2 operations in [gpu-gateway-canary.md](gpu-gateway-canary.md). ## Appendix A — admin "still doesn't work" triage (plan §5.3) Human on tailnet, read-only: 1. `curl -sI https://admin.usebrioso.com/dash/` from a tailnet device. No route → you're not on the tailnet, or Caddy is down. 2. `systemctl status pocketbase` and `grep ADMIN_UPSTREAM /etc/brioso/pocketbase.env`. Post-U7a the env keys are `ADMIN_UPSTREAM_PROD`/`ADMIN_UPSTREAM_BETA` (loopback :8090/:8091 — no slot ports); a leftover slot port (:3000/:3002) is BRI-366. 3. The GPU tab loads but edits do nothing on **prod**: prod's chain is `config/gpu-gateway.prod.yaml` — import it once prod's deploy carries the push-only schema (same replace-semantics caveat). --- ## [GPU gateway operations (post-BRI-439 push-only)](https://fitxers.usebrioso.com/internal/gpu-gateway-canary/) (Internal) # GPU gateway operations (post-BRI-439 push-only) Supersedes the BRI-420 canary/weight-ramp checklist — weights, canary rules, and ramp stages no longer exist. Dispatch order IS the policy: `fallbackChain` in the resolved gateway config, walked top to bottom by `dispatchViaPushChain`. This doc is the operator's reference for how a job moves and where to look when it doesn't. ## How a job dispatches 1. `POST /jobs` creates the row (`state: new`, `runner:` its assigned runtime id or the default) and fires `dispatchJob` — the event-driven path. The 60 s `dispatch-sweep` loop picks up anything left `new`. 2. `dispatchViaPushChain` walks `dispatchCandidates(job.runner)` — the job's runner first, then `fallbackChain`, dedup'd — skipping runtimes that are disabled, model-mismatched, missing an impl, `unreachable`, or `over_budget`. 3. For each candidate: `promoteRunner` the row, take a 600 s dispatcher lease (`claimForRunner(id, "-dispatcher")`), `impl.submit`. A retryable failure (429 busy, unreachable, timeout) retries once in place, then walks to the next candidate; a non-retryable failure releases the queue row and runs `failOrFallback`. 4. Success stamps `leased_by = ":"`. Then: - `kind: home`/`local` — the push daemon (BRI-437) drives state transitions itself via the worker-secret routes (`PATCH /jobs/:slug`, `/lease`, `/fail`, `/claimed`). The reaper polls `GET push.url/` as a stale-call backstop. - `kind: modal` — the spawned `generate` FunctionCall writes to R2 via the payload's presigned URLs and returns the result to the `status(callId)` poll; nothing on Modal calls the backend. The row goes `composing` on submit and the reaper's poll is the only completion path. 5. `failOrFallback` promotes the row to the next `fallbackChain` entry on exhausted attempts; after the last entry the job fails terminally. Config changes (chain order, enable/disable, budgets) go through the admin console (GPU Gateway) or the import/export routes — import is **replace-semantics**, so export first. ## Cost guardrails (always on) - `budget.dailyCents = 1000` per Modal runtime ($10/day soft cap; the runtime reads `over_budget` at 2x and is skipped in the chain walk). - `maxConcurrency: 2` on beta. - `idleExitSeconds: 60`. - A Modal dashboard budget alert at $10. ## Stop conditions — disable + drain immediately - The same slug submitted more than 3 times. - Stuck count > 3. - A 401/403 on `submit` (proxy token wrong/expired). Drain verb: `modal app stop brioso-worker -e ` (proxy-auth) or disabling the runtime in the admin console — a disabled runtime is skipped in the walk and `modal:app-stop` releases in-flight calls on next poll. Rollback layers: disable in the UI → `modal app stop` → restore the exported gateway file via import. ## Observability — where a stuck job is stuck | Hop | Look at | Symptom → cause | |---|---|---| | 1 `POST /jobs` | web journal; `jobs` row | no row → auth/validation; `unverified` → OTP | | 2 Dispatch | `job_events` + sweep log (`dispatch-sweep`) | `new` with no submit → sweep down or all candidates skipped (disabled/unreachable/over_budget) | | 3 Submit | `gateway.submit` span; `job_events.routed` | 401/403 → proxy token/env names; 404 → `modal:` block URL or `push.url`; timeout → Modal control plane / daemon down → walks to next candidate | | 4 Container/daemon start | Modal dashboard `generate` calls, or daemon log | spawn queued → `max_containers`/capacity; image pull; 429 busy → daemon saturated | | 5 Generate + upload | worker/daemon spans; R2 keys | heartbeat stops → crash/OOM → lease expiry → reaper promotes | | 6 Completion | `status(callId)` poll (modal) or PATCH writebacks (daemon); `completed` event | stale call → reaper `failOrFallback` | | 7 UI | job JSON, browser network | `paid` without audio → delivery/presign, don't rerun | The admin console's runtime detail shows hops 2–4 live per runtime: queue depth, last submit, health state, spend ledger. --- ## [Home GPU inference server setup (BRI-249)](https://fitxers.usebrioso.com/internal/home-gpu-inference-server-setup/) (Internal) Superseded by `apps/worker/` + `home-worker-local-run.md`. # Home GPU inference server setup (BRI-249) Phase 1 of [BRI-248](https://linear.app/brioso/issue/BRI-248) — turning the home PC's RTX 4070 Ti Super into a dedicated, headless Ubuntu Server running `apps/worker`, the poll-based home GPU runner (see that package's own code and `apps/worker/AGENTS.md` for the worker itself). Phase 2 (benchmarking, BRI-250) and Phase 3 (BRI-251, folding the earlier push-based HTTP server into this poll worker) are complete — this box now runs `apps/worker` under pm2, with no inbound HTTP surface. This doc covers the parts that need physical/BIOS access to the box and can't be done from a coding session: disk wipe, OS install, boot order, base hardening, and the NVIDIA/CUDA stack. For the worker's own code, env vars, and pm2 supervision, see `apps/worker/AGENTS.md`. ## Decision recap The home PC dual-drives Windows: Drive C has Windows, Drive D is a second physical disk with only a disposable Steam library. Nothing on either drive needs preserving. A same-drive dual-boot was ruled out (risks Windows during partition resize, can hang on a boot menu with no monitor attached — breaks the "auto-starts after power loss" requirement). Instead: wipe Drive D, install Ubuntu Server there as a fully separate disk, and set the BIOS to boot Drive D by default. Drive C / Windows is never touched, and stays reachable later via the BIOS's one-time boot-device menu. ## 0. Pre-flight check Before wiping anything, confirm from BIOS or Windows Disk Management: - Drive D is a real second physical disk, not a partition on the same disk as C. - It has 100-200GB free (plenty for OS + venv + model weights). - Note whether it's SSD or HDD — doesn't block anything, just useful context for Phase 2 benchmarking. ## 1. Wipe Drive D and install Ubuntu Server 24.04 LTS 1. Delete the Steam library on Drive D. 2. Boot the Ubuntu Server 24.04 LTS installer from USB. 3. Use manual/custom partitioning and explicitly select **Drive D** as the install target. Do not let the installer touch Drive C. 4. During install: enable OpenSSH server, create a non-root user with sudo. ## 2. BIOS boot order - Set Drive D (Ubuntu) as the default/primary boot device. - Enable "restore power state on AC loss" / "power on after power failure" if the BIOS supports it — this is what satisfies the "auto-starts after power loss" requirement (BRI-249 DoD #9-10). - Leave Windows on Drive C selectable via the BIOS's one-time boot menu (commonly F8/F11 at boot) — no shared bootloader, nothing to configure on the Windows side. ## 3. Base OS setup ```bash sudo apt update && sudo apt upgrade -y # Unattended security patches sudo apt install -y unattended-upgrades sudo dpkg-reconfigure -plow unattended-upgrades # Firewall — allow SSH only. The worker is a poll-based outbound client # (apps/worker), not a server — it opens no inbound port, so nothing else # needs a firewall exception. sudo apt install -y ufw sudo ufw allow OpenSSH sudo ufw enable # Time sync (systemd-timesyncd ships enabled by default on 24.04; verify) timedatectl status # Disk monitoring sudo apt install -y smartmontools sudo systemctl enable --now smartd ``` Static/reserved LAN IP: prefer a DHCP reservation on the router (simplest, survives netplan changes across reinstalls). If you'd rather set it on the box, edit `/etc/netplan/50-cloud-init.yaml` (or whatever the installer generated) with a static `addresses:`/`routes:`/`nameservers:` block and run `sudo netplan apply`. Hostname: `sudo hostnamectl set-hostname ` if the installer's default isn't what you want. journald is enabled by default. pm2 (section 6 below) manages its own log files under `~/.pm2/logs/`; `brioso logs` tails those directly, no journald integration needed. ## 4. NVIDIA driver + CUDA `packages/stable-audio-3/pyproject.toml` pins `torch==2.7.1` / `torchaudio==2.7.1` off the `pytorch-cu128` index for Linux — the driver and CUDA toolkit need to support CUDA 12.8. ```bash # Driver — ubuntu-drivers picks the recommended version for the installed GPU sudo apt install -y ubuntu-drivers-common sudo ubuntu-drivers install sudo reboot # After reboot, confirm the driver loaded nvidia-smi # Expect: RTX 4070 Ti Super, 16384MiB total, temps/power draw visible # CUDA toolkit (nvcc etc.) — only needed if something requires compiling # against CUDA headers; the PyTorch wheels bundle their own CUDA runtime. # Install if the `apps/worker` setup below hits a missing-toolkit # error (Triton JIT-compiles a small driver shim at runtime — see # apps/runpod/Dockerfile.sa3's build-essential comment for why that needs a # real GPU present, which this box has and RunPod's build stage doesn't). sudo apt install -y build-essential ``` Validate CUDA is visible to PyTorch once the Python env below is set up: ```bash uv run --project apps/worker python -c \ "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))" # Expect: True NVIDIA GeForce RTX 4070 Ti Super ``` ## 5. Python environment / repo checkout Mirrors `.agents/docs/home-worker-local-run.md`'s prerequisites and `apps/runpod/Dockerfile.sa3`'s dependency sequence (same `pytorch-cu128` pin, same flash-attn wheel-matching, same `build-essential`-at-runtime gotcha for Triton). **Git auth:** the repo is private and org policy disables repo-scoped deploy keys ("Disabled by briosoco" under Settings → Deploy keys) — use a fine-grained Personal Access Token instead (`https://github.com/settings/personal-access-tokens/new`, resource owner `briosoco`, repository access limited to `brioso`, Contents: Read-only). Clone over HTTPS and enter the token as the password when prompted; `git config --global credential.helper store` caches it for future pulls (only takes effect on prompts *after* it's set — if you configured it after the first clone, the next `git fetch`/`pull` will prompt once more, then it's cached in `~/.git-credentials`). ```bash curl -LsSf https://astral.sh/uv/install.sh | sh source ~/.local/bin/env # puts uv on PATH for this shell; new shells pick it up automatically sudo apt install -y git ffmpeg build-essential sudo mkdir -p /opt/brioso-inference sudo chown "$USER:$USER" /opt/brioso-inference git clone https://github.com/briosoco/brioso.git /opt/brioso-inference/repo cd /opt/brioso-inference/repo # apps/worker is a member of the repo's root uv workspace — see # apps/worker/AGENTS.md for the full install (uv pip install -e apps/worker # plus the stable-audio-3/musicgen-adapters/inference-core packages it needs # at generation time). This installs sa3-worker, inference-core, # musicgen-adapters, and the pytorch-cu128 torch/torchaudio build for linux # into the shared workspace .venv. mise trust bun install uv sync --all-packages ``` Repo root's `.python-version` pins `3.12` — needed because Ubuntu 26.04's default `python3` is 3.14, and `torch==2.7.1+cu128` only ships wheels for cp312/cp313/cp313t. Without the pin, `uv sync` picks the system 3.14 interpreter and fails with "doesn't have a source distribution or wheel for the current platform". `uv` downloads its own managed CPython 3.12 automatically the first time, no manual Python install needed. flash-attn is not part of the uv-resolvable dependency set (no prebuilt wheel on PyPI) — `apps/runpod/Dockerfile.sa3` installs it as a separate step against a matching cu12/torch2.7/cxx11-abi wheel from the flash-attention GitHub releases. Only needed if `sa3_worker.model` ends up requiring it at import time; skip until proven necessary — `stable-audio-tools==0.0.20`'s default attention path does not hard-require it. ## 6. Run the worker Supervision is pm2, the same on every platform (see `apps/worker/AGENTS.md`'s "Supervision" section) — no systemd unit, no per-box service file to copy or `chown`. The worker reads plain env vars (not Infisical) from its own `.env` in `apps/worker/`. Copy `apps/worker/.env.example` and fill in at minimum: ```bash cd /opt/brioso-inference/repo/apps/worker cp .env.example .env ``` `SA3_MODEL_ID=stable-audio-3-medium` is a **gated** HuggingFace repo — you'll need an account that has accepted the license at `huggingface.co/stabilityai/stable-audio-3-medium`, and a read-scope token from `huggingface.co/settings/tokens`: ``` HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxx ``` Also set `WORKER_SHARED_SECRET` (must match the backend's value — pull it from Infisical dev/prod, this box is not on Infisical) and `JOBSTORE_API_URL=https://usebrioso.com`. No tailnet, no admin request — the worker reaches the backend over the public API the same way any HTTPS client does; see `apps/worker/AGENTS.md`'s "Environment" section for the full variable list. Then start it under pm2: ```bash pm2 start deploy/pm2/ecosystem.config.cjs pm2 save pm2 startup # prints a native init command; run the command it prints ``` Watermarking (`packages/inference-core/src/inference_core/watermark.py`, `embed_watermark`) is not wired into the generation path — it isn't called from either RunPod's handler or the home worker's live path today either. Open question for the team before this box's output reaches production catalog tracks, not a setup blocker. ## 7. Verify (BRI-249 DoD) ```bash brioso status # human summary: pm2 process state brioso status --json # same, machine-readable — useful from a coding session brioso logs --lines 50 # confirm the claim loop is polling, and the model # load log line appears only once across jobs brioso engine # prints SA3/diffusers on this box (CUDA) ``` Presence telemetry (the `/health` replacement — state, `model_loaded`, accelerator, idle seconds, `load_error`) lands in Phoenix rather than a local endpoint; see `apps/worker/AGENTS.md` for where to look. To confirm end-to-end generation, insert a `runner="home"` job against the real backend (`JOBS_DEFAULT_RUNNER=home` on `apps/web` routes new jobs here without manual row patching) and confirm it reaches `paid` with tracks in R2 — a second identical job should reuse the resident model (no reload log line the second time). ## 8. Unattended reboot test ```bash sudo reboot ``` With zero manual intervention afterward, confirm: - BIOS boots Drive D (Ubuntu), not the boot menu waiting on input. - The box comes up on the network at its reserved IP and SSH is reachable. - `brioso status` shows the worker running — pm2's `pm2 startup` unit restarts it automatically, no per-service systemd config needed. - The model auto-loads without any manual step — `brioso logs` reaches a `ready`/ engine-loaded line within a few minutes of the box coming up. This is what satisfies DoD items 9-10 ("service survives/restarts after reboot", "after reboot, machine automatically reaches READY with model loaded"). --- ## [Local Stack — Home Worker + Garage S3](https://fitxers.usebrioso.com/internal/home-worker-local-run/) (Internal) # Local Stack — Home Worker + Garage S3 Run the full Brioso stack locally with the home GPU worker and a local Garage S3 bucket, without touching real Cloudflare R2. This covers BRI-199 (runner routing + Garage S3). Garage is bootstrapped via Docker directly. ## Overview ``` apps/web (TanStack Start) ├── POST /jobs → jobs.runner = "home" (when JOBS_DEFAULT_RUNNER=home) └── dispatch sweep → POST to the daemon's /jobs (push; see home-worker-wake.md) │ ├── SA3 generation (MLX on Apple Silicon, diffusers on CUDA) └── PUT to Garage (presigned PUT URLs arrive inside the push payload — no worker-side R2 credentials) ``` Jobs default to the gateway chain head in production. `JOBS_DEFAULT_RUNNER=home` routes every new job to the home worker. ## Prerequisites ```sh mise trust && bun install # one-time per worktree uv pip install -e apps/worker uv pip install -e packages/stable-audio-3 packages/musicgen-adapters packages/inference-core # ffmpeg must be on PATH ``` ## One-time: Garage S3 ### Garage on Docker Start Garage and initialise a bucket and key: ```sh docker run -d --name garage \ -p 3900:3900 -p 3903:3903 \ -v garage-data:/data \ garagehq/garage:v1 # Wait for startup, then initialise a layout (one-time) docker exec garage garage layout assign /data garage -r 1 # Create the jobs bucket docker exec garage garage bucket create brioso-jobs # Create an API key for the web app docker exec garage garage key create local-dev # Allow read+write on the bucket # (replace with the name printed by the previous command) docker exec garage garage bucket allow brioso-jobs \ --key --read --write ``` The `key create` command prints an access key ID and secret. Copy them for the config step below. ### Garage on bare-metal or with nono sandbox You need to have it pre-installed. ## Configure apps/web Infisical's `dev` environment already provides Garage-pointed `R2_*` values — a team-wide, 2-dev convenience setup. Each developer still needs their own local Garage running, since the endpoint and key are machine-local; point Infisical dev's `R2_ACCESS_KEY_ID`/`R2_SECRET_ACCESS_KEY` at your own `garage key create` output if they differ. Because of this, `apps/web/.env` normally doesn't need its own `R2_*` block to run `bun run dev:infisical`. Set the following in `apps/web/.env` (needed either way for runner routing and worker auth; needed for the full `R2_*` block only when running `bun run dev`, with no Infisical): ```env # ── Runner routing ─────────────────────────────────────────────── # Every new job inserts with this runner when the caller doesn't specify one. # "home" → the dispatch sweep pushes jobs to the daemon's POST /jobs. # Unset → home-capacity bias, then the gateway chain head. JOBS_DEFAULT_RUNNER=home # Optional: allow POST /jobs callers to pin a specific runner. # Default-deny; only intended for local dev. Unset to disable. # JOBS_ALLOW_RUNNER_OVERRIDE=true # ── Garage S3 (local) ─────────────────────────────────────────── # Only needed for `bun run dev` (no Infisical) — dev:infisical already gets # these from Infisical's dev environment (see above). R2_ENDPOINT=http://localhost:3900 R2_REGION=garage R2_FORCE_PATH_STYLE=true R2_BUCKET=brioso-jobs R2_ACCESS_KEY_ID= R2_SECRET_ACCESS_KEY= # ── Worker auth ───────────────────────────────────────────────── # Must match WORKER_SHARED_SECRET set in the worker's environment. WORKER_SHARED_SECRET=local-dev-secret ``` Restart the dev server after changing env vars. ## Configure apps/worker The worker reads plain env vars, not Infisical. Set in its `.env` or shell: ```env # Auth — must match WORKER_SHARED_SECRET above WORKER_SHARED_SECRET=local-dev-secret # Backend to poll JOBSTORE_API_URL=http://localhost:3000 # Optional: MLX vs diffusers (auto-detected on macOS vs other platforms) # SA3_ENGINE=mlx # Optional: HuggingFace token for first-run SA3 weight download # HF_TOKEN= # Optional: refine service (no-op if unset) # BRIOSO_REFINE_URL= # BRIOSO_REFINE_API_KEY= # Optional: observability (no-op if unset) # PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006 # POSTHOG_API_KEY= ``` ## Run the stack (one command) For the common case (default SA3/MLX model, no flag overrides), skip the manual two-terminal dance below: ```sh mise run dev:home-worker ``` This applies pending `packages/db` migrations, then starts apps/web (`dev:infisical`, with `JOBS_DEFAULT_RUNNER=home`) and apps/worker together, resolving `WORKER_SHARED_SECRET` from your shell env or, failing that, Infisical's dev project (`infisical secrets get WORKER_SHARED_SECRET --env=dev`), and reading vite's actual bound port from its own stdout (vite auto-increments past `:3000` when another worktree already has a dev server running). It exits immediately if the secret isn't set either way — add it to Infisical's dev project or export `WORKER_SHARED_SECRET` yourself first. Logs from both interleave in one terminal; Ctrl+C stops both. You still need Garage running first (`docker start garage`, or `mise run s3:up`). See `scripts/dev-home-worker.ts` for exactly what it runs. If you need to override the model, pin a fixed secret across restarts, or run Garage/web/worker in separate terminals for closer inspection, use the manual flow below instead. ## Run the stack ```sh # Terminal 1 — Garage S3 docker start garage # Terminal 1 — web app (from repo root) JOBS_DEFAULT_RUNNER=home bun run dev:infisical # Terminal 2 — home GPU worker uv run brioso-worker worker \ --model sa3 \ --worker-id "$(hostname)" \ --store api ``` ## Verifying it works 1. Open `http://localhost:3000/dev/login?persona=creator&redirect=/studio` to authenticate. (`agency` is a kept alias for `creator`; both mint the same session — canonical persona for studio QA is `creator`, see `.agents/docs/dev-seed-states.md`.) 2. Create a job through the studio UI. 3. Confirm the `jobs.runner` column is `"home"`: ```sql SELECT slug, runner, state FROM jobs ORDER BY created_at DESC LIMIT 5; ``` 4. Watch the worker terminal — it should log a claim and begin generation. (On the push daemon the equivalent log is `push_daemon.job_accepted` plus a `POST /jobs` 202 — the sweep pushes instead of the box polling.) 5. After the worker finishes, `state` should be `"paid"` and Garage should contain preview/master objects under the slug key namespace: ```sh docker exec garage garage object list brioso-jobs ``` ## Running a single job via script (no GPU needed) To land a `runner="home"` job without a GPU, for fast funnel testing: ```sh RUNNER=home JOBS_ALLOW_RUNNER_OVERRIDE=true bun scripts/fake-complete.ts ``` This hits real HTTP endpoints so it exercises auth, schema validation, idempotency, and `completeGeneration` — but skips the GPU entirely (track keys point at objects that don't exist, so audio players render empty). ## Switching back to RunPod Comment out or unset `JOBS_DEFAULT_RUNNER`. New jobs insert with `runner="runpod"` and the dispatch sweep handles them. Restart the dev server. ## Troubleshooting **Worker never claims jobs** - Is `jobs.runner = 'home'` in the DB? Check with `SELECT runner FROM jobs WHERE slug = '[[ORCA_RICH_MD:907aa2a23f6294e8b43528c898725d33:inline-html:%3Cslug%3E]]'`. - Is `WORKER_SHARED_SECRET` the same in both web app and worker env? - Is the worker pointing at the right `JOBSTORE_API_URL`? **Garage uploads return 403** - Did you run `garage bucket allow` for the key you created? - Is `R2_FORCE_PATH_STYLE=true` set in the web app env? - Is `R2_ENDPOINT=http://localhost:3900` (not the real R2 endpoint)? **MLX weights download / venv bootstrap on first run** - The worker auto-bootstraps the Stable Audio 3 MLX repo on first use: `ensure_mlx_repo_cloned()` clones the repo (if missing) and runs its `optimized/mlx/install.sh -y` (if `.venv/bin/python` is missing). This is non-interactive and can take a few minutes (dependency install + optional uv install); the first generation job may appear stalled while it runs. - Set `HF_TOKEN=[[ORCA_RICH_MD:907aa2a23f6294e8b43528c898725d33:inline-html:%3Cyour%20HuggingFace%20token%3E]]` in the worker's env — `sa3_mlx.py` lazy-fetches the ~4 GB model weights from HuggingFace on first use after the venv exists. - If bootstrap fails (network, disk, permissions), the job fails and retries on the next lease cycle (5 min); fix the underlying error rather than re-running `install.sh` by hand. --- ## [Home push daemon ("serverless home", BRI-437)](https://fitxers.usebrioso.com/internal/home-worker-wake/) (Internal) # Home push daemon ("serverless home", BRI-437) Deploy-simplification step 5b. The home box joins the `Runtime` registry as a **push** runtime: the backend's dispatch sweep POSTs each job to the box over the tailnet (`submit(job) → call_id`) instead of the box polling `/jobs/claim`. This doc is the tailnet-host runbook. History: this daemon grew out of the BRI-410 wake agent — same stdlib HTTP server, same BRI-422 auth chain — which spawned a claim loop on `POST /wake`. BRI-437 deleted the spawn machinery and made the daemon the worker itself (`brioso-worker serve` is now THE home worker unit; the old `POST /wake` stayed on as a deprecated no-op through rollout; BRI-439 removed it). ## What it is A tiny stdlib HTTP server (`apps/worker/src/apps/worker/wake_agent.py`) that runs `process_job` in-process on a supervised single-flight slot: - `POST /jobs` — authorized per the [auth model](#auth-bri-422) below. Body is the push contract: `{slug, params, uploadUrls[{url,key}], deadlineUnix}` — `params` is the job's full JobResponse dict; the flat `uploadUrls` are the presigned R2 PUT URLs minted at submit, re-nested into the tiered shape `PresignedUploader` expects. Returns **202** `{call_id}` on accept, **429** `{detail}` while the slot is busy (single in-flight call, mirroring `generation_slot(1)`), **409** on a duplicate in-flight slug or a passed deadline, **422** on a malformed payload. - `GET /jobs/{call_id}` — same auth. `{state, call_id, slug, result?, error?}` with `state ∈ queued|running|done|failed|cancelled`; a call_id the daemon never saw answers `{"state": "unknown"}` — the web side treats that as failed per sweep rules. - `GET /health` — unauthenticated (the tailnet is the boundary): `{ok, runtime, agent, engine, in_flight, gpu}`. Per-call state persists as JSON under `/jobs/.json`, so `GET /jobs/...` survives a restart; calls still `queued`/`running` at boot are marked `cancelled` (threads don't survive — the reaper's expired-lease arm promotes the row onward). Completion is unchanged: the call's `worker_id` is `:` — the same string the sweep writes into `jobs.leased_by` — so heartbeat `extend_lease`, the `/fail` `leased_by` guard, and the paid-PATCH gate all apply exactly as for the old claim loop (pull-era; removed in BRI-439). Results land via the same `PATCH /jobs/{slug}` upserts + presigned-PUT uploads. ## Install (tailnet host) Requires: repo checkout, `bun install` done, `uv` + worker deps installed (`uv pip install -e apps/worker` + the SA3 venv, same as today), `brioso doctor` green, Tailscale up and reachable from the backend box. ### 1. Zero secrets (BRI-422) No `openssl rand` step, no shared token: the daemon authorizes tailnet peers by `tailscale whois` and loopback callers unconditionally, so a home GPU joins with nothing provisioned — run `brioso-worker serve` and register the runtime with just a push URL. ### 2. `.env` ```bash # required JOBSTORE_API_URL=https://usebrioso.com WORKER_SHARED_SECRET= # optional HOME_WAKE_TOKEN= # only for non-tailnet remote callers BRIOSO_WAKE_ALLOW_NODES=vps-1 # restrict which tailnet nodes may submit BRIOSO_WORKER_RUNTIME=home # runtime id; also the leased_by prefix BRIOSO_WAKE_PORT=7311 # must match the runtime row's push URL BRIOSO_WAKE_STATE_DIR=/path/to/state # default /tmp/wake-agent BRIOSO_PUSH_DRAIN_SECONDS=10 # SIGTERM grace for the in-flight call ``` ### Auth (BRI-422) `POST /jobs` and `GET /jobs/{call_id}` authorize when ANY of these holds — first match wins and is reported as `authorized_by`: 1. **Bearer** — `HOME_WAKE_TOKEN` is configured AND the request's `Authorization: Bearer ` matches (constant-time compare). Back-compat: provisioned deployments keep working unchanged. 2. **Loopback** — the caller's address is a loopback address. Covers local ops AND the local-E2E recipe below — no token, no tailscale. 3. **Tailnet peer** — `tailscale whois ` resolves the caller to a tailnet node. With `BRIOSO_WAKE_ALLOW_NODES` (comma-separated node names, e.g. the VPS node) the resolved name must be listed; unset = any resolved tailnet node. whois failure / timeout / missing CLI fails closed → the rule does NOT authorize. Otherwise `401 {"error": "unauthorized"}`. The bearer remains the right choice for a remote caller outside the tailnet. When the runtime row sets `push.tokenEnv`, the backend sends `Authorization: Bearer ` only when the var resolves non-empty — a declared-but-empty var sends no header (never a 401 from dispatch's side). ### Register the runtime Admin console → GPU Gateway → home runtime → set the **push URL** (`push.url`), leave **Token env name** blank unless calling from off the tailnet: ```yaml - id: home kind: home push: url: "http://.:7311/jobs" # tokenEnv omitted — tailnet peer auth ``` A `push`-configured runtime is dispatched by `submit()` — the sweep's push arm POSTs the job, writes `leased_by="home:"`, and a retryable submit failure (429 busy, 409 stale, 5xx, unreachable) walks the `fallbackChain` in order (modal-espetro on beta). ### Local home GPU (no token, no tailscale) Run the daemon on the same host as `apps/web` and register the runtime with `push.url=http://127.0.0.1:7311/jobs` — the loopback rule authorizes the sweep's submits. This enables a real `home`-kind E2E generation (audiocpp/small model) from `dev:fake`: the sweep POSTs the job to the daemon, the daemon generates and PATCHes the result back. ```bash # 1. Stack (the fake runner is the `local` runtime; the real daemon is a # separate `home`-kind runtime so they never contest) mise run dev:fake -- --no-runner # 2. Register the home runtime + fallback chain via the admin API # (DB layer wins over GPU_GATEWAY_CONFIG_FILE — source: "db"). source .brioso/dev-secrets.env curl -X PUT localhost:3000/internal/admin/gpu-gateway/runtime/home-local \ -H "Authorization: Bearer $WORKER_SHARED_SECRET" \ -H 'content-type: application/json' -d '{ "id":"home-local","kind":"home","enabled":true, "push":{"url":"http://127.0.0.1:7311/jobs"}, "idleExitSeconds":60,"claimTimeoutSeconds":90}' curl -X PUT localhost:3000/internal/admin/gpu-gateway/chain \ -H "Authorization: Bearer $WORKER_SHARED_SECRET" \ -H 'content-type: application/json' -d '{"chain":["home-local"]}' # GET /internal/admin/gpu-gateway → source "db", chain head home-local. # With the DB config live, `local` is unregistered, so new jobs land on # the chain head at insert and the sweep's push arm submits to it # (audit: submit:home-local → dispatcher lease leased_by # "home-local:" → new→composing→paid). # 3. Worker env — apps/worker/.env (gitignored). Source dev:fake's # generated secrets instead of copying the literal value: # source "$(dirname "${BASH_SOURCE[0]}")/../../.brioso/dev-secrets.env" # JOBSTORE_API_URL=http://localhost:3000 # BRIOSO_WORKER_RUNTIME=home-local # BRIOSO_WORKER_MODEL=audiocpp # AUDIOCPP_BACKEND=cpu # AUDIOCPP_BIN= # AUDIOCPP_MODEL_PATH= # AUDIOCPP_TIMEOUT_S=2400 # CPU is far slower than the 900s GPU default # 4. audiocpp on CPU-only Linux. The pinned v0.7.3 release's Ubuntu # prebuilt needs glibc ≥ 2.38 (24.04) — on 22.04 build it instead: # git clone --depth 1 --branch v0.7.3 https://github.com/0xShug0/audio.cpp # cd audio.cpp && scripts/build_linux.sh --backend cpu \ # --model-set custom --models stable_audio --target audiocpp_cli # (needs cmake ≥ 3.20 + gcc; binary lands in build/linux-cpu-release/bin/) # Lightest real model: # Stable-Audio-3-Small-Music-GGUF/stable-audio-3-small-music-q8_0.gguf # (~1.7 GB) from HF audio-cpp/audio.cpp-gguf. 15 s stereo 44.1 kHz # generates in ~17 s on 8 CPU cores. # 5. Push daemon — loopback auth, no token cd apps/worker && uv run brioso-worker serve --port 7311 # 6. Submit (any persona; POST /jobs defaults runner → chain head → swept) curl -b /tmp/cookies -X POST localhost:3000/jobs -H 'content-type: application/json' \ -d '{"description":"lo-fi hip hop beat","email":"creator@brioso.test", "duration_seconds":15,"track_count":1}' ``` Expected daemon log for a landed job: `push-daemon: "POST /jobs" 202` → `push_daemon.job_accepted` (obs) → `PATCH /jobs/` (composing → paid) → `push_daemon.call_finished`. ### 3. Run — pm2 (recommended, mirrors today's setup) ```bash pm2 start deploy/pm2/ecosystem.wake-agent.config.cjs pm2 save && pm2 startup ``` ### 3-alt. Run — systemd (Linux) Edit `deploy/pm2/brioso-wake-agent.service` (User/WorkingDirectory/ EnvironmentFile), then: ```bash sudo cp deploy/pm2/brioso-wake-agent.service /etc/systemd/system/ sudo systemctl daemon-reload && sudo systemctl enable --now brioso-wake-agent ``` ### Foreground/debug ```bash brioso worker serve --port 7311 # host CLI wrapper brioso-api worker serve --port 7311 # same, via @brioso/cli uv run brioso-worker serve # canonical Typer entrypoint ``` All three land on `run-wake-agent.sh` (sources `.env`; `HOME_WAKE_TOKEN` optional — without it the daemon warns on stderr and authorizes loopback + tailnet peers only). `brioso-api worker serve` needs `BRIOSO_REPO_ROOT` when invoked outside a checkout. ## Verify ```bash # local (no token needed for /health) curl -s http://127.0.0.1:7311/health | jq . # status of a call — unknown is the never-seen/restart answer curl -s http://127.0.0.1:7311/jobs/ | jq . # remote callers still use the bearer when HOME_WAKE_TOKEN is set curl -s http://.:7311/jobs/ \ -H "Authorization: Bearer $HOME_WAKE_TOKEN" | jq . # over the tailnet, from the backend host curl -s http://.:7311/health ``` Expected `/health` shape: ```json { "ok": true, "runtime": "home", "agent": {"pid": 1, "uptime_seconds": 12}, "engine": {"state": "ready", "load_error": null}, "in_flight": null, "gpu": {"detected": true, "accelerator": "cuda", "name": "RTX 4090", "memory_used_mb": 0, "memory_total_mb": 24564} } ``` ## Teardown semantics - SIGTERM/SIGINT (pm2 `stop`/`restart`, `systemctl stop`, Ctrl-C): the in-flight call's record is marked `cancelled` and the runner thread gets `BRIOSO_PUSH_DRAIN_SECONDS` (default 10) to land in-flight PATCHes before the process exits. pm2's `kill_timeout` (10 s) bounds it; the job row's lease expires and the reaper promotes it onward. - Reboot/power loss: `queued`/`running` records flip to `cancelled` on the next boot (JobBook reopen); `GET /jobs/` then reports `cancelled`. ## Troubleshooting - `401` on /jobs — caller matched no rule: wrong/absent bearer AND non-loopback AND `tailscale whois ` didn't resolve (or the node isn't in `BRIOSO_WAKE_ALLOW_NODES`). Check `tailscale whois ` from the daemon host. - `429` on every submit — a call is genuinely in flight (`GET /health` → `in_flight`), or a wedged thread holds the slot; restart the daemon to release it. - `409` on submit — duplicate in-flight slug (the web re-dispatched a job already running — same call_id is returned) or the payload's `deadlineUnix` already passed. - `422` on submit — payload doesn't parse as the push contract; the `detail` names the bad field (usually `uploadUrls` key convention drift — keys must be `track-.`). - `/health` unreachable over tailnet — `tailscale status` on the host, and confirm the port isn't bound to localhost only (`BRIOSO_WAKE_HOST`). - Port already in use — another daemon running? `pgrep -af brioso-worker`. --- ## [Infisical environment & secrets guide (apps/web)](https://fitxers.usebrioso.com/internal/infisical-env-guide/) (Internal) # Infisical environment & secrets guide (apps/web) Secrets never live in `apps/web/.env`. That file is empty on purpose. All runtime secrets come from Infisical and are injected into the process by the Infisical CLI. The workspace is configured in `.infisical.json` (workspaceId `66410021-9c0a-4075-803b-f9286f21145f`). - `apps/web/.env.example` is **documentation only**. It is the catalogue of every variable the web app understands, with a comment per key. Do not copy it into a real `.env` and paste secret values there; nothing loads it at runtime. Read it to learn what a key means, never to source its value. - To run anything that needs secrets (the dev server, scripts, one-off tasks), wrap the command in the Infisical CLI so it fetches the `dev` environment and exports the variables into the child process: ``` # canonical dev server (defined as a package.json script) bun run dev:infisical # = infisical run --env=dev -- vite dev # any other secret-needing command uses the same wrapper: infisical run --env=dev -- ``` - `apps/web/.env` is intentionally empty. Do not hand-fill it. If you need a value locally that Infisical does not hold, set it in your shell or pass it inline rather than committing it. - The config loader (`src/server/config.ts`) resolves each key as `infisicalSnapshot[key] ?? process.env[key]`, trimmed, so a value present in Infisical wins over a same-named shell var. An unconfigured Infisical (no `INFISICAL_CLIENT_ID/SECRET/PROJECT_ID`) is a supported no-op and everything falls back to plain env vars. - Infisical's `dev` environment's `R2_*` and `SESSION_SECRET` are intentionally kept in sync with the team's local Garage setup (see `.agents/docs/home-worker-local-run.md`) rather than pointing at real Cloudflare R2 — a deliberate choice while the team is small (2 devs). Anyone expecting `dev:infisical` to hit real R2 needs to know this changed; reverting means resetting `R2_ENDPOINT`/`R2_ACCESS_KEY_ID`/ `R2_SECRET_ACCESS_KEY`/`R2_BUCKET` in Infisical dev back to real R2 values and removing `R2_REGION`/`R2_FORCE_PATH_STYLE`. - The CLI reuses your `infisical login` session. Run `infisical login` once if a wrapped command errors with an auth/credentials message. ## Infisical folder scheme (1 level, no nesting) - `/` (root) — app-runtime secrets consumed by apps/web via `cfg.get` / `process.env` / `import.meta.env`. - `/stripe` — Stripe keys + pricing (`STRIPE_*`, `*_PRICE_CENTS`). The key set matches across dev and prod; only values differ (test vs live ids). - `/infra` — **prod only**, deploy/infra-only service config NOT consumed by apps/web at runtime (Caddy's `CLOUDFLARE_API_TOKEN` + `MCP_SQLITE_API_KEY`, Phoenix server config `PHOENIX_SECRET` / `PHOENIX_ENABLE_AUTH` / `PHOENIX_ENABLE_MCP_SERVER`). Synced by the VPS deploy into service-specific env files. Do not create `/infra` in dev. ## Intentional dev/prod parity gaps — do NOT "fix" these These keys exist in one env only because they only make sense there. Forcing parity breaks things: | Key | In | Why | |-----|----|-----| | `R2_REGION`, `R2_FORCE_PATH_STYLE` | dev only | dev points at local Garage; prod at real R2 (see above). | | `DATABASE_URL` | prod only | There is no Turso today; the prod DB is a local libSQL/SQLite file (see `data-model.md`). | | `INTERNAL_BASE_URL` | prod only | RunPod completion webhook callback; dev doesn't dispatch to real RunPod. | | `PHOENIX_SECRET`, `PHOENIX_ENABLE_AUTH`, `PHOENIX_ENABLE_MCP_SERVER` | prod only | Phoenix **server** config, read by `deploy/phoenix/phoenix.env`, not the app. Dev Phoenix runs without auth. | | `PHOENIX_API_KEY`, `PHOENIX_COLLECTOR_ENDPOINT` | prod only | app-side OTel export target; dev defaults to `http://localhost:6006` with no auth. | | `POSTHOG_API_KEY`, `POSTHOG_PERSONAL_API_KEY` | prod only | optional; empty in dev = server-side analytics disabled locally. | | `RUNPOD_ENDPOINT_ID_SMALL_MUSIC`, `RUNPOD_ENDPOINT_ID_SMALL_SFX` | — | endpoint ids; deliberately absent from the config-audit manifest while no job requests a small variant. | - **`bun run build` hard-fails on a missing required env var.** The build script runs `scripts/audit-build-env.ts` first, which reuses the same `CONFIG_KEYS` manifest as the runtime audit and exits 1 (printing each gap and its impact) before `vite build` runs. The runtime audit stays non-fatal at boot on purpose (a downed server is its own incident); the build is where a missing secret turns into a failed deploy instead of a broken prod. Run it with dev secrets via `mise run build:web` (= `cd apps/web && infisical run --env=dev -- bun run build`). --- ## [Infrastructure map](https://fitxers.usebrioso.com/internal/infrastructure/) (Internal) # Infrastructure map Operator inventory for the VPS fleet and its GitHub Actions runners. This is an entry point; use the linked runbooks for procedures. The two VPS hosts are distinct. The home GPU runtime and Modal workers are described in the [worker deployment guide](worker-deployment.md). ## Hosts | Host | Role and services | Network access | Configuration source | Last verified | | --- | --- | --- | --- | --- | | Application VPS (`ovhcloud`, `prod.vps.brioso.internal` and `beta.vps.brioso.internal` are two logical inventories on one machine) | Public prod and beta web slots, Caddy, PocketBase admin, Litestream, Phoenix, Drizzle Gateway, MCP SQLite, local Garage backed Turbo cache, and CI runners | Public web traffic through Cloudflare; admin and operations over Brioso Tailscale `tag:server`; CI SSH and cache access from `tag:ci` | [pyinfra v2 inventory](../../deploy/pyinfra/v2/inventory.py), [deploy guide](../../deploy/README.md), [tailnet policy mirror](tailscale-acls.jsonl) | 2026-10-10 for runner registrations; service layout checked against repository code | | Garage VPS (`vps-e311841e`, Tailscale name `brioso-vps-ovh.tail7d3568.ts.net`, `100.95.227.60`) | Production catalogue Garage at `/srv/garage`; scale-to-zero ephemeral GitHub Actions runners (up to 4) on `/srv/ci` | Garage S3 binds loopback `127.0.0.1:3900` and is forwarded over the Brioso tailnet; SSH is tailnet access; no public inbound Garage or SSH port | [Garage runbook](../../deploy/garage-catalog/README.md), [Garage unit](../../deploy/systemd/brioso-garage.service), [OVH runner setup](../../deploy/ci-runner/README.md), [OVH runner plan](../plans/2026-10-10-ovh-github-runner.md) | 2026-10-10, SSH confirmed `ghrunner` and the 20 GiB mount, Garage active, plus GitHub runner online | The application VPS's `garage-cache` is a separate local Turbo cache store. It is not the production catalogue on the Garage VPS. The application VPS configuration is managed by pyinfra; the Garage VPS and its CI runner are currently documented through the linked runbooks and dated operations notes. ## Linux accounts Access describes the intended path and sudo boundary, not a list of SSH keys. Named OVH administrator accounts were checked on 2026-10-10; application service accounts below come from provisioning code, so inspect the host before relying on their live status. | Host | Account | Purpose and access | Sudo boundary | Provisioning source | | --- | --- | --- | --- | --- | | Application | `debian` | pyinfra SSH account for CI and operators over the tailnet | Passwordless general sudo for pyinfra | [v2 inventory](../../deploy/pyinfra/v2/inventory.py) and host cloud init | | Application | `deploy` | Legacy CI deploy SSH account | Passwordless only for reviewed deploy scripts | [users.py](../../deploy/pyinfra/provision/users.py), [sudoers](../../deploy/sudoers/brioso-deploy) | | Application | `diag` | Tailnet SSH for diagnostics and journal reads | Reviewed web slot status and restart commands only | [v2 diag state](../../deploy/pyinfra/v2/state/diag_user.py), [access runbook](diag-access-runbook.md) | | Application | `web-services` | Web, PocketBase, Litestream, Drizzle Gateway, Phoenix, MCP SQLite, and Turbo cache service processes; service account, not operator SSH | No general sudo | [users.py](../../deploy/pyinfra/provision/users.py), [service units](../../deploy/systemd/) | | Application | `garage-cache` | Local Turbo cache Garage process; no login shell | None | [users.py](../../deploy/pyinfra/provision/users.py), [cache provisioning](../../deploy/pyinfra/provision/garage_cache.py) | | Application | `ghrunner` | Rootless Podman runner fleet; user systemd services | Container jobs run as root inside their container; no host sudo grant documented | [runner definitions](../../deploy/ci-runner/README.md) | | Garage | `ubuntu` | Existing recovery and host administration via VPS specific SSH key | Host administration sudo | [OVH runner plan](../plans/2026-10-10-ovh-github-runner.md), [admin observation](../notes/2026-10-10-carlos-vps-admin-access.md) | | Garage | `brioso-admin-carlos` | Named administrator, account local SSH key | Passwordless general sudo | [dated access check](../notes/2026-10-10-carlos-vps-admin-access.md) | | Garage | `brioso-admin-quino` | Named administrator, account local SSH key | Passwordless general sudo | [dated access check](../notes/2026-10-10-quino-vps-admin-access.md) | | Garage | `garage` | Catalogue Garage service; no login shell | None | [Garage install guide](../../deploy/garage-catalog/README.md), [service unit](../../deploy/systemd/brioso-garage.service) | | Garage | `ghrunner` | Rootless Podman runner service; jobs run inside disposable containers | Container root is bounded by the container; no host sudo grant documented | [OVH runner setup](../../deploy/ci-runner/README.md) | ## GitHub Actions routing and capacity The repo variable `RUNNER_TARGET` was `self-hosted-tailnet` when checked on 2026-10-10. Jobs using `${{ vars.RUNNER_TARGET || 'ubuntu-latest' }}` share the `self-hosted-tailnet` runner label. [CI](../../.github/workflows/ci.yml), [artifact builds](../../.github/workflows/build-artifact.yml), and [deploys](../../.github/workflows/deploy-pyinfra.yml) use that variable; [runner budget](../../.github/workflows/runner-budget.yml), [VPS health](../../.github/workflows/vps-health.yml), and [pyinfra drift](../../.github/workflows/pyinfra-drift.yml) are pinned to the self hosted label. See the [runner policy](ci-runner-policy.md) for the budget watchdog and recovery procedure. GitHub's repository runner API reported these registrations **online on 2026-10-10**, before the move to the scale-to-zero orchestrator on the Garage VPS (BRI-433). The old `ovhcloud-ghrunner-*` runners and `brioso-vps-ovh-1` are retired. Runners are now registered on demand, so an empty list is normal when idle; use `ci-fleet-status` on the host for live state. | Host | Registered runner names | Slots | | --- | --- | ---: | | Garage VPS (CI host, BRI-433) | `brioso-vps-1` to `brioso-vps-4`, created per job (scale to zero, none online when idle) | up to 4 | On the Garage VPS, the runner image, disposable job files, and Bun cache use one **shared 20 GiB `/srv/ci` filesystem**. This is total CI capacity, not a 20 GiB allowance for each job. The job container is limited to **5 GiB RAM and 4 CPUs**. A provisional 5 GiB filesystem filled during `bun install`; after the filesystem was enlarged, a completed smoke build measured **8.4 GiB disk use**. These are the 2026-10-10 capacity observations, not sizing guarantees for future jobs. The [OVH runner plan](../plans/2026-10-10-ovh-github-runner.md) keeps the trial history. --- ## [Live-E2E QA harness (BRI-444)](https://fitxers.usebrioso.com/internal/live-e2e-qa/) (Internal) # Live-E2E QA harness (BRI-444) How to run the real login flow on a deployed env (beta/prod) with no real email delivery and no code path that bypasses auth. ## Pieces - **Loopback inbox** — `EMAIL_LOOPBACK_DOMAINS` (comma/space-separated; default `brioso.test`; `-` disables). `sendOtpEmail`/`sendOnce` route any recipient whose domain is listed to the `loopback_inbox` table instead of the configured provider. Routing is per-recipient — Brevo delivery for real users is untouched. - **`GET /internal/admin/inbox`** — reads `loopback_inbox`, newest first. Query params: `to=` (optional filter), `limit` (default 20, max 100). Each message carries `otp` extracted from the html when present. - **`POST /internal/admin/qa-fixture`** — idempotent upsert of the QA account: `qa@brioso.test` user, permanent `active` Pro `subscriptions` row (period end 2099), permanent space `qa-e2e-space` (deep-linkable at `/spaces/qa-e2e-space`), and `users.daily_job_cap` (default 20; override with `{"daily_job_cap": n}`). `?reset=1` first deletes every QA-scoped row (jobs + dependent rows, kits, inbox, verifications, usage counters, catalog state) — never other users' rows. - **Daily job cap** — `users.daily_job_cap`, NULL = uncapped (every normal account). Enforced in `checkGenerationQuota` ahead of entitlement tiers, so it bounds Pro too; `429 DAILY_JOB_CAP_EXCEEDED` when today's `jobs` rows for the email reach the cap. This is the spend bound for a runaway QA generation loop. All `/internal/*` routes require `Authorization: Bearer $WORKER_SHARED_SECRET` (the Nitro `sensitive-route-guard` plugin enforces it — no route-local auth needed). ## Setup ```bash BASE=https://beta.usebrioso.com AUTH="Authorization: Bearer $WORKER_SHARED_SECRET" # 1. (Re)create the fixture. ?reset=1 = clean slate. curl -sf -X POST "$BASE/internal/admin/qa-fixture?reset=1" \ -H "$AUTH" -H 'content-type: application/json' -d '{}' # → {"email":"qa@brioso.test","user_id":"…","space_id":"qa-e2e-space", # "plan":"pro","daily_job_cap":20,"reset":true,"removed_jobs":N} ``` ## Login — the real OTP flow, no alternate session path ```bash # 2. Request the OTP exactly like a user login. curl -sf -X POST "$BASE/auth/request-otp" \ -H 'content-type: application/json' -d '{"email":"qa@brioso.test"}' # 3. Read it back from the loopback inbox. curl -sf "$BASE/internal/admin/inbox?to=qa@brioso.test&limit=1" -H "$AUTH" # → {"messages":[{"otp":"123456", …}]} # 4. Verify — sets a real session cookie. curl -sf -X POST "$BASE/auth/verify-otp" -c qa.cookies \ -H 'content-type: application/json' \ -d '{"email":"qa@brioso.test","code":"123456"}' # 5. Drive the product with the cookie jar. curl -sf "$BASE/studio" -b qa.cookies ``` The verified session is the same `session` cookie a real user gets — every subsequent request (generation, spaces, billing UI) exercises the real auth path. ## Cleanup - Between runs: `POST /internal/admin/qa-fixture?reset=1` re-creates the fixture from scratch. Re-running without `reset` is a no-op upsert. - Removing everything QA-owned is the same call — reset deletes all rows the fixture can produce. The `qa-e2e-space` canvas itself is re-emptied. - Orphaned `sent_emails` claim rows (`otp-` slugs) are per-issuance and inert; they're deliberately not swept. ## Caveats - **OTP rate limits are in-memory** (`@brioso/auth-core` rate-limit Map): they are per-process and not resettable via the DB or the fixture endpoint. A loop that spams request-otp will 429 until the window expires or the process restarts. - **Loopback is recipient-scoped, not provider-scoped**: disabling it (`EMAIL_LOOPBACK_DOMAINS=-`) sends `qa@brioso.test` OTPs to real Brevo — leave the default unless a smoke specifically needs real delivery. - The inbox stores the rendered html; `otp` is extracted at read time (same regex as the file provider's outbox reader), so non-OTP mail shows `otp: null`-ish (absent). ## Deliberately not automated - **Stripe/payments** — the fixture's Pro row is synthetic (`qa_sub_…`); checkout, webhooks, and portal flows are never exercised by this harness. - **Real email delivery** — Brevo for `brioso.test` recipients is defined away; testing real deliverability needs `EMAIL_LOOPBACK_DOMAINS=-` and a human inbox. - **GPU job completion** — the cap bounds submissions, not runner execution; a job that reaches a runtime still burns GPU per the normal pipeline. Pair QA runs with the fake/dev runner or a cheap runtime. - **OTP rate-limit reset** — per-process state, restart-gated by design. - **Concurrent fixture resets** — `qa-fixture` is not atomic across parallel callers; run it once per environment before the QA sweep starts. --- ## [Local stack — env resolution and measured budgets](https://fitxers.usebrioso.com/internal/local-stack/) (Internal) # Local stack — env resolution and measured budgets How `mise run dev:fake` sources env vars and how much memory the local stack actually uses. Plan 1 (`.agents/plans/2026-10-04-dev-fake-no-infisical.md`) §2 D4 for the design; BRI-398 for the implementation. ## Env resolution (`scripts/dev-env.ts`) `resolveDevEnv()` picks one of two sources for the apps/web process: - **Infisical** (canonical, unchanged): the CLI is on PATH **and** authenticated (`INFISICAL_TOKEN` set, or `infisical user get token` exits 0) **and** `BRIOSO_INFISICAL` is not `0`. apps/web is wrapped in `infisical run --env=dev --recursive --` exactly as before. - **Local files** (fallback): anything else. Layered, lowest precedence first: | layer | file | role | |---|---|---| | defaults | `apps/web/.env.example` | committed defaults; empty and `` values count as unset | | overrides | `apps/web/.env.local` | gitignored per-developer overrides | | generated | `.brioso/dev-secrets.env` | `WORKER_SHARED_SECRET`, `SESSION_SECRET` — generated once, persisted, 0600 | Plus defaults applied unless `.env.local` or the shell sets them: `PUBLIC_ORIGIN=http://localhost:3000`, `NODE_ENV=development`, `EMAIL_PROVIDER=file`, `MAIL_OUTBOX_FILE=/.brioso/mail/outbox.jsonl`. Non-empty shell env vars always win. `EMAIL_PROVIDER=file` currently means "no provider configured" — the app warns and skips emails exactly like an unset provider until BRI-399 lands the `file`/`mailpit` adapters that read `MAIL_OUTBOX_FILE`. The generated `WORKER_SHARED_SECRET` is what `resolveWorkerSecret()` returns, so web and the fake runner always agree. Force the local path even with a working CLI: `BRIOSO_INFISICAL=0`. ## Measured memory (Linux x64, bun 1.3.14, vite dev) Measured with `ps -o rss` ~60 s after `mise run dev:fake` was fully up, after hitting `/dev/login?persona=free-new`, `POST /jobs` (Free job → `paid`), and `/spaces`: | process | RSS | plan budget (D2) | |---|---|---| | web: Vite dev + Nitro SSR (one node proc) | **~1.75 GiB** (1,796,700 KiB) | 1.5 GB est. | | fake runner (bun) + dev-fake orchestrator | ~114 MiB combined | 0.2 GB est. | | garage (docker, `dxflrs/garage:v1.0.1`) | idle ≪ 128m limit | 0.13 GB limit | The web process runs ~0.25 GiB **over** the D2 estimate. Total for the default `dev:fake` loop (web + fake runner + garage) is ≈ 1.9 GB of the 8 GB budget — still fits, but the headroom is thinner than the estimate suggested. The heavier combinations in D2 (+audio.cpp ≈ +2 GB, +Playwright ≈ +1 GB) should be re-measured in U7/U6 rather than trusted blindly. ## Verified locally (BRI-398 acceptance, 2026-10-04) - `mise run dev:fake` boots on a box with **no `infisical` on PATH** — no `ERR_MODULE_NOT_FOUND` (the `predev` intl compile runs inside `bun run dev`). - `GET /dev/login?persona=free-new` → 302 + `brioso_session` cookie. - Free job (`POST /jobs`) → fake runner claims, encodes, uploads to Garage → state `paid` in ~4 s. - Boot audit warns (non-fatal, expected): `EMAIL_FROM`, `BREVO_API_KEY`, `RUNPOD_*`, `R2_CATALOG_*` are unset under the local source — the fake path needs none of them. --- ## [Mac worker Cloudflare Tunnel](https://fitxers.usebrioso.com/internal/m5-cloudflare-tunnel/) (Internal) # Mac worker Cloudflare Tunnel BRI-397. This is a dedicated tunnel named `brioso-m5`, separate from the VPS tunnel. The existing worker bearer authenticates requests; Cloudflare Access is not used. Only `/jobs` and `/jobs/{call_id}` reach the workers. The checked in configuration prepares both beta and production with the same route restrictions and authentication. Production uses its own hostname and localhost worker port. Applying the private tunnel configuration and updating the live gateway row are separate rollout steps; this preparation does not change live routing. The diagram shows the prepared production route beside the last verified live SSH route. | Hostname | Origin | Runtime | | --- | --- | --- | | m5-beta.usebrioso.com | http://127.0.0.1:7312 | home-m5-beta | | m5-prod.usebrioso.com | http://127.0.0.1:7311 | home-m5-production | `deploy/cloudflared/m5.yml.example` is the shareable template. On the Mac, PM2 starts `brioso-m5-cloudflared` with `--config ~/.config/brioso/m5/cloudflared.yml`; that private file is the configuration the running connector reads. It has the same `ingress` rules as the template, plus the real tunnel UUID and credentials path. Recreate it from the template on a new Mac, or copy only changed `ingress` entries into an existing private file so its UUID and credentials stay intact. Validate and restart the connector after either change. An ingress `hostname` matches the address in an incoming web request. It does not identify or allowlist the computer sending that request. Cloudflare routes matching `/jobs` and `/jobs/{call_id}` requests to the corresponding local worker; the final rule returns 404 for other paths. Both names are public DNS names, subject to Cloudflare's edge rules. The workers require their own `HOME_WAKE_TOKEN` bearer for the job routes, including requests forwarded from the Mac's loopback interface. This setup does not use Cloudflare Access or a source computer allowlist. ## Prepared routes and recorded rollout status ![M5 worker routing and rollout status](../diagrams/2026-10-17-bri-397-m5-current.png) [Editable Excalidraw source](../diagrams/2026-10-17-bri-397-m5-current.excalidraw). Recorded on 2026-10-17: beta completed a 15 second M5 generation and its FLAC was downloaded and decoded. Missing, wrong, and opposite environment bearers were rejected; status requests bypassed cache. The production HTTPS canary passed authentication and generated audio, but its R2 upload failed TLS validation. The Mac received a certificate for `core1.netops.test` instead of the R2 host. The job was cancelled, the production gateway restored to `http://127.0.0.1:17311/jobs`, and production tunnel ingress disabled. Production M5 end to end verification has not passed. The same Mac egress problem may affect new beta uploads. Keep TLS validation enabled and retain SSH until production passes its own acceptance test. ## Baseline and authentication Read each environment's authenticated `GET /internal/admin/gpu-gateway`. Save the response outside Git with mode 0600. Record the matching runtime row and its `runtimes[].version`, plus the full fallback chain. Do not save bearer values or PM2 environment dumps in the rollout evidence. The verified 2026-10-17 baseline uses `HOME_WAKE_TOKEN` in both live rows, version 4. Beta points to `http://127.0.0.1:17312/jobs`; production points to `http://127.0.0.1:17311/jobs`. Each chain contains its M5 runtime then `modal-espetro`. Re-read before activation; these are historical values. Both private worker files under `~/.config/brioso/m5/` must contain `BRIOSO_WAKE_REQUIRE_TOKEN=1` and distinct, nonempty `HOME_WAKE_TOKEN` values. Verify each against the backend variable named by its live `push.tokenEnv`. The addon gateway examples use a different token reference; do not apply them over the live rows. The worker launcher sources the private env file after PM2 sets its environment, so check that file as well as the profile. Keep worker bindings at `127.0.0.1`, the model at `127.0.0.1:7309`, and both workers on the same `BRIOSO_ADMISSION_LOCK_PATH`. Before restarting a worker, verify it is idle via its local `/health` endpoint. ## Provision and start Install cloudflared through mise. Use an authenticated Cloudflare operator session for `usebrioso.com`, then create a new locally managed tunnel: ```bash mise exec -- cloudflared tunnel login mise exec -- cloudflared tunnel create brioso-m5 ``` Copy `deploy/cloudflared/m5.yml.example` to `~/.config/brioso/m5/cloudflared.yml`. Replace the tunnel UUID and absolute credential path with those emitted by creation. Keep the configuration and credential JSON mode 0600 and their directory mode 0700. Do not reuse the VPS tunnel token. The Mac connector metrics bind only to `127.0.0.1:7313`. The example includes both environments. For the existing tunnel, retain its UUID and credential path and add the production ingress from the example. ```bash mise exec -- cloudflared tunnel --config "$HOME/.config/brioso/m5/cloudflared.yml" ingress validate mise exec -- cloudflared tunnel --config "$HOME/.config/brioso/m5/cloudflared.yml" ingress rule https://m5-beta.usebrioso.com/jobs mise exec -- cloudflared tunnel --config "$HOME/.config/brioso/m5/cloudflared.yml" ingress rule https://m5-prod.usebrioso.com/jobs mise exec -- cloudflared tunnel --config "$HOME/.config/brioso/m5/cloudflared.yml" ingress rule https://m5-prod.usebrioso.com/health ``` Expect the matching worker origin for each `/jobs` URL and `http_status:404` for `/health`. Check existing DNS records before creating missing proxied CNAMEs; reuse records already pointing to this tunnel: ```bash mise exec -- cloudflared tunnel route dns brioso-m5 m5-beta.usebrioso.com mise exec -- cloudflared tunnel route dns brioso-m5 m5-prod.usebrioso.com ``` In the zone's Cache Rules, add a rule with expression `http.host in {"m5-beta.usebrioso.com" "m5-prod.usebrioso.com"}` and Cache eligibility **Bypass cache**. Place it after any overlapping cache rules. Preserve other zone rules and check that no Worker route overrides caching. This Cloudflare setting is required separately from tunnel ingress. Start just the new connector alongside SSH: ```bash mise exec -- node_modules/.bin/pm2 start deploy/pm2/ecosystem.m5-cloudflared.config.cjs mise exec -- node_modules/.bin/pm2 save ``` For an existing connector, restart only `brioso-m5-cloudflared` after validating the private configuration and confirming both workers are idle. Use mise for PM2 startup so `cloudflared` is on its process PATH. Confirm the existing PM2 login startup service restores the connector after reboot. The profile disables automatic binary updates; upgrade through mise. ## Verify and activate For the active HTTPS hostname, test both `POST /jobs` and `GET /jobs/test-call`. Missing, incorrect, and opposite environment bearer values must return 401. The matching bearer with an empty JSON POST must return 422 without starting work; a nonexistent call GET returns 200 with the worker's unknown state. Read credentials in memory and never place them in shell arguments or logs. Verify `/`, `/health`, `/wake`, `/v1/models`, `/docs`, `/jobs/`, and nested paths return 404. Repeat authenticated status requests and inspect cache headers; they must never be served as a cache HIT. Check the configured rule as well, since a cache MISS alone does not establish bypass. For beta, copy the captured runtime object and change only `push.url` to `https://m5-beta.usebrioso.com/jobs`. PUT that object to `/internal/admin/gpu-gateway/runtime/home-m5-beta` with `If-Match` set to the fresh row version. Authenticate with the existing admin credential. On 412, re-read and review the concurrent change; never retry without the guard. Re-read the complete gateway and confirm the only configuration difference is that URL. Preserve `tokenEnv`, runtime IDs, and fallback chains. For production, apply the same URL-only update with `https://m5-prod.usebrioso.com/jobs` and `/internal/admin/gpu-gateway/runtime/home-m5-production`, using its own fresh row version and existing token reference. The production gateway example contains the prepared HTTPS URL; do not import its whole registry or replace the live credential reference. Repeat the authentication and route checks above with the production bearer. Submit one beta generation through the application. Correlate the accepted call, HTTPS status polling, M5 runner, terminal paid state, and downloaded audio. A Modal completion does not pass this M5 check. Production uses the same setup, but a full generation and download remain unverified until a production acceptance run is possible. Configuration validation and HTTP checks do not establish that R2 uploads succeed. Production MCP may be disabled; do not enable it for this rollout. With both workers idle, restart `brioso-m5-cloudflared` and repeat beta HTTPS checks. For a future fallback check, stop the connector briefly during a controlled beta submission; verify that the existing chain completes through Modal, then restart the connector and verify both hosts recover. Avoid this interruption while either M5 worker has an accepted call. Verify both workers still share the same admission lock. Record job IDs and output checks; revoke test keys. Only after production also moves to HTTPS and passes its checks: ```bash mise exec -- node_modules/.bin/pm2 stop brioso-m5-tunnel mise exec -- node_modules/.bin/pm2 save ``` Keep the SSH profile for manual rollback. Starting the entire home profile again also starts SSH; select individual processes for routine restarts. ## Rollback Start `brioso-m5-tunnel` and verify both VPS loopback health endpoints. Read the current gateway rows and restore each saved original push URL with its current `If-Match` version. Preserve concurrent unrelated changes. Verify the restored URLs, token references, and chains, then stop `brioso-m5-cloudflared` and save PM2 state. Keep mandatory bearer auth. Never register SSH as an extra runtime, which could duplicate dispatch. References: [ingress configuration](https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/do-more-with-tunnels/local-management/configuration-file/), [cache settings](https://developers.cloudflare.com/cache/how-to/cache-rules/settings/). --- ## [MCP SQLite Server Runbook](https://fitxers.usebrioso.com/internal/mcp-sqlite-runbook/) (Internal) # MCP SQLite Server Runbook Read-only MCP server exposing the prod SQLite DB to AI agents over Tailscale. ## What this is A [Model Context Protocol](https://modelcontextprotocol.io) server that lets MCP-capable agents (Claude Code, Cursor, opencode, Claude Desktop via `mcp-remote`) run read-only SQL queries against the production database without direct DB access. Agents get `read_query`, `list_tables`, `describe_table` tools. Write tools (`write_query`, `create_table`, `drop_table`) exist in the tool surface but fail at runtime: the DB is opened with `readonly: true`. ## Architecture ``` MCP client (Claude Code, Cursor, etc.) │ HTTPS db.usebrioso.com/mcp (Tailscale, autogroup:member) + │ Authorization: Bearer ▼ Caddy (100.105.198.36:443) — checks the bearer header, 401s if missing/wrong │ reverse_proxy, Host normalized to localhost ▼ brioso-mcp-sqlite.service (node22 + better-sqlite3, localhost:4984) │ readonly ▼ /opt/brioso/shared/data/local.db (prod SQLite, WAL mode) ``` Three independent layers, in order: 1. **Tailscale ACL** (out-of-repo) — only tailnet clients can reach Caddy's shared IP:port at all (`autogroup:member` — see §Tailscale ACL setup). 2. **Caddy bearer-token gate** — `db.usebrioso.com/mcp` requires `Authorization: Bearer ` (see `deploy/caddy/Caddyfile` and `deploy/README.md`'s "Custom subdomains" section); this server has no auth of its own, so this is the actual access-control boundary for it. 3. **SQLite VFS** — `better-sqlite3` opens with `{ readonly: true, fileMustExist: true }`. All write attempts fail with `attempt to write a readonly database`. ## Why this fork and commit [`Sebasteuo/sqlite-mcp-server`](https://github.com/Sebasteuo/sqlite-mcp-server) is a fork of `etugrand/sqlite-mcp-server` that adds Streamable HTTP transport (the `--http` flag). HEAD of the fork switched from native `better-sqlite3` to `sql.js` (pure JS/WASM), which loads the entire DB into memory and bypasses the native VFS: `readonly: true` does not apply. We pin commit `d264a86` (2026-03-07), the last commit using native `better-sqlite3` before the WASM switch. To verify the pin rationale hasn't drifted, check the fork's recent commits: ```bash gh api repos/Sebasteuo/sqlite-mcp-server/commits --jq '.[0:3] | .[] | .sha[:7] + " " + .commit.message' ``` ## The read-only patch The upstream constructor opens writable: ```ts this.db = new Database(path); ``` The install script patches `src/index.ts` to: ```ts this.db = new Database(path, { readonly: true, fileMustExist: true }); ``` This is applied via `sed` after checkout (see `deploy/scripts/install-mcp-sqlite.sh`). If the install fails with `"expected 'new Database(path)' not found"`, upstream changed the constructor: re-review before updating the pin. The server has no `--readonly` flag and `better-sqlite3` does not honor `file:...?mode=ro` URIs, so a source patch is the only reliable enforcement point. The alternative (wrapping every tool handler) is more code and more surface area to get wrong. ## WAL mode precondition The prod DB must already be in WAL mode before the MCP server starts. `readonly: true` + WAL works because SQLite reads existing `-wal`/`-shm` files without writing. The constructor runs `PRAGMA journal_mode = WAL`: on an already-WAL readonly DB this is a no-op (returns `wal`, does not throw). Prod DB is already WAL (Drizzle Gateway requires it). If the DB is ever rebuilt from a non-WAL backup, switch it to WAL **before** starting the MCP service: ```bash sudo -u web-services sqlite3 /opt/brioso/shared/data/local.db 'PRAGMA journal_mode=WAL;' ``` ## PII hygiene The systemd unit sets `StandardOutput=null`. MCP query and response bodies contain row data (potentially PII): suppressing stdout keeps them out of journald. Startup messages and errors go to stderr, which still reaches the journal (`StandardError=journal`). ## Operations ### Install / update ```bash sudo /opt/brioso/repo/deploy/scripts/install-mcp-sqlite.sh sudo systemctl restart brioso-mcp-sqlite ``` The installer is idempotent: if the pinned commit is already installed and `build/index.js` exists, it skips. To force a rebuild: ```bash sudo rm /opt/brioso/shared/.mcp-sqlite-version sudo /opt/brioso/repo/deploy/scripts/install-mcp-sqlite.sh sudo systemctl restart brioso-mcp-sqlite ``` To pin a new commit: edit `VERSION` in `install-mcp-sqlite.sh`, delete the marker, re-run. **Always review the constructor line in the new commit before updating the pin** (the `sed` patch targets `new Database(path)` exactly). ### Verify read-only enforcement ```bash # 1. Service is running systemctl status brioso-mcp-sqlite # 2. Health endpoint responds curl -s http://localhost:4984/health # 3. Read works sudo -u web-services /usr/local/bin/node22 -e ' const Database = require("/opt/brioso/mcp-sqlite/node_modules/better-sqlite3"); const db = new Database("/opt/brioso/shared/data/local.db", { readonly: true, fileMustExist: true }); console.log(db.prepare("SELECT count(*) as n FROM sqlite_master").get()); ' # 4. Write fails (the security guarantee) sudo -u web-services /usr/local/bin/node22 -e ' const Database = require("/opt/brioso/mcp-sqlite/node_modules/better-sqlite3"); const db = new Database("/opt/brioso/shared/data/local.db", { readonly: true, fileMustExist: true }); try { db.exec("CREATE TABLE IF NOT EXISTS _ro_test (x)"); console.log("FAIL: write succeeded"); } catch (e) { console.log("OK: " + e.message); } ' # Expected: OK: attempt to write a readonly database ``` If step 4 ever prints `FAIL: write succeeded`, treat it as a security incident: the readonly patch is missing or ineffective. Re-run the installer. ### Custom subdomain (Caddy) MCP SQLite is reached in prod at `https://db.usebrioso.com/mcp` via the shared Caddy instance — see `deploy/README.md`'s "Custom subdomains (Caddy)" section for the full DNS/Caddy/ACL setup (shared with Drizzle Gateway on the same domain's `/` path, plus Arize Phoenix and `apps/admin` on their own subdomains). `MCP_SQLITE_API_KEY` itself lives in Infisical under `/infra/caddy` and syncs into `/etc/caddy/caddy-secrets.env` automatically on every deploy (restarting `brioso-caddy` only if it changed) — rotate it there, not by hand on the VPS. Verify: ```bash curl -s https://db.usebrioso.com/mcp -H "Authorization: Bearer " ``` ## Tailscale ACL setup (one-time, out-of-repo) Shared with Drizzle Gateway, Arize Phoenix, and `apps/admin` — all four sit behind the same Caddy instance on one IP:port (`100.105.198.36:443`), and Tailscale ACLs are IP:port-scoped, not per-hostname or per-path, so they can't be split per subdomain. The ACL is `autogroup:member`: ```json { "action": "accept", "src": ["autogroup:member"], "dst": ["tag:server:443"] } ``` This tailnet is now just the two engineer accounts, which is why `autogroup:member` (previously avoided here — see history below) is safe: unlike the old per-service `tailscale serve` setup, reaching `db.usebrioso.com:443` no longer implies prod DB query access by itself. The actual access-control boundary for MCP SQLite is now the Caddy-layer bearer token (`MCP_SQLITE_API_KEY`, see §Architecture and `deploy/README.md`) — the ACL only gates who can reach Caddy at all. (Historical note: an earlier user-scoped-only ACL model was used back when `:8002` was reached directly with no app-level auth in front of it; that constraint no longer applies now that the bearer-token gate exists.) ## Client configuration ### Claude Code / Cursor / opencode These clients support remote MCP URLs directly. Add to the MCP config: ```json { "mcpServers": { "brioso-sqlite": { "url": "https://db.usebrioso.com/mcp", "headers": { "Authorization": "Bearer " } } } } ``` ### Claude Desktop (requires mcp-remote bridge) Claude Desktop does not support remote MCP URLs natively. Use [`mcp-remote`](https://www.npmjs.com/package/mcp-remote): ```json { "mcpServers": { "brioso-sqlite": { "command": "npx", "args": [ "mcp-remote", "https://db.usebrioso.com/mcp", "--header", "Authorization: Bearer " ] } } } ``` ## Troubleshooting | Symptom | Check | |---|---| | Service won't start | `journalctl -u brioso-mcp-sqlite -n 50`. Common causes: node22 not at `/usr/local/bin/node22`, DB missing, DB not in WAL mode. | | `unable to open database file` | `SQLITE_DB_PATH` points to a nonexistent file, or the file is not a valid SQLite DB. | | Write tool succeeds | **Security incident.** Readonly patch missing or ineffective. Re-run installer, verify step 4 above. | | Clients can't connect | Querying device not on the tailnet, or Tailscale ACL not applied for `100.105.198.36:443`. | | 401 from `db.usebrioso.com/mcp` | Missing/stale `Authorization: Bearer ` header, or the key in `/etc/caddy/caddy-secrets.env` on the VPS doesn't match what the client sent — check Infisical (`/infra/caddy` folder, prod env) for the current value; if you just rotated it there, the next deploy syncs it in and restarts `brioso-caddy` automatically. | | `expected 'new Database(path)' not found` during install | Upstream changed the constructor. Review the new source before updating the pin. | | 404 on `/mcp` | Server not built with `--http` flag, or wrong commit checked out. Check `ExecStart` in the unit. | --- ## [Modal + Beam runbook (GPU gateway v2, BRI-359)](https://fitxers.usebrioso.com/internal/modal-beam-runbook/) (Internal) # Modal + Beam runbook (GPU gateway v2, BRI-359) > Retired in BRI-439: the Beam runtime kind and this runbook's weight-routing > are gone. The live Modal app is `apps/modal-worker` (see > `gpu-gateway-beta-runbook.md`). Kept for history. Operational guide for running the SA3 medium variant on Modal and Beam, alongside the existing RunPod path. Both new providers reuse the same container image and handler as RunPod (apps/runpod/Dockerfile.sa3 -> apps/runpod/handler.py), so payloads, upload-urls result posting, and obs are provider-agnostic. - Modal app: `apps/modal-sa3/modal_app.py` - Beam app: `apps/beam-sa3/beam_deployment.py` - CI: `.github/workflows/modal-deploy.yml`, `.github/workflows/beam-deploy.yml` ## One-time secret setup CI (GitHub repo secrets, before first deploy run): - `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET`: Modal workspace -> API tokens, scoped to deploy the `brioso-sa3` app. - `BEAM_TOKEN`: Beam dashboard -> API keys. Platform secret stores (mirror of the RunPod endpoint env): - Modal: `modal secret create brioso-sa3 JOBSTORE_API_URL=... WORKER_SHARED_SECRET=...` - Beam: `beam secret create JOBSTORE_API_URL=...` and `beam secret create WORKER_SHARED_SECRET=...` Values come from Infisical, same entries the RunPod endpoints use. Leave `SA3_MODEL_ID` unset so the handler defaults to the medium variant. ## Deploy 1. Ensure the base image tag exists: the runpod image workflow (`build-runpod.yml`) must have pushed the tag referenced by `SA3_IMAGE_TAG` (repo variable, defaults to `v8`). Bump in both places together so RunPod, Modal, and Beam stay in lockstep. 2. Trigger `deploy-modal-sa3` (or `deploy-beam-sa3`) via Actions -> Run workflow, typing `yes` in the confirm box. Also available via `repository_dispatch` for automation. Never on push: deploys pay third-party compute, and the CI runner policy (ci-runner-policy.md) plus the cost guardrails below rely on that gate. 3. Verify in the platform dashboard that the app deployed and one manual smoke job completes (track URLs land in the job row). Local deploy alternative (from a trusted machine): ```sh uv tool run modal deploy apps/modal-sa3/modal_app.py uv tool run beam deploy apps/beam-sa3/beam_deployment.py ``` ## Pointing traffic at a provider Routing lives in the `inference_routes` table (BRI-359 phase 2): each variant has provider rows with a weight and an enabled flag, resolved by the cached resolver in packages/dispatch. To send jobs to Modal: - Set the medium variant's modal row weight (e.g. runpod 90, modal 10) and enabled = true. - Watch the weight-drift obs metric and job outcomes before raising the weight. - Beam works the same way via its own row. Shadow-test before weighting: dispatch a single seeded job with the route temporarily pinned to modal/beam (weight 100), confirm track upload works, then return to blended weights. ## Rollback Set the provider row's enabled flag to false (or weight 0) in `inference_routes`; the resolver stops selecting it within the cache TTL (30s by default) and new jobs route elsewhere. In-flight jobs keep polling: a composing job leased as `modal:`/`beam:` is completed by the reaper poll backstop, so draining needs no redeploy. A submit that keeps failing accumulates attempts like any runner and flips to failed at the queue's terminal threshold, unless the operator configures a fallback chain entry (e.g. `JOBS_RUNNER_FALLBACK_CHAIN=modal>runpod`) to promote instead. If the app itself is bad, also stop the platform deployment (Modal dashboard -> stop app; Beam -> deactivate the deployment) to avoid paying idle GPU time. ## Cost guardrails - Both apps run min_containers=0 / no minimum workers: idle time costs nothing; you pay per job plus brief container idle windows. - GPU classes are deliberately modest (Modal L40S, Beam L4). Changes to them require a review of expected monthly spend, same as the RunPod scaler work (BRI-359 phase 6). - Weights are the main leak: a misconfigured modal 100 route burns budget. Keep enabled toggles false for any provider not under active evaluation, and watch the chosen-vs-configured weight-drift obs metric. - Deploy workflows gate on a typed `yes` confirmation; never relax that to push triggers. --- ## [Spaces happy path: perf percentiles (phase span map)](https://fitxers.usebrioso.com/internal/observability/perf-percentiles/) (Internal) # Spaces happy path: perf percentiles (phase span map) Status: PR 1 (`feat/bri-337/perf-span-coverage`, merged to beta) adds the span/event coverage. This doc (PR 2) holds the HogQL recipes and the insight checklist. All 10 insights are created in PostHog (tag `performance/spaces happy path`) with short_ids recorded below; they return empty until PR 1's span coverage has ~24h of data on the events listed. Linear: BRI-337. ## Phase -> span/event map Every phase emits a Phoenix span (when the work happens inline) and a PostHog event `${name}.completed` with `duration_ms`, both via the obs facade (`withSpan` / `with_span` / `event`). Never raw OTel/PostHog calls. When comparing environments in an insight, filter PostHog events on `deployment.environment` (beta vs prod; `prod` is the default when `DEPLOYMENT_ENV` is unset). | Phase | Span/event name | Where | Notes | |---|---|---|---| | space.create | `space.create` | apps/web spaces POST route | wraps the create | | compose.dispatch | `compose.dispatch` | packages/dispatch sweep.ts `dispatchJobBody` | `dispatch.runpod` nests inside | | queue.wait | `queue.wait.completed` (event only) | packages/jobs completion.ts | no span: the wait already happened; `duration_ms = queueWaitS * 1000`; Phoenix already has the `runpod.queue_wait_s` attribute on the completion server span | | runner.generate | `runner.generate` | runpod handler + worker main | inner span around SA3 wall time, inside `runpod.handler` / `worker.process_job` | | runner.encode | `runner.encode` (runpod) / `runner.encode_and_upload` (home) | both runners | split out of the former bundled encode_and_upload on runpod (handler.py:457); the home worker keeps one combined `runner.encode_and_upload` span because encode+upload run in a shared thread pool inside inference_core (worker main.py:387), so home has no separate `runner.upload` event | | runner.upload | `runner.upload` | both runners | split out of the former bundled encode_and_upload | | delivery.url_signed | `delivery.url_signed` | packages/jobs delivery.ts `downloadFileUrl` | `resign: true` attribute on re-signs; insights filter `resign != true` | ## Property conventions All phase spans/events carry: - `runner`: `"home"` or `"runpod"` (which runner executed the job) - `slug`: job slug (maps to Phoenix span `session.id`) - `trace_id`: present in PostHog event props when `ctx.traceId` is set (via obs-js `contextProperties`); enables Phoenix deep links `phoenix://projects/brioso/traces/` - `track_count`: number of tracks in the job - `duration_s`: requested audio duration, where meaningful - `duration_ms`: always present on `${name}.completed` PostHog events (emitted by `withSpan`/`with_span` on success) Runner phase spans additionally carry `audio.duration_s` and `track.count` as span attributes. ## PostHog insights (PR 2) Project: https://eu.posthog.com/project/219634 (insight URLs below follow the `/insights/` pattern used by the existing KPI docs). There is no in-repo script that creates insights via the PostHog API today; the closest tooling is `posthog-cli` (see `.agents/notes/2026-08-14-posthog-cli.md`), used for manual UI/CLI work with `--host https://eu.posthog.com`. Insights are therefore created in the PostHog UI (HogQL mode) and only their short_ids are recorded here. ### Recipe A: per-phase percentiles One insight per phase event. HogQL query (replace the event name per phase): ```sql SELECT properties.runner AS runner, quantiles(0.5, 0.75, 0.9, 0.95)(properties.duration_ms) AS p50_p75_p90_p95_ms FROM events WHERE event = 'runner.generate.completed' AND timestamp > now() - interval 24 hour GROUP BY runner ORDER BY runner ``` Phase events to cover: `space.create.completed`, `compose.dispatch.completed`, `queue.wait.completed`, `runner.generate.completed`, `runner.encode.completed` (runpod only), `runner.upload.completed` (runpod only), `runner.encode_and_upload.completed` (home only), `delivery.url_signed.completed`. Extra breakdowns (swap `properties.runner` for, or add as extra groups): - `properties.track_count`: does parallelism hurt encode/upload? - `properties.duration_s`: does requested audio length drive generate time? PostHog UI breakdowns cap at 25 values per breakdown in trends; using HogQL `GROUP BY` in a SQL insight avoids the limit entirely, so prefer the SQL insight type for these. ### Recipe B: end-to-end runner time The runner root span duration is the E2E runner time (generate + encode + upload inside one process). `runpod.handler.completed` covers the RunPod path, `worker.process_job.completed` the home path; combine both in one insight: ```sql SELECT properties.runner AS runner, quantiles(0.5, 0.75, 0.9, 0.95)(properties.duration_ms) AS e2e_runner_ms FROM events WHERE event IN ('runpod.handler.completed', 'worker.process_job.completed') AND timestamp > now() - interval 24 hour GROUP BY runner ORDER BY runner ``` Queue wait is the missing upstream leg (RunPod path only, per completion.ts:286). Read it alongside the E2E number: ```sql SELECT quantiles(0.5, 0.75, 0.9, 0.95)(properties.duration_ms) AS queue_wait_ms FROM events WHERE event = 'queue.wait.completed' AND timestamp > now() - interval 24 hour ``` True user-perceived E2E (space create through signed URL) would need a cross-event sum per trace; the two insights above are the practical proxy: `space.create + compose.dispatch + queue.wait + runner root + delivery`. ### Recipe C: slow contributors top 10 One insight per slow phase (start with `runner.generate.completed` and `queue.wait.completed`). SQL insight type: ```sql SELECT distinct_id, properties.slug AS slug, properties.trace_id AS trace_id, properties.duration_ms AS duration_ms, timestamp FROM events WHERE event = 'runner.generate.completed' AND timestamp > now() - interval 24 hour ORDER BY duration_ms DESC LIMIT 10 ``` Deep-link the worst offenders into Phoenix: `phoenix://projects/brioso/traces/` (paste the trace_id into the Phoenix trace search; the URL form is the convention used in the runbooks). Note: `delivery.url_signed.completed` has no first-sign marker in PR 1 (every sign fires, see tracks.ts:254), so for that phase either accept that repeat downloads pollute the top-10 or filter `duration_ms > 1000` to surface only meaningfully slow signs. ### Insight checklist Folder: "Performance / Spaces happy path" (create it if missing). Create each insight with the recipe above, name it as suggested, and record the short_id here during execution. | # | Insight name | Recipe | Event(s) | short_id | |---|---|---|---|---| | 1 | perf.space.create.percentiles | A | space.create.completed | cORE8P9d | | 2 | perf.compose.dispatch.percentiles | A | compose.dispatch.completed | HEPH8xTl | | 3 | perf.queue.wait.percentiles | A + B | queue.wait.completed | RnlSpS0e | | 4 | perf.runner.generate.percentiles | A | runner.generate.completed | 65Zl30oa | | 5 | perf.runner.encode.percentiles | A | runner.encode.completed + runner.encode_and_upload.completed | kDcpwCB9 | | 6 | perf.runner.upload.percentiles | A | runner.upload.completed | JxALWFmv | | 7 | perf.delivery.url_signed.percentiles | A | delivery.url_signed.completed | B2AwZkUj | | 8 | perf.spaces.happy_path (E2E runner) | B | runpod.handler.completed, worker.process_job.completed | 7rYrJPDS | | 9 | perf.slow_contributors.generate | C | runner.generate.completed | egAoKuxb | | 10 | perf.slow_contributors.queue_wait | C | queue.wait.completed | DC8rOarD | ## Reading the bottleneck Compare each phase's p95 against its p50: a wide gap means a tail problem (capacity, cold start, one slow shard), a uniformly high p50 means a constant cost (model, egress bandwidth). - queue.wait high -> RunPod cold start / capacity (check whether p95 spikes cluster at job bursts; cross-reference `runpod.handler` cold starts) - runner.generate high -> model/engine (SA3 wall time; compare `audio.duration_s` vs `duration_ms`) - runner.encode high -> ffmpeg/encode workers (scales with `track_count`) - runner.upload high -> R2 egress (scales with `track_count`; on the home runner read `runner.encode_and_upload` minus `runner.generate` as the combined encode+upload proxy) - space.create / compose.dispatch / delivery.url_signed should be milliseconds; if not, the web app or R2 presign is the problem ## AC deviations (record on BRI-337) - "Aggregation dashboard loads in < 2s": moot. There is no in-repo dashboard page; the dashboard IS PostHog, so this AC collapses into PostHog's own insight load performance. - "Percentiles for 24h window within 5s": reinterpret as PostHog insight load time. Verify once (after the first 24h of PR 1 data) by timing each insight in the checklist; if any exceed 5s, note it rather than optimizing. - Encode/upload split exists only on the RunPod runner (handler.py:457,466). The home runner keeps a combined `runner.encode_and_upload` span (worker main.py:387) because encode and upload run concurrently in a shared thread pool inside inference_core, so insights 5 and 6 show home data only as the combined proxy. - `queue.wait.completed` fires only on the RunPod completion path (completion.ts:286): the home runner records no claim latency. --- ## [AGENTS.md — docs/papers](https://fitxers.usebrioso.com/internal/papers/AGENTS/) (Internal) # AGENTS.md — docs/papers Every paper here gets two files: the source `.pdf` (kept for humans to read) and a generated `.md` (kept so agents can grep/read it without OCR/PDF-parsing detours). ## Adding a paper 1. Save the PDF as `.pdf` in this directory. 2. Generate the markdown via the [arxiv2md](https://github.com/timf34/arxiv2md) API (no auth, 30 req/min): ```bash curl -s "https://arxiv2md.org/api/markdown?url=&remove_refs=false&remove_toc=false&remove_citations=false" \ -o .md ``` `` is the numeric arXiv ID (e.g. `2605.17991`), with or without version suffix. 3. Add an entry to `README.md` linking the PDF (human-facing). ## Why arxiv2md, not a local PDF converter arxiv2md parses arXiv's structured HTML (not the PDF), so output has real `##` headings, a TOC, proper `|` tables, and LaTeX math (`$L_{\max}$`) instead of mangled Unicode. Tried and rejected for this repo: - **`markitdown` on the PDF directly** — reversed/garbled watermark text, broken tables, unreadable in places. - **`markitdown` on the arXiv HTML page** — better than the PDF route, but bloats every inline citation with the full paper title as hover text (`[26](#bib.bib7 "Denoising diffusion probabilistic models")` repeated hundreds of times), and mangles inline math (`4096×\times`). - **markxiv.org** — despite serving `Content-Type: text/markdown`, output has zero markdown syntax: no headings, no tables, no links, just page-numbered PDF text. Only use PDF-parsing as a fallback for papers with no arXiv HTML rendering (very old papers, non-arXiv sources). Keep `remove_refs=false&remove_toc=false&remove_citations=false` — we want the full reference list and TOC preserved for agent lookups; the API's defaults strip all three. --- ## [Papers](https://fitxers.usebrioso.com/internal/papers/) (Internal) # Papers Reference papers relevant to Brioso. - [Stable Audio 3](stable-audio-3.pdf) ([md](stable-audio-3.md)) — Evans et al., Stability AI technical report, arXiv:2605.17991 (May 2026). Family of latent diffusion models (small/medium/large) for variable-length audio generation and editing, with inpainting, a semantic-acoustic autoencoder, and adversarial post-training for fast inference. `small` and `medium` weights are open, along with training and inference code. See [AGENTS.md](AGENTS.md) for how the `.md` companions are generated. --- ## [Untitled](https://fitxers.usebrioso.com/internal/papers/stable-audio-3/) (Internal) ## Contents - 1 Introduction - 1.1 Related Work - Open models. - Variable length. - Semantic latent spaces. - Controllability. - Few-step generation. - 2 Architecture - 2.1 Semantic-Acoustic Autoencoder - 2.2 Diffusion Transformer - Transformer blocks. - Adaptive layer normalisation (AdaLN): diffusion timestep and duration conditioning. - Cross-attention: text and duration conditioning. - Local-additive conditioning for inpainting. - Differential attention. - 3 Training - 3.1 Variable-Length Training - Variable-length attention and masked loss. - Per-element timestep shifts. - Silence augmentation. - 3.2 Flow Matching Pre-Training - Inpainting training. - Minibatch optimal transport coupling. - Timestep sampling. - 3.3 Distillation Warmup - 3.4 Adversarial Post-Training - Generator architecture. - Discriminator architecture. - Adversarial relativistic loss. - Contrastive loss. - CLAP loss. - 3.5 Training Implementation Details - Training data. - Prompt preparation. - Flow matching pre-training. - Discriminator architecture. - Timestep sampling. - Optimizer. - Exponential Moving Average (EMA). - 4 Inference - 5 Discussion - 5.1 Methodology - 5.2 Instrumental music generation - 5.3 Sound effects generation - 5.4 Variable-length instrumental music generation - 5.5 Variable-length sound effects generation - 5.6 Audio editing capabilities - 5.7 Adversarial Post-Training discussion - 5.8 VRAM Memory Usage - 5.9 Inference Times Across Hardware Platforms - Implementation details - 6 Conclusion - References ## Abstract Abstract Stable Audio 3 is a family of fast latent diffusion models ( small , medium , large ) for variable-length audio generation and editing. Since our models can generate several minutes of audio, variable-length generations are key to avoid the cost of producing full-length generations for short sounds. We also support inpainting, enabling targeted audio editing and the continuation of short recordings. Our latent diffusion models operate on top of a novel semantic-acoustic autoencoder that projects audio into a compact latent space, enabling efficient diffusion-based generation while preserving audio fidelity and encouraging semantic structure in the latent. Finally, we run adversarial post-training to both accelerate inference and improve generation quality, reducing the number of inference steps while improving fidelity and prompt adherence. Stable Audio 3 models are trained on licensed and Creative Commons data to generate music and sounds in less than a 2s on an H200 GPU and less than a few seconds on a MacBook Pro M4. We release the weights of small and medium , that can run on consumer-grade hardware, together with their training and inference pipeline. https://github.com/Stability-AI/stable-audio-tools http://github.com/Stability-AI/stable-audio-3 ## 1 Introduction Recent progress in music and audio generation has been driven by two broad families of models: autoregressive models [8, 1, 94, 91] and latent diffusion models [14, 15, 50, 6, 75, 52]. Autoregressive models have achieved strong results by operating sequentially on discrete audio tokens. In contrast, latent diffusion models generate continuous latent representations that are subsequently decoded with a separate autoencoder, offering an alternative that avoids discrete tokenisation and autoregressive sampling. Complementing these approaches, hybrid methods have been proposed using an autoregressive model to produce tokens that are then refined by a diffusion model [20, 95]. Stable Audio 3 consists of three latent diffusion models at different scales (small, medium, large, see Table [1](#S0.T1)). Variable-length generation is a key capability of Stable Audio 3, particularly because our models generate very long audio (Table [1](#S0.T1)). While autoregressive models naturally support variable-length outputs due to their sequential nature, diffusion models typically require generating the entire audio length at once [14, 15] (Figure [2](#S1.F2): a). This means that, e.g., generating a short sample with small-music would require producing a 2m audio with mostly silence. To address this compute and memory inefficiency, Stable Audio 3 supports variable-length generation (Figure [2](#S1.F2): b), enabling efficient synthesis without incurring full-length computation for short outputs. Such efficiency gains are critical for deploying open-weight models on consumer-grade hardware, where compute and memory budgets are constrained. Figure: Figure 2: Fixed- vs. variable-length generation. (a) Fixed-length generation allocates $L_{\max}$ embeddings regardless of the requested duration $d$, wasting computation on zero-padded silence for short clips. (b) Variable-length generation allocates $L$ embeddings that are proportional to the requested duration $d$ (silence padding is also used, see Section [3.1](#S3.SS1)). Controllability is also an important feature of modern generative audio and music models. Stable Audio 3 includes inpainting capabilities that allow editing targeted segments of audio, such as modifying a single segment (Figure [3](#S1.F3): first row), performing multi-segment edits (Figure [3](#S1.F3): second row), or supporting continuation (Figure [3](#S1.F3), third row), where the model can extend a given audio coherently beyond its original endpoint. This enables applications such as transient editing in percussive sounds, generating ideas for an unfinished song, or the extension of short recordings. Figure: Figure 3: Editing with inpainting. Users provide audio and specify target segments for editing (gray, masked) while preserving original audio (hatched). This enables tasks from single or multi-segment editing to causal continuation. Stable Audio 3 comprises latent diffusion models built on top of a semantic-acoustic autoencoder. This latent representation is designed to preserve reconstruction fidelity while remaining generatively tractable and semantically structured for downstream use. Our aim is to maintain high-fidelity audio reconstruction while learning a compact latent space (with 4096$\times$ downsampling) that is both easy to model generatively with diffusion and structured in a semantically meaningful way. Specifically, acoustic fidelity is enforced using spectral reconstruction losses and adversarial training [12, 41], while semantic structure is induced through latent-space regression objectives, including chroma and interaural level difference regression. One important characterisic of the employed autoencoder is its $4096\times$ downsampling ratio, substantially higher than the $1024$ to $2048\times$ ratios common in prior work [12, 41, 85]. This aggressive downsampling is central to our goals: it reduces sequence lengths enough for medium and small to generate long-form music and sound effects on consumer-grade GPUs and on a MacBook Pro using CPU. Diffusion models typically require several inference steps to generate high-quality outputs, as they progressively refine noise through iterative denoising [26, 79]. Yet, fast inference is essential for responsive creative tools to feel engaging and inspiring. To address this, we use adversarial post-training, which allows reducing the number of sampling steps while maintaining (or improving) output quality [60]. Overall, our latent diffusion training pipeline consists of three stages: flow matching pre-training [54, 48, 2], ODE warmup distillation [72, 56], and adversarial post-training [60]. Stable Audio 3 is designed for broad community adoption as it is trained on licensed and Creative Commons data, enabling artists and developers to use it without legal concerns. Further, our models can scale from datacenter GPUs (e.g., H200) down to consumer-grade GPUs and even a MacBook Pro. The main contributions of Stable Audio 3 are: - • Release the weights for small and medium, suitable to run on consumer-grade hardware (Table [1](#S0.T1)). - • State-of-the-art results for text-to-audio generation for instrumental music and sounds (Section [5](#S5)). - • Fast inference: less than 2s to generate up to 6m 20s on an H200 (Sections [5.2](#S5.SS2) and [5.3](#S5.SS3)). - • Audio editing via inpainting, including single- and multi-segment edits and continuation (Section [5.6](#S5.SS6)). - • Propose a new method for variable-length audio generation with latent diffusion models (Section [3.1](#S3.SS1)). - • Several technical innovations: a semantic-acoustic autoencoder that learns a compact latent for diffusion by preserving high-fidelity reconstruction and semantic information (Section [2.1](#S2.SS1)); the use of Transformer Resampling Blocks (TRBs, Section [2.1](#S2.SS1)) for down/up-sampling; a diffusion transformer improved with differential attention [92], adaptive layer normalization conditioning [67], and memory embeddings [3, 11] (Section [5](#S2.F5)); minibatch optimal transport coupling for flow matching training (Section [3.2](#S3.SS2)); and a distillation warmup stage followed by adversarial post-training for improved few-step generation (Sections [3.3](#S3.SS3) and [3.4](#S3.SS4)). ### 1.1 Related Work #### Open models. Early open models were predominantly based on either autoregressive approaches [8, 40] or latent diffusion methods [50, 6, 52, 19, 57, 16, 59]. More recent open models continue to explore autoregressive approaches [94, 91], while also introducing flow matching methods [60, 30, 53] and hybrid architectures that combine autoregressive modeling with flow matching [95, 20, 30]. These models have been applied to a range of audio generation tasks, including instrumental music [6, 8, 52], sound effects [50, 52, 60, 40, 19, 16], and songs with vocals [94, 91, 20, 95, 59, 53, 30]. Stable Audio 3 is a family of open-weight models (small, medium) based on flow matching for instrumental music and sound effects generation. For evaluation, we compare against the most competitive open models available. #### Variable length. Autoregressive models naturally support variable-length generation by producing tokens sequentially until an end-of-sequence token is produced, making variable length generation an emergent property. In contrast, latent diffusion models are typically defined over fixed-length sequences, requiring shorter inputs to be padded [15, 14]. This ties inference cost to a predefined maximum length rather than the actual content, leading to inefficiencies and limiting its practical scalability to long-form generation. A similar issue has been addressed in image diffusion: early models [68] relied on resolution conditioning and cropping to handle varying sizes, whereas modern transformer-based approaches rely on positional encodings to digest inputs of various sizes organically [4, 46]. Audio diffusion is beginning to follow this shift with approaches like autoregressive block-wise diffusion [30]. Yet, fully native variable-length audio generation with diffusion remains largely unaddressed. To our knowledge, Stable Audio 3 models are the first to tackle this challenge in a manner analogous to recent advances in image diffusion. #### Semantic latent spaces. Most latent diffusion models operate on low-dimensional (64, 32) latents from VAEs trained focusing on acoustic reconstruction [40, 16]. Representation autoencoders (RAE) [96, 82] have shown that diffusion in higher-dimensional, semantically structured latent spaces yields faster convergence and better generation quality in the image domain. To our knowledge, Stable Audio 3 models are the first to explore this idea in the audio domain by relying on the Semantically-Aligned Music autoEncoder (SAME) [65], which produces 256-dim latents designed to encode both acoustic fidelity and high-level semantic structure at a high downsampling ratio (4096$\times$). #### Controllability. The demand for controllable audio generation is increasing as creative workflows require control beyond prompts [69]. Prior work can be categorized as follows: mask-based, instruction-based, inference-time control, global control, time-varying control, and lyrics editing. Mask-based methods enable localized editing or continuation by generating the masked segments of a given audio [18, 45, 76]. Instruction-based approaches support operations such as adding, removing, processing, or replacing sound sources through structured commands [86, 24, 66]. Inference-time controls include guidance-based and inversion methods [60, 44, 64, 43, 61, 62]. Global conditioning methods generate audio based on a reference signal [81, 71] while time-varying controls introduce temporally dynamic constraints [87, 17, 84, 37]. Lyrics editing allows additional control by modifying textual content [91, 20, 53]. Stable Audio 3 focuses on mask-based editing, as it does not require additional training data annotation. Training instead relies on simple random and causal masking. We do not consider instruction-based approaches, which typically require training datasets with stems, nor lyrics editing, which lies outside the scope of our work. We also exclude inference-time, global, and time-varying controls, as these often rely on model fine-tuning (LoRA [37, 84]) or auxiliary models (ControlNet [87]). Note that such controls can be included by fine-tuning Stable Audio 3 after its release. #### Few-step generation. The iterative denoising process of diffusion incurs high inference latency, motivating few-step generation methods. Reducing the number of sampling steps in diffusion can be achieved through distillation [72, 78] or adversarial approaches [88, 74]. In (step) distillation approaches the teacher provides direct supervision to train a distilled few-step generator that learns to map multiple inference steps into a single step, or a small number of steps, by distilling the teacher’s trajectories. However, most distillation approaches come with practical drawbacks like online methods [70, 83, 78, 55, 5, 36, 63, 89, 93], which are costly to train as they require 2-3 full models held in memory, or offline methods [54, 72, 33], which require significant resources to generate and store trajectories to later train on. To avoid such drawbacks, some explored adversarial post-training (without distillation) [90, 47]. These works are primarily adversarial, as opposed to distillation methods that use adversarial auxiliary losses [93, 63, 36, 73, 74], and use real data rather than teacher-generated samples, thus freeing the costly requirement of using trajectories. The adversarial loss encourages realism, making each estimate better than the standard distilled estimates. Such improved estimates enable post-trained models to use fewer sampling steps [90, 47]. In the audio domain: AudioLCM [51] used latent consistency distillation, Presto [63] combined step and layer distillation, ARC [60] combined relativistic and contrastive adversarial losses, and Woosh used MeanFlow distillation [23]. Stable Audio 3 is based on ARC adversarial post-training but also uses distillation as a warmup. ## 2 Architecture Stable Audio 3 consists of two components: a semantic-acoustic autoencoder that maps waveforms to and from a continuous latent space; and a diffusion transformer generating latent sequences that is conditioned on text prompts, duration information, and inpainting masks. Figure [4](#S2.F4) depicts the overall system. Training details are in Section [3](#S3) and further implementation details are available online via our code release. Figure: Figure 4: Stereo audio at 44.1 kHz is encoded to a 256-dim latent sequence by a SAME autoencoder (4096$\times$ downsampling). A diffusion transformer generates latent sequences conditioned on text embeddings from T5Gemma (via cross-attention), a duration embedding (via both cross-attention and AdaLN), and a diffusion timestep $t$ (via AdaLN). Inpainting conditioning (masked input of 256$\times$L is concatenated with a binary mask, resulting in a size of $257\times L$) is projected to the model dimension and added at each transformer block. SAME decoder reconstructs the latents. **Table 2: Stable Audio 3 models where $d$, $D$, and $H$ denote transformer hyperparameters: latent dimensionality, number of transformer blocks, and number of attention heads, respectively. small uses no differential attention. Parameter counts are for the diffusion transformer only. SAME-S and SAME-L have 108M and 852M parameters, respectively.** | | | | | differential | SAME | maximum | # learnable | | --- | --- | --- | --- | --- | --- | --- | --- | | | $d$ | $D$ | $H$ | attention | autoencoder | length | parameters | | small | 1024 | 20 | 16 | — | SAME-S | 2m | 459M | | medium | 1536 | 24 | 24 | ✓ | SAME-L | 6m 20s | 1.4B | | large | 2048 | 26 | 32 | ✓ | SAME-L | 6m 20s | 2.7B | ### 2.1 Semantic-Acoustic Autoencoder Figure: Figure 5: SAME autoencoder [65]. Stereo audio is reshaped into patch embededdings ($256\!\times$ downsampling), downsampled by an encoder TRB (further $16\!\times$), and passed through a soft-normalisation bottleneck with projection to latent dimension $d$. Latents are then reconstructed by a decoder TRB and unpatching. Total downsampling: $4096\!\times$. Our autoencoder builds on SAME [65], a transformer-based autoencoder for audio that combines an initial patching stage with a Transformer Resampling Block (TRB, Figure [6](#S2.F6)). Patching reshapes stereo audio into non-overlapping patches of 256 samples (per channel, resulting in $256\times$ downsampling). TRB layers perform an additional $16\times$ downsampling by interleaving learnable output embeddings with the input sequence, and processing the resulting sequence with a stack of transformer layers using differential attention [92] and rotary position embeddings [80]. Figure: Figure 6: Example of TRB using embedding interleaving for 2$\times$ downsampling. Inputs are segmented into groups of 2 and a learnable output embedding is appended to each segment. The full interleaved sequence is processed by $D$ transformer layers $\mathcal{T}_{D}$. Output embeddings $y$ are extracted (discard $x$) to form the downsampled representation. The combined compression ratio is $4096\times$, yielding 256-dimensional latent embeddings at approximately 10.76 Hz for 44.1 kHz stereo input. Between encoder and decoder, a soft-normalisation bottleneck constrains the scale of the latent by using a learnable affine transform with running standard deviation tracking, providing a deterministic encoding. For upsampling, the TRB process is reversed: each input embedding is paired with a number of output embeddings that are then extracted after processing. For example, to upsample 2$\times$, we interleave two output embeddings with each input embedding to retain the output embeddings after processing and discard the original input embeddings. The SAME autoencoder is trained with a combination of (i) spectral reconstruction, (ii) adversarial, (iii) diffusion alignment, (iv) semantic regression, and (v) contrastive latent alignment losses that are designed to preserve reconstruction fidelity while remaining generatively tractable and semantically structured for downstream use. More specifically, SAME uses a multi-resolution STFT loss computed at seven resolutions (FFT sizes from 32 to 2048, each with 75% overlap). A K-weighting pre-emphasis filter is applied before the STFT. At each resolution, the loss combines a spectral contrast term, a modified log-magnitude L1 distance, and an instantaneous frequency + group delay (IFGD) phase loss [65]. To handle stereo audio, the STFT loss is computed independently on both the sum-and-difference (mid/side) and per-channel (left/right) rep-resentations. Furthermore, the adversarial loss is formulated using a relativistic GAN objective. Then, the diffusion alignment loss consists of a small diffusion transformer (4 layers, 768-dimensional embeddings) that is trained jointly on the autoencoder’s latent space using a flow matching objective such that gradients flow back through the encoder, encouraging the latent geometry to be amenable to diffusion-based generation. SAME semantic regression losses include two lightweight linear regressors (single 1$\times$1 convolutions) to predict chroma and interaural level difference (ILD) features. Finally, the contrastive latent alignment loss employs a transformer-based critic (4 layers, 1024-dimensional) that is trained to distinguish whether the latent sequence, wavelet (audio) features, and a T5Gemma text embedding (triplet) originate from the same input, encouraging the latent to preserve audio-level and cross-modal semantics. As a result, these losses focus on both high-quality acoustic reconstruction (spectral and adversarial losses) and semantic structure (semantic regression and contrastive latent alignment losses) for downstream diffusion (diffusion alignment loss). The SAME autoencoder is frozen during diffusion training. small uses SAME-S, a distilled variant with fewer parameters (108M) designed for CPU inference, while medium and large use SAME-L (852M parameters). Both variants share the same compression ratio and latent dimensionality. Further details are in the original SAME publication [65]. ### 2.2 Diffusion Transformer Our generative model is a diffusion transformer operating on SAME latents [67]. Transformers replaced U-Nets for latent diffusion [4], and Stable Audio 3 adapts the diffusion transformer for text-to-audio with modifications including editing capabilities with inpainting, differential attention [92], memory embeddings [3], and variable-length support. Figure: Figure 7: Diffusion transformer architecture. SAME latents are linearly projected from $256$ to $d$ channels. A set of $64$ memory embeddings is prepended, providing a global memory buffer that every position can attend to. The resulting sequence is processed by $D$ transformer blocks with latent dimensionality $d$. After the final block, memory embeddings are discarded and the sequence is projected back to $256$ channels. In/out 1$\times$1 convolutions are omitted. SAME latents first pass through a 1$\times$1 convolution with a residual connection. A linear projection then maps SAME frames to the transformer’s latent dimensionality ($256\to d$). Before entering the transformer, 64 learned memory embeddings are prepended. These embeddings serve as context that every position can attend to, effectively providing a global memory buffer. The resulting sequence is processed by a stack of $D$ transformer blocks with latent dimensionality $d$ and $H$ heads. After the final block, the memory embeddings are removed, and the sequence is projected back to the 256 dimensions of SAME. A final 1$\times$1 convolution with a residual connection produces the final output. Conditioning information enters the transformer through three distinct pathways (Figure [8](#S2.F8)). First, the diffusion timestep and duration (length of generation) are mapped to a global embedding that modulates each self-attention and feed-forward layers in each transformer block with adaptive layer normalization (AdaLN). Second, text embeddings from a frozen T5Gemma encoder, concatenated with a duration embedding, employ cross-attention for conditioning. Third, for inpainting, a local-additive conditioning signal with the reference audio (to inpaint) and a binary mask (signaling where to inpaint) is projected through an MLP and added to the hidden state of each transformer block. We train 3 models that share the same design but differ in transformer capacity, maximum generation length, and autoencoder (Table [2](#S2.T2)). medium and large use differential attention [92] in both self-attention and cross-attention layers, which roughly doubles the Q and K projection sizes relative to standard multi-head attention that small uses. #### Transformer blocks. Each transformer block is composed of self-attention, cross-attention, local-additive conditioning, and a feed-forward network (Figure [8](#S2.F8)). Figure: Figure 8: High-level (left) and detailed (right) overview of a single transformer block. $\sigma$ denotes the sigmoid function. Adaptive layer normalization (AdaLN with gate, scale, and shift) is used for diffusion timestep and duration conditioning, cross-attention for text and duration conditioning, and local-additive conditioning for inpainting. Self-attention follows a pre-norm design with AdaLN [67, 4, 23]. The input is normalised via RMSNorm [49], then diffusion timestep and duration conditioning signals are injected (jointly) via AdaLN. In the following self-attention layer each head employs QK-RMSNorm to prevent dot-product outputs from growing unconstrained [25]. Positional embeddings are RoPE [80] with partial rotation: only 32 of each head’s dimensions are rotated, while the remainder carry no positional information. Finally, an AdaLN gate further conditions the output before the residual connection. Cross-attention follows the same pre-norm design but without AdaLN. Embeddings are first normalised via RMSNorm and projected into queries, while keys and values are derived from the conditioning context (text and duration embeddings) through a separate projection. As in self-attention, each head also employs QK-RMSNorm [49, 25]. No positional embeddings are applied. The output embedding is added to the residual stream. Local-additive conditioning enables inpainting by adding a frame-aligned signal before the feed-forward network. The inpainting conditioning signal (a binary mask concatenated with the masked reference audio) is projected through a 2-layer MLP with SiLU and added directly to the cross-attention output. MLP layers are zero-initialised, such that the inpainting pathway can be introduced into pretrained models without disrupting its learned representations. The feed-forward network is a SwiGLU [77] where the gated linear unit operates at 4$\times$ the model dimension $d$ and the gate is a swish (SiLU) gate instead of a sigmoid. After gating, a linear layer projects back from $4d$ to $d$. This part also uses RMS-based pre-norm and AdaLN as self-attention for diffusion timestep and duration conditioning. #### Adaptive layer normalisation (AdaLN): diffusion timestep and duration conditioning. The diffusion timestep $t\in[0,1]$ is mapped to a 256-dim Fourier features vector and then projected to $d$ by an MLP with SiLU. The duration (in seconds) is normalised to $[0,1]$ and also encoded into a 256-dim Fourier features vector and then projected to $d$ by an MLP with SiLU. These two $d$-dimensional embeddings are summed element-wise and passed through another MLP with SiLU that computes a shared conditioning embeddings that are fed to each transformer block. As a result, every AdaLN ($\gamma_{s}$, $\beta_{s}$, $g_{s}$, $\gamma_{f}$, $\beta_{f}$, $g_{f}$) gets conditioning embeddings ($c_{\gamma,s}$, $c_{\beta,s}$, $c_{g,s}$, $c_{\gamma,f}$, $c_{\beta,f}$, $c_{g,f}$) that are shared across transformer blocks. Finally, each transformer block independently learns 6 bias terms ($b_{\gamma,s}$, $b_{\beta,s}$, $b_{g,s}$, $b_{\gamma,f}$, $b_{\beta,f}$, $b_{g,f}$) that are added to the shared conditional embeddings to obtain the final AdaLN conditioning [67]. For example, the self-attention AdaLN scale is $\gamma_{s}=c_{\gamma,s}+b_{\gamma,s}$ where $c_{\gamma,s}$ is shared across blocks and $b_{\gamma,s}$ is block specific. The resulting AdaLN parameters at each transformer block ($\gamma_{s}$, $\beta_{s}$, $g_{s}$, $\gamma_{f}$, $\beta_{f}$, $g_{f}$) are applied following equations in Figure [8](#S2.F8). This variant is referred to as AdaLN-Single [4], since the conditioning embeddings are shared across all transformer blocks, substantially reducing the number of conditioning parameters compared to standard AdaLN. Further, the multiplicative gating terms ($g_{s}$, $g_{f}$) are inspired by the modulation mechanism introduced in FLUX [42]. #### Cross-attention: text and duration conditioning. Text is encoded by a T5Gemma (google/t5gemma-b-b-ul2) frozen encoder into a sequence of 256 embeddings of dimension 768. Short prompts are padded to 256 with a learned padding embedding, and long prompts are truncated to 256. The duration (in seconds) is normalised to $[0,1]$ and encoded into a 256-dim Fourier features vector and then projected to $d$ by an MLP with SiLU. These two conditioning sources are concatenated along the sequence dimension, forming a context sequence of 257 embeddings. Text and duration conditioning enter each transformer block via cross-attention. Per-head QK-RMSNorm is applied to stabilize attention logits. Note that duration conditioning thus enters each transformer block through two complementary pathways: AdaLN together with the diffusion timestep, and cross-attention alongside the text prompt. #### Local-additive conditioning for inpainting. The region to edit with inpainting is signaled with a binary mask where ones mark frames to preserve and zeros mark frames to generate. To that end, the original audio is encoded into latent space by the SAME autoencoder and element-wise multiplied by the mask, zeroing out the inpaint region. The single-channel mask and the 256-channel masked latent are concatenated along the channel dimension into a 257-dimensional per-frame conditioning tensor. In each transformer block, this tensor is projected to the transformer block dimension $d$ by a MLP with SiLU and added element-wise to the residual stream between the cross-attention and feed-forward network. Because the output layer of the MLP is zero-initialized, the local-additive conditioning used for inpainting has no effect at the start of training, allowing smooth fine-tuning from a non-inpainting checkpoint. Figure: Figure 9: Local-additive conditioning for inpainting. Waveforms are encoded by a frozen SAME autoencoder into a latent sequence ($256\times T$), then element-wise multiplied by a binary mask (1=keep, 0=inpaint). The masked latent and mask are concatenated along the channel dimension. Each transformer block projects this through an MLP with SiLU and adds the result to the residual stream. Left-padding leaves space to preserve the memory embeddings. #### Differential attention. Instead of a single set of queries and keys, we build two pairs $(Q,K)$ and $(Q^{\prime},K^{\prime})$ that share a common set of values $V$. Two independent attention maps are computed and their outputs are subtracted: $\operatorname{Attn}(Q,\,K,\,V)-\operatorname{Attn}(Q^{\prime},\,K^{\prime},\,V)$, canceling the attention patterns common to both heads. Both pairs undergo the same per-head QK-RMSNorm and, in the case of self-attention, partial RoPE is used. medium and large use differential attention [92] in both self-attention and cross-attention, while small uses the standard multi-head attention. RMSNorm. We use RMSNorm [49] as a pre-normalization layer in transformer blocks. Given an input vector $\mathbf{x}$: $$ $\mathrm{RMSNorm}(\mathbf{x})=\frac{\mathbf{x}}{\sqrt{\frac{1}{d}\lVert\mathbf{x}\rVert^{2}+\epsilon}}\odot\boldsymbol{\gamma},$ (1) $$ where $\boldsymbol{\gamma}\in\mathbb{R}^{d}$ is a learnable scale parameter initialized to ones and $\epsilon=10^{-5}$. Unlike LayerNorm, RMSNorm omits mean centering and the learnable bias, reducing computation while performing comparably in practice. QK-RMSNorm. We apply per-head RMSNorm independently to Q and K after projection but before adding RoPE: $$ $\hat{Q}=\text{RMSNorm}_{q}(Q),\quad\hat{K}=\text{RMSNorm}_{k}(K),$ (2) $$ where $\text{RMSNorm}_{q}$ and $\text{RMSNorm}_{k}$ have separate learnable scale parameters shared across all heads. This prevents the attention logits to grow unboundedly. QK-RMSNorm is applied in both self-attention and cross-attention [49, 25]. ## 3 Training Stable Audio 3 models allow variable-length generation and are trained following a multi-stage pipeline (Figure [10](#S3.F10)). The first stage trains the diffusion transformer (base model) with flow matching. Later stages (distillation warmup, adversarial post-training) refine the model for improved speed and sample quality (post-trained model). All stages operate on pre-encoded SAME latents encoded offline, and use the variable-length training schema below. Figure: Figure 10: Stable Audio 3 training pipeline. First, we train a flow matching model that learns a velocity field $v_{\theta}(x_{t},t)$ defining an ordinary differential equation (ODE) transporting noise $\epsilon$ to data $x_{0}$. At inference, this ODE is solved numerically over many $t$ steps (50–100). Second, we perform a distillation warmup that repurposes the model as a one-step denoiser. Given any intermediate state $x_{t}$ sampled along the teacher’s ODE trajectory, the student (same architecture as the teacher) learns to predict the trajectory’s endpoint $\hat{x}_{0}$ (generation) directly, trained with an MSE loss. This effectively straightens the learned flow (collapsing the multi-step ODE solve into a single function evaluation $x_{t}\to\hat{x}_{0}$ for every $t$) but the MSE objective causes the student to regress toward the conditional mean $\mathbb{E}[\hat{x}_{0}\mid x_{t}]$, producing outputs that lack fine-grained detail. Third, adversarial post-training replaces the teacher signal with a relativistic adversarial setup that directly compares the student’s one-step predictions $x_{t}\to\hat{x}_{0}$ against real data ${x}_{0}$. This shifts the student’s mapping from approximating the conditional mean toward sampling from the true data distribution $p(x_{0}\mid x_{t})$, recovering the perceptual sharpness that MSE distillation smooths over. Crucially, this stage discards the teacher entirely, allowing the student to surpass the teacher’s quality ceiling by optimizing directly against real data $x_{0}$. While the resulting adversarially trained model can generate audio in a single forward pass, the mapping from pure noise $\epsilon\to\hat{x}_{0}$ in one step remains challenging (Section [5.7](#S5.SS7)). In Section [4](#S4), we describe how ping-pong sampling alleviates this by decomposing the single large step into multiple smaller ones. At each iteration, the model produces a denoised estimate $\hat{x}_{0}$, which is then renoised with new noise at a reduced level before the next denoising step. This iterative denoise-then-renoise schedule allows the model to progressively refine its output, correcting errors from earlier steps while leveraging the one-step denoising $x_{t}\to\hat{x}_{0}$ capability learned during adversarial post-training. ### 3.1 Variable-Length Training Previous latent diffusion models in the audio domain operate on fixed-length sequences, padding shorter audio with silence to match the maximum training length [15, 14]. Generating a short audio clip thus requires inference at full length, with most of the computation spent producing silence, since inference on shorter sequences than it was trained on leads to output degradation (Section [5.4](#S5.SS4)). This effectively ties inference cost to the chosen maximum length, whilst many practical usecases need audio much shorter than that maximum. Stable Audio 3, instead, natively supports variable-length generation. During training, where batching requires uniform sequence lengths for efficiency, it relies on the following mechanisms: variable-length attention and masked loss computation, per-element timestep shifts, and silence augmentation. Figure [11](#S3.F11) illustrates these mechanisms on a batch of three sequences. Figure: Figure 11: Variable-length training. A batch contains sequences of different lengths, padded to a common (variable) size. Padding embeddings are excluded (masked) from the loss. Each audio receives a length-dependent timestep shift ($\mu$), with longer sequences shifted toward higher noise levels. The signal is randomly extended with silence. #### Variable-length attention and masked loss. Sequences shorter than the batch maximum length are right-padded in latent space. Padding embeddings are excluded (masked) from both self-attention and feed-forward using variable-length flash attention [10]. Since padding positions are excluded (masked) from attention, their outputs are uninformative and the loss is computed only over valid signal positions (using a mask over the loss). Cross-attention is not masked. Memory embeddings participate in all attention layers without masking, but are removed before any loss computation. Right-padded positions are also excluded (masked) from the adversarial loss. #### Per-element timestep shifts. The noise schedule is adapted per sample based on its (unpadded) length. Note that long sequences are harder to corrupt because of correlation across sequence elements. Hence, when noise is added independently to each element of a sequence, longer sequences retain more recoverable structure at a given noise level due to redundancy between neighbouring elements [27, 7]. This means that a fixed noise schedule can under-noise long sequences relative to short ones, biasing the model toward learning to denoise at insufficiently high noise levels for long inputs. To compensate, the timestep distribution is shifted toward higher noise levels for longer sequences. The shift pushes longer sequences toward noisier timesteps (Figure [12](#S3.F12)), giving the model more training budget in the high-noise regime. The proposed shift uses the logistic form proposed by Esser et al. [13]. Given a parameter $\mu$ that interpolates between $\mu_{\min}$$=$0.5 and $\mu_{\max}$$=$1.15 as a function of the sequence length, the shifted timestep is: $$ $t^{\prime}=1-\frac{e^{-\mu}}{e^{-\mu}+\frac{t}{1-t}}.$ (3) $$ Figure: Figure 12: Effect of the per-element timestep shift on the timestep mapping. For short audios ($\mu_{\min}$$=$0.5), the shift is mild. For long audios ($\mu_{\max}$$=$1.15), timesteps are pushed substantially toward higher noise levels. #### Silence augmentation. To improve robustness and break the direct correspondence between duration conditioning and signal length, the signal region is randomly extended with silence embeddings with a length drawn from an exponential distribution (on average 4 sec of silence extension). The padding is filled with a pre-computed silence latent (obtained by encoding a zero-valued waveform) so that the model encounters realistic silence representations. This teaches the model to terminate audio cleanly with natural silence rather than abrupt cutoffs or artifacts. ### 3.2 Flow Matching Pre-Training The initial training stage uses a flow matching objective [54, 48]. Given a data sample $x_{0}$ (SAME latents) and noise $\epsilon\sim\mathcal{N}(0,I)$, the noised input at timestep $t\in[0,1]$ is the linear interpolation: $$ $x_{t}=(1-t)\,x_{0}+t\,\epsilon,$ (4) $$ and the model is trained to predict the velocity $v=\epsilon-x_{0}$ via a mean squared loss with inpainting masks. #### Inpainting training. All models are trained jointly for generation and inpainting. At each training step, a random binary mask is sampled per example, where $m$$=$1 denotes positions to keep the audio and $m$$=$0 positions to inpaint. Three mask types are drawn with probabilities: *full mask* with all zeros is equivalent to unconditional generation (probability 80%), *random segments* where 1 to 10 segments are masked out for inpainting (probability 10%), and *causal mask* where a random prefix is kept and the remainder is masked for continuation (probability 10%). Inpainting mask examples are in Figure [3](#S1.F3). The mask and the element-wise product of the clean latent with the mask are concatenated along the channel dimension and provided to the model as local-additive conditioning. The model is trained to predict the velocity and the loss is split into two independently averaged terms: a generation loss over the inpainted embeddings ($m$$=$0) and a context preservation loss over the kept audio ($m$$=$1). $$ $\frac{1}{N_{\text{gen}}}\!\sum_{i:\,m=0}\!\left\|\hat{v}_{\theta}(x_{t},t,c)_{i}-(\epsilon-x_{0})_{i}\right\|^{2}\;+\;\frac{1}{N_{\text{ctx}}}\!\sum_{j:\,m=1}\!\left\|\hat{v}_{\theta}(x_{t},t,c)_{j}-(\epsilon-x_{0})_{j}\right\|^{2}\,,$ (5) $$ where $\hat{v}_{\theta}$ is the predicted velocity, $c$ is any conditioning signal, and $N_{\text{gen}}$, $N_{\text{ctx}}$ are the number of inpainted ($m\!=\!0$) and context ($m\!=\!1$) embeddings, respectively. #### Minibatch optimal transport coupling. It is used to find a permutation of noise samples that minimises the squared $L_{2}$ transport cost within each minibatch, computed via Sinkhorn iterations on GPU [54, 9]. Concretely, given a batch of $B$ data samples and $B$ independently drawn noise vectors, the algorithm reassigns which noise vector is paired with which data sample. Standard flow matching pairs each data sample $x_{0}$ with an independently drawn noise sample $\epsilon$. When these happen to be far apart, the resulting transport path is long and may cross paths from other pairs. By solving an approximate assignment problem within each minibatch, optimal transport coupling pairs each $x_{0}$ with the closest available $\epsilon$, producing shorter, straighter, and less entangled trajectories. This straightens the velocity field that the model learns, improving both training and sampling [43]. #### Timestep sampling. Timesteps are drawn from a truncated logit-normal distribution [42, 13]. Samples from a logit-normal distribution ($t=\sigma(z),\;z\sim\mathcal{N}(0,1)$) are truncated at $t=0.075$ and rescaled to $[0,1]$. This removes very-low-noise timesteps and concentrates training budget on intermediate-high noise levels. Remember that due to our variable-length training schema, the sampled $t$ are individually shifted $t^{\prime}$ based on Equation [3](#S3.E3). ### 3.3 Distillation Warmup The distillation warmup bridges the gap between the many-step flow matching model and the one-step regime required by adversarial post-training, providing a smoother initialisation than directly addressing the adversarial objective. Before adversarial training, the flow matching model is refined through a distillation warmup stage of 10k steps [56]. A frozen copy of the pre-trained flow matching model serves as a teacher, and the student is also initialized with the pre-trained flow matching model. The teacher generates a multi-step ODE trajectory (15 DPM++ steps with CFG set to 5) from random noise $\epsilon$, and the intermediate states $(x_{t},t)$ along with the final denoised output $\hat{x}_{0}$ are cached. The teacher trajectory is refreshed periodically (every 4 iterations) to balance compute cost with target diversity. The student is then trained to match the teacher’s endpoint $\hat{x}_{0}$ in a single step: given a randomly selected intermediate state $x_{t}$ from the cached trajectory, the student predicts a velocity $v_{\theta}(x_{t},t,c)$ and produces a one-step Euler estimate $\hat{x}_{0,\theta}=x_{t}-t\,v_{\theta}(x_{t},t,c)$. The loss is the MSE between $\hat{x}_{0,\theta}$ and the teacher’s denoised output $\hat{x}_{0}$: $$ $\mathcal{L}=\|(x_{t}-t\,v_{\theta}(x_{t},t,c))-\hat{x}_{0}\|^{2}.$ $$ Note that our technique is related to ReFlow [54], which straightens the transport paths of a trained flow model. In standard flow matching, each data sample $x_{0}$ is paired with an independently drawn noise sample $\epsilon$, resulting in random couplings that may produce long, crossing transport paths. ReFlow addresses this by generating coupled endpoints $(\hat{x}_{0},\epsilon)$. It samples noise $\epsilon$ and integrates the trained model’s ODE to obtain the corresponding output $\hat{x}_{0}$. A new model is then trained to connect these coupled endpoints (with straighter paths that require fewer sampling steps, enabling faster inference). Although our approach also unrolls the teacher’s ODE, it differs in a key respect to ReFlow. Rather than retraining a flow matching model (predict velocity) on these new endpoint pairs, our student learns to map any intermediate state $x_{t}$ along the teacher’s trajectory directly to the final output $\hat{x}_{0}$ in a single step. It is also related to one-step ReFlow [54], which further distills the reflowed model into a single-step flow matching model (velocity prediction) that maps $\epsilon\to\hat{x}_{0}$ with one Euler step. Our approach shares the same one-step generation approach, but bypasses the intermediate ReFlow training stage entirely. Further, our student learns to map any intermediate state $x_{t}$ along the teacher’s ODE trajectory to the final output $\hat{x}_{0}$ based on the MSE loss (instead of velocity prediction). This is handy because during adversarial post-training our losses operate in signal ${x}_{0}$ space instead of vector field $v_{\theta}(x_{t},t)$ space, but this can introduce regression-to-the-mean artifacts despite our teacher providing a unique $\hat{x}_{0}$ for each $x_{t}$. This motivates the adversarial post-training stage, where the discriminator’s loss in ${x}_{0}$ space directly penalizes perceptual degradation and recovers the fine-grained structure that MSE distillation can smooth over. Finally, our setup is also related to Consistency Distillation [78] which trains a student model to map any point on the teacher’s ODE trajectory to the endpoint $x_{t}\to\hat{x}_{0}$. It relies on a local-consistency loss where the model’s predictions at two adjacent steps ${x}_{t}$ and ${x}_{t+1}$ along the same ODE trajectory must be the same $f_{\theta}({x}_{t+1},t+1){\approx}f_{\theta^{-}}({x}_{t},t)$, where $\theta^{-}$ denotes an exponential moving average of the student weights. Yet, local consistency alone does not determine the value trajectory steps should map to. What anchors the local consistency chain to the actual endpoint $\hat{x}_{0}$ is a boundary condition at $t{=}0$. During training, the model is architecturally constrained to bypass $\hat{x}_{0}$ so that $f_{\theta}(\hat{x}_{0},t{=}0){=}\hat{x}_{0}$ by construction. The consistency loss then chains everything together: the prediction at $t{=}0.1$ must match the prediction at $t{=}0$, which is hardwired to $\hat{x}_{0}$. The prediction at $t{=}0.2$ must match $t{=}0.1$, which already equals $\hat{x}_{0}$. This cascade propagates all the way to $t{=}1$, so that even from pure noise the model can predict $\hat{x}_{0}$. Our approach does not propagate information through a chain of local consistency losses anchored by a boundary condition. Instead, we regress directly to the teacher’s endpoint $\hat{x}_{0}$ from any intermediate state $\hat{x}_{t}$. A trade-off of our MSE-based regression approach is that if the teacher’s ODE is curved, our endpoint $\hat{x}_{0}$ estimates won’t be accurate (which is why ping-pong sampling is required, see Section [4](#S4)). Consistency Distillation sidesteps this by focusing on consistency across adjacent time-steps. ### 3.4 Adversarial Post-Training Adversarial post-training turns our pre-trained model into a one-step generator by supplanting the MSE-based conditional mean loss (of both flow matching and distillation warmup) with an adversarial loss. To that end, a discriminator evaluates the realism of denoised samples, providing distribution-level feedback that goes beyond the broad conditional mean loss. As such, if the denoised output $x_{t}\to\hat{x}_{0}$ is sufficiently real and higher-quality, fewer sampling steps are required. Another key advantage of adversarial post-training over distillation methods is that it sidesteps reliance on the performance of the pre-trained (teacher) model. The goal of adversarial post-training is to map $x_{t}\to\hat{x}_{0}$ where ${x}_{t}=(1{-}t)\,{x_{0}}+t\,{\epsilon}$ and $\hat{x}_{0}$ is an estimate of $x_{0}$. A relativistic discriminator, operating in ${x}_{0}$ space, provides the training signal required to recover the details that MSE-based distillation smooths over, effectively trading regression-to-the-mean artifacts for perceptually sharp outputs while preserving the one-step capabilities of the distilled model. We fine-tune the pre-trained model (flow matching plus distillation warmup) using adversarial post-training with three complementary losses: an adversarial relativistic loss $\mathcal{L}_{R}$, a contrastive loss $\mathcal{L}_{C}$, and a CLAP loss $\mathcal{L}_{\text{CLAP}}$. Training alternates between generator (the pre-trained one-step model) and discriminator updates: $$ Generator: $\displaystyle\mathcal{L}_{G}=\mathcal{L}_{R}^{(G)}+\mathcal{L}_{\text{CLAP}}^{(G)},$ (6) Discriminator: $\displaystyle\mathcal{L}_{D}=\mathcal{L}_{R}^{(D)}+\mathcal{L}_{C}^{(D)}.$ (7) $$ $\mathcal{L}_{R}$ drives perceptual quality, $\mathcal{L}_{C}$ regularizes the discriminator to be semantically aligned, and $\mathcal{L}_{\text{CLAP}}$ gives the generator an explicit text-alignment signal such that the generator improves both audio fidelity and prompt alignment. #### Generator architecture. It is the pre-trained model with flow matching and distillation warmup. In principle, one could parameterize the network to output $\hat{x}_{0}$ directly. Yet, we retain the original velocity parameterization $v_{\theta}(x_{t},t,c)$ from the base model and recover the clean estimate via one-step Euler sampling: $\hat{x}_{0}=x_{t}-t\,v_{\theta}(x_{t},t,c)$. Note this reparameterization also imposes a useful architectural constraint: at $t{=}0$ the model must output $\hat{x}_{0}{=}x_{0}-0\cdot v_{\theta}{=}x_{0}$. As $t$ grows, the network’s influence scales linearly with the noise level, preventing it from making disproportionately large corrections at low noise where the input is already close to clean. Further, it preserves initialization quality: since the generator starts from $v_{\theta}$, the initial outputs are already meaningful predictions, improving early training stability. The same reparameterization is used for the distillation warmup: maintaining the one-step Euler sampling reparametrization throughout post-training ensures a smooth transition from flow matching without any discontinuities. #### Discriminator architecture. The discriminator reuses the same architecture as the generator as a feature extractor, initialized from the base model pretrained with flow matching (no distillation warmup). It is a fully-conditioned discriminator that receives the text prompt, the duration conditioning, the inpainting mask and context, and a timestep $t_{D}$ (independent from $t$ used by the generator). All signals go through the same conditioning mechanisms (cross-attention, adaptive layer norm) as the generator. Having been pre-trained to process noisy data $x_{t}$ under arbitrary conditions, it already produces semantically rich intermediate representations without any additional training. Such intermediate representations are then processed by a convolutional head that produces frame-wise realness scores. The discriminator operates at a noise level $t_{D}$ that is independent of the generator’s noise level $t$. Specifically, after the generator produces its denoised estimate $\hat{x}_{0}=x_{t}-t\,v_{\phi}(x_{t},t)$, we renoise it to a fresh noise level for the discriminator: $$ $x_{t_{D}}^{\text{fake}}=(1-t_{D})\,\hat{x}_{0}+t_{D}\,\epsilon^{\prime}\quad x_{t_{D}}^{\text{real}}=(1-t_{D})\,x_{0}+t_{D}\,\epsilon^{\prime}$ (8) $$ where $t_{D}$ evaluates samples at timesteps drawn from a logit-normal distribution ($t_{D}=\sigma(z),\;z\sim\mathcal{N}(0,1)$), focusing on intermediate noise levels, and $\epsilon^{\prime}\sim\mathcal{N}(0,I)$ is shared between real and fake. The decoupling of $t$ and $t_{D}$ allows the discriminator to evaluate the generator’s output at multiple noise scales, providing training signal about both global structure (high $t_{D}$) and fine detail (low $t_{D}$), regardless of which $t$ the generator was trained on in that iteration. Finally, by using the same noise $\epsilon^{\prime}$ when constructing ${x}_{t_{D}}^{\text{real}}$ and ${x}_{t_{D}}^{\text{fake}}$, the noise component cancels in the relativistic comparison, ensuring the discriminator judges the quality difference between ${x}_{0}$ and $\hat{{x}}_{0}$ rather than incidental noise patterns. #### Adversarial relativistic loss. The generator is trained to minimize $D_{\psi}({x}_{t_{D}}^{\text{real}},t_{D},c)-D_{\psi}({x}_{t_{D}}^{\text{fake}},t_{D},c)$ and the discriminator is trained to maximize this margin. Our loss relies on the $\text{softplus}(x)=\log(1+e^{x})$ as a smooth surrogate for $\max(0,x)$. Unlike a raw margin loss that would continue rewarding an already-winning player (with low $\mathcal{L}_{R}$), softplus saturates: when it is large and negative, $\text{softplus}(x)\approx 0$, gradients vanish gracefully. Conversely, when the argument is positive, $\text{softplus}(x)\approx x$ and the loss grows linearly, providing a strong corrective signal. $$ $\displaystyle\mathcal{L}_{R}^{(G)}$ $\displaystyle=\mathbb{E}\!\left[\,\text{softplus}\!\Big(D_{\psi}({x}_{t_{D}}^{\text{real}},t_{D},c)-D_{\psi}({x}_{t_{D}}^{\text{fake}},t_{D},c)\Big)\,\right]$ (9) $\displaystyle\mathcal{L}_{R}^{(D)}$ $\displaystyle=\mathbb{E}\!\left[\,\text{softplus}\!\Big({-}\big(D_{\psi}({x}_{t_{D}}^{\text{real}},t_{D},c)-D_{\psi}({x}_{t_{D}}^{\text{fake}},t_{D},c)\big)\Big)\,\right]$ (10) $$ The loss is calculated on pairs of real/generated data [28, 31], such that the generator minimizes its detection relative to its paired real sample with the same prompt. Thus, the generator wants every generated sample to be “more real than its paired real sample", while the discriminator wants every real sample to be “more real than its paired generated sample". Critically, our pairs are always highly related due to our text-conditional task, where pairs of real/generated share the same prompt, thus providing a stronger gradient signal than relying on random pairings. Figure: Figure 13: Adversarial Post-Training. (a) Pairs of generated and real samples (with the same text prompts) are passed into the discriminator (with additive noise), where the generator and discriminator are trained to minimize and maximize (respectively) the difference between fake and real outputs. (b) The discriminator is also trained to maximize the difference between audios with correct and incorrect (shuffled) prompts. Dashed lines denote noise injection. #### Contrastive loss. To prevent the discriminator from relying on audio-only artifacts and ignoring the text conditioning, we add a contrastive term. Given a batch of real audio-prompt pairs, we create negative examples by cyclically shifting (random) the prompts across the batch. The discriminator is then trained to distinguish correctly paired from incorrectly paired audio-prompt combinations, using the same relativistic objective that maximizes their difference: $$ $\mathcal{L}_{C}^{(D)}=\mathbb{E}\!\left[\,\text{softplus}\!\Big({-}\big(D_{\psi}(\mathbf{x}_{t}\mid c_{\text{correct}})-D_{\psi}(\mathbf{x}_{t}\mid c_{\text{shuffled}})\big)\Big)\,\right].$ (11) $$ This loss can be viewed as a contrastive loss [22] where the discriminator is mapping correct audio-text pairs closer than mismatched pairs. Note that this is only a loss for the discriminator, as this encourages it to understand the alignment between prompts and noisy inputs, and prevents the model from focusing on easier (e.g., high-frequency [63]) audio features. This forces the discriminator to understand audio-text alignment, not just audio quality. #### CLAP loss. A frozen CLAP model provides direct semantic guidance to the generator. The generator’s denoised output $\hat{\mathbf{x}}_{0}$ and the text prompt are encoded, and we minimize the squared geodesic distance on the unit hypersphere: $$ $\mathcal{L}_{\text{CLAP}}^{(G)}=2\,\arcsin^{2}\!\!\left(\frac{\lVert{e}_{\text{text}}-{e}_{{\hat{x}_{0}}}\rVert_{2}}{2}\right),$ (12) $$ where ${{e}}_{\text{text}}$ and ${{e}}_{\text{audio}}$ are $\ell_{2}$-normalized embeddings from the CLAP text and audio encoders, respectively. Since decoding latents (to waveform) at each training step would be expensive, we train a CLAP model that operates directly on SAME embeddings, avoiding the need for waveform synthesis during adversarial post-training. We adopt the squared geodesic distance on the unit hypersphere as our CLAP alignment objective, since unlike cosine distance (whose gradient vanishes at both small and large angular separations) it provides a gradient magnitude proportional to the angle between embeddings across the full range. This ensures a consistent learning signal, particularly in early stages when text and audio embeddings may be misaligned. Hence, the CLAP loss provides a semantic anchor that prevents mode collapse during adversarial training by encouraging the generator to remain aligned with the prompt. ### 3.5 Training Implementation Details #### Training data. medium and large are trained on a combination of licensed audio from AudioSparx and Creative Commons recordings from Freesound. The AudioSparx portion (806,284 audios) contains music tracks, instruments, and sound effects with text metadata. The Freesound portion consists of recordings licensed under CC-0, CC-BY, or CCSampling+. To ensure no copyrighted content was present in the Freesound data, music recordings were identified using the PANNs [39] tagger. We flagged audio that activated music-related tags for at least 30s (threshold of 0.15), that was sent to a trusted content detection company to verify the absence of copyrighted material. All identified copyrighted content was removed. After filtering, the Freesound part includes 266,324 CC-0, 194,840 CC-BY, and 11,454 CC-Sampling+ recordings(^1^11The same subset of Freesound audio we used to train Stable Audio Open: [https://info.stability.ai/attributions](https://info.stability.ai/attributions).). All small models are initially pre-trained on a mixture of AudioSparx and Freesound. But for the final stage of pre-training, distillation warmup, and post-training, we use AudioSparx for small-music and a higher-quality subset of Freesound for small-sfx. As a result, note that medium and large models are able to handle both music and sound effect generation within a single unified model. However, we find that for small models the inclusion of sound effects data degrades musical coherence. By isolating the sound effects subset into small-sfx, we mitigate this interference and obtain improved perceptual quality in both domains. #### Prompt preparation. For AudioSparx tracks, metadata includes natural-language descriptions, titles, keywords, and domain-specific metadata such as BPM, genre, moods, or instruments. For Freesound recordings, metadata includes titles, tags, and natural-language descriptions. During training, prompts are constructed by concatenating a random subset of available metadata fields. For AudioSparx, with 50% probability, the metadata field identifier is prepended (Instruments: Guitar, Drums, Bass Guitar, Moods: Uplifting) or omitted. For Freesound, with 50% probability, we use an LLM-rewritten caption of the audio or we construct a prompt from the title, shuffled tags, and description. In both cases (AudioSparx and Freesound), with 50% probability, all candidate fields are shuffled or randomly subsampled. Finally, fields are joined with commas and, with 50% probability, the entire prompt is lowercased. #### Flow matching pre-training. Classifier-free guidance (CFG) is enabled at training time by randomly replacing the conditioning embeddings (text and timing) with zero vectors with probability $p{=}0.1$, allowing guidance at inference. #### Discriminator architecture. Latent features are extracted from layer 14 of the discriminator’s diffusion transformer and passed through a convolutional head consisting of: an input convolution, 4 residual blocks, each containing 2 convolutions with GroupNorm and LeakyReLU with a skip connection, and a final scoring module with 2 convolutions reducing to a single channel, producing per-embedding scores. All convolutional kernels are of size 3. #### Timestep sampling. We use different distributions for each training stage. In all cases, the sampled timestep $t$ is subsequently shifted based on the effective (unpadded) sequence length, yielding $t^{\prime}$ as defined in Equation [3](#S3.E3). - • Flow matching: Timesteps are drawn from a truncated logit-normal distribution [13, 42]. Samples from a logit-normal ($t=\sigma(z),\;z\sim\mathcal{N}(0,1)$) are truncated at $t=0.075$ and rescaled to $[0,1]$. This removes near-clean timesteps where the task is trivial and concentrates training on intermediate-to-high noise levels. - • Distillation warmup: The teacher generates reference trajectories using DPM++ with 15 steps and CFG 5.0. The teacher’s schedule uses a log-SNR schedule *not* shifted based on sequence length, see Section [4](#S4). - • Generator: the same setup (truncated logit-normal distribution) as described above for flow matching. - • Discriminator: Both real and generated samples are noised to the same $t_{D}$ (logit-normal) with shared noise $\epsilon$. #### Optimizer. We use the Muon+AdamW hybrid optimizer [32]. Muon (momentum 0.95, learning rate $10^{-5}$) is applied to attention QKV projections and feed-forward network projections, while AdamW (learning rate $10^{-6}$, $\beta=(0.9,0.95)$, weight decay $0.01$) handles all remaining parameters. The same optimizer configuration is used for both generator and discriminator. The learning rate follows an inverse power-law schedule ($\gamma{=}10^{6}$, power $0.5$). #### Exponential Moving Average (EMA). We maintain an EMA of the generator weights ($\beta{=}0.9995$, power-law warmup with exponent $0.75$) throughout training and inference. EMA is not applied to the discriminator. ## 4 Inference With adversarial post-training, our model is fine-tuned to directly estimate clean outputs $\hat{x}_{0}$ from noisy inputs $x_{t}$ at arbitrary noise levels $t$. As such, the resulting adversarially post-trained model can generate audio in a single pass. Yet, one step mappings $\epsilon\to\hat{x}_{0}$ remain challenging (Section [5.7](#S5.SS7)). To adjust for that, we employ ping-pong sampling that addresses this by decomposing the single large step into multiple smaller ones. At each iteration, the model produces a denoised estimate $\hat{x}_{0}$, which is then renoised with new noise at a reduced level before the next denoising step. This iterative denoise-then-renoise schedule allows the model to progressively refine its output, correcting errors from earlier steps while leveraging the one-step denoising $x_{t}\to\hat{x}_{0}$ capability learned during adversarial post-training. Figure: Figure 14: Ping-pong sampling. Starting from pure noise $\epsilon\sim\mathcal{N}(0,I)$ at $t{=}1$, each step alternates between denoising to an $\hat{x}_{0}^{(t)}$ estimate (solid arrows) and stochastically re-noising with new noise at a lower timestep (dashed arrows). The re-noising magnitude decreases as $t\to 0$, producing a zigzag trajectory that converges to the data manifold. Note that ping-pong sampling is inherently self-correcting. If early steps (where $x_{t}\to\hat{x}_{0}$ is more difficult) produce an inaccurate estimate, the subsequent re-noising step yields a new state that can correct the previous (difficult) estimate. In contrast, ODE solvers propagate errors forward, e.g.: a bad Euler step displaces $x_{t}$ from the actual best trajectory, and all subsequent steps integrate from the wrong starting point. This self-correcting property of ping-pong sampling means that our approach is tolerant to moderate deviations and errors throughout inference steps. Instead of spacing timesteps linearly in $t\in[0,1]$, we construct a schedule that is uniform in logSNR space. We define $N{+}1$ ($N{=}8$) equally-spaced logSNR steps $\lambda_{0},\dots,\lambda_{N}$ in the interval $[\lambda_{\min},\lambda_{\max}]=[-6.2,\;2.0]$ and recover(^2^22In flow matching $x_{t}=(1{-}t)\,x_{0}+t\,\epsilon$, so the logSNR is $\lambda(t)=\log\!\bigl(\tfrac{1-t}{t}\bigr)$. Inverting this relationship gives $t=\tfrac{1}{1+e^{\lambda}}=\sigma(-\lambda)$.) timesteps via $t_{i}=\sigma(-\lambda_{i})$. This follows from perceptual salience being approximately uniform in logSNR space [38]. We found that 8 sampling steps provide a favorable trade-off between inference efficiency and generation quality. During training, we employ a length-dependent timestep shift. Yet, at inference, we use a logSNR-uniform schedule that is *not* length-dependent (the same schedule is used regardless of the requested duration). While this introduces a train–inference mismatch, this setup works well in practice. Note that during training a large amount of timesteps are being sampled (coverage at higher-noises matters), while inference takes only 8 steps (placement matters). Crucially, the generation length (input noise $\epsilon$ length) is variable, see Figure [2](#S1.F2). Given a requested duration of $d$ seconds, we allocate a latent sequence of $L=\lceil(d+d_{\text{silence}})\cdot f_{s}/r\rceil$ embeddings, where $d$ is the generation duration requested by the user, $d_{\text{silence}}{=}6\,\text{s}$ is silence padding, $f_{s}{=}44{,}100\,\text{Hz}$ is the sample rate, and $r{=}4{,}096$ is the autoencoder downsampling ratio. Only the first $L_{\text{eff}}=\lceil d\cdot f_{s}/r\rceil$ embeddings correspond to the target audio content. The remaining $L-L_{\text{eff}}$ embeddings are silence padding. Silence padding serves two purposes: it prevents boundary artifacts that arise when the model produce an abrupt ending exactly at the sequence edge, and it provides a fade-out buffer for the decoder. After generation, the output can be trimmed to the requested $d$ seconds, discarding the padding region. Our model does not require classifier-free guidance (CFG) at inference. Standard diffusion models rely on CFG to improve sample quality and text alignment, at the cost of two forward passes per denoising step (one conditional and one unconditional). In our pipeline, guidance is baked into the model through distillation warmup where the student is trained to match CFG-enhanced teacher trajectories, internalizing the quality boost that CFG provides. Adversarial post-training further refines text–audio alignment via the $\mathcal{L}_{\text{CLAP}}$ directly. As a result, our models do not rely on CFG during inference. This is a critical advantage for on-device deployment where memory and compute are constrained. ## 5 Discussion Stable Audio 3 is a family of text-prompted models for instrumental music and sound effects generation and editing, trained exclusively on licensed or creative commons data and designed to run fast and on consumer-grade hardware. In the following sections, we discuss the results below: - • State-of-the-art results for instrumental music and sound effects generation (Sections [5.2](#S5.SS2) and [5.3](#S5.SS3)) - • Fast inference: less than 2s to generate up to 6m 20s on an H200 (Sections [5.2](#S5.SS2) and [5.3](#S5.SS3)). - • Robust variable-length audio generation (Sections [5.4](#S5.SS4) and [5.5](#S5.SS5)). - • Audio editing via inpainting, including single- and multi-segment edits and continuation (Section [5.6](#S5.SS6)). - • Adversarial post-training enables improved inference speed and sample quality (Section [5.7](#S5.SS7)). - • small and medium can run on consumer-grade GPUs and small on a MacBook Pro (Sections [5.8](#S5.SS8) and [5.9](#S5.SS9)). ### 5.1 Methodology We evaluate the Stable Audio 3 models against both open-weight and internal baselines using metrics and a subjective listening test. We compare a diverse set of systems spanning diffusion and autoregressive architectures: - • Stable Audio 2.5 [15]: our internal latent diffusion baseline for (up to 190s) music generation. We use 8 sampling steps, CFG scale 6, and the DPM++ 3M SDE sampler. - • Stable Audio Open [16]: an open-weight latent diffusion model for (up to 47s) sound effects generation. We use 100 sampling steps, CFG scale 7, and the DPM++ 3M SDE sampler. - • Stable Audio Open Small [16]: a compact open-weight latent diffusion model optimized for efficient short-form (up to 11s) sound effects generation. We use 8 sampling steps and the PingPong sampler. - • Stable Audio 3 ‘base model’: our flow matching Stable Audio 3 variants (small, medium, large), without post-training. All models are evaluated using 50 sampling steps, CFG scale 7, and the Euler sampler. - • Stable Audio 3 ‘post-trained’: our Stable Audio 3 variants (small, medium, large) that are post-trained with distillation warmup and adversarial post-training. We use 8 sampling steps and PingPong sampler. - • ACE-Step 1.5 [20]: an open-weight hybrid diffusion and autoregressive model for full-length song generation. We use the acestep-5Hz-lm-4B autoregressive model with the acestep-v15-xl-turbo backbone, since the authors found to provide a better quality–speed tradeoff than acestep-v15-xl-sft. Generation is performed using task_type="text2music", lyrics="[Instrumental]", thinking=True. - • DiffRhythm 2 [30]: an open-weight semi-autoregressive block flow-matching model for full-length song generation. Prompts are provided through the style_prompt argument, while lyrics conditioning is disabled. - • Woosh Flow [23]: an open-weight flow matching model for (up to 5s) sound effects generation. The model uses an adaptive-step ODE solver, resulting in 72 sampling steps on average. - • Woosh DFlow [23]: an open-weight distilled version of Woosh Flow for (up to 5s) sound effects generation. We use 4 generation steps, following the official implementation. - • TangoFlux [29]: an open-weight flow matching model for (up to 30s) sound effects generation baseline. We use 50 sampling steps and CFG scale 4.5. Note that different models generate audio at varying lengths. For a fair comparison, we evaluate all methods considering each model’s maximum supported duration, e.g., Woosh generates up to 5s or small generates up to 120s. For models capable of vocal generation (ACE-Step 1.5 and DiffRhythm 2), evaluation is restricted to instrumental prompts to ensure fair comparison with Stable Audio 3, which focuses on instrumental music generation. We also considered JAM [53], HeartMuLa [91], and YuE [94]. JAM was excluded because it relies on lyrics and reference audio conditioning instead of focusing on instrumental music generation. YuE and HeartMuLa were excluded because they rely on tags prompting and are designed for vocal music rather than instrumental music generation from text prompts. All Stable Audio models trained on the AudioSparx dataset (Stable Audio 2.5 and 3 variants, excluding Stable Audio Open and Open Small) prepend prompts with "TrackType: Music, VocalType: Instrumental," for music generation and "TrackType: SFX," for sound effects generation. These prefixes indicate the target audio modality and significantly improve generation quality. We therefore recommend all Stable Audio 3 users to use those prefixes at inference time. We report three metrics: - • Fréchet Audio Distance (FAD) [34, 21]: measures distributional similarity between generated and reference audio distributions. We compute FAD using embeddings from the LAION-CLAP audio encoder checkpoint 630k-audioset-best.pt, which we found to provide the best perceptual correspondence. A low FAD implies that the generated audio is plausible and closely matches the reference audio. - • CLAP score: cosine similarity between text and audio embeddings from the same LAION-CLAP model above, measuring semantic alignment between prompts and generated audio. The higher the better. - • Inference time: wall-clock generation latency measured for fixed-length audio generation under standardized hardware and inference settings. Unless stated the contrary, inference time is measured on a H200. We run our evaluation on two datasets: - • Song Describer Dataset (SDD) [58]: a dataset of 120s-long music tracks paired with human-written captions. We exclude prompts containing vocals, speech, or sound effects, retaining only instrumental examples. We additionally remove incoherent or ambiguous prompts. This results in 424 music-caption pairs. - • BBC Sound Effects Dataset: a collection of recorded environmental and sound effects with text prompts. For the BBC Sound Effects dataset, we first filter the dataset to samples with duration up to 120s. We select this cutoff as it is sufficient to cover both sound effects and environmental recordings, and matches the maximum generation length of small. Further, due to the duration limitations of several baselines, we construct multiple evaluation subsets: - • $\leq 120$s subset: 10,491 audio-caption pairs. - • $\leq 30$s subset: 5,406 audio-caption pairs. - • $\leq 10$s subset: 1,537 audio-caption pairs. - • $\leq 5$s subset: 393 audio-caption pairs. For all subsets above, generated audio durations match the duration of the corresponding reference. This variable-length setup enables fair comparison between models with different maximum generation durations while preserving realistic prompt-length distributions (e.g., shorter/longer durations for sound effects/environmental recordings). We do not use the AudioCaps[35] dataset for evaluation, as we found that its (reference) audio is bandwidth-limited. Instead, we use the BBC Sound Effects dataset, whose recordings are professionally produced and full-bandwidth. We run a listening test using a Mean Opinion Score protocol with 14 participants, evaluating generated samples on: - • Overall quality (OVL): evaluates production quality, perceptual realism, and the absence of artifacts. - • Text relevance (REL): evaluates how accurately the generated audio matches the conditioning text prompt. - • Musicality (MUS): evaluates the capacity to articulate musically coherent melodies and harmonies. For our inpainting music evaluations, we use the SDD with music of 120s long. For our inpainting evaluations with sound, we use BBC Sound Effects samples with durations between 30s and 120s (5088 audio-caption pairs), excluding shorter clips to ensure meaningful continuation and multi-region inpainting evaluation. We evaluate three settings: - • Single inpaint: a randomly selected region corresponding to between 2% and 20% of the audio duration is masked and regenerated. The masked region is constrained to be at least 1s long. - • Double inpaint: two independent masked regions are generated using the same procedure as single inpainting. The two regions are constrained to be separated by at least 6s. - • Continuation: an initial segment is randomly selected (between 5s and 20% of the audio) is preserved, and the remainder is regenerated until the end. Identical randomly sampled inpainting regions are used across all compared models to ensure fair comparison. In our inpainting evaluation these signals are available: the original audio (to be inpainted), the generated audio (with the inpainted part), and the prompt (used to guide inpainting). In our setup, the original audio and the prompt are paired and available through our evaluation datasets, and one of the models under evaluation generates part of the audio (via inpainting). Below, we use this terminology to describe the metrics we use to evaluate inpainting capabilities: - • FAD full is computed between the (full) generated audio and the (full) original reference audio. This metric assesses how well the inpainted region integrates within the original audio, also taking into account the transitions between generated (inpainted) and original regions. - • FAD inpaint is computed only considering the generated (inpainted) regions and its corresponding reference (original) segments. This metric is useful to evaluate the generated (inpainted) part alone. - • CLAP text-gen measures the similarity between the text prompt and the generated (inpainted) region. - • CLAP gen-orig measures the similarity between the audio embeddings of the generated (inpainted) region and the original audio in this region. For double-inpainting experiments, each inpainted region is treated as an independent evaluation, effectively doubling the number of evaluated inpaint segments. Yet, in this case, full-audio FAD metrics are computed only once per audio. ### 5.2 Instrumental music generation We evaluate instrumental music generations on the SDD at two durations: 120s and 190s. The 120s setting corresponds to the maximum generation length of small, while the 190s setting evaluates longer-form generation and corresponds to the maximum generation length of Stable Audio 2.5. small is therefore excluded from the 190s evaluation. **Table 3: Instrumental music generation results on the SDD dataset with 120s generations.** | | | | | | | | inference | | --- | --- | --- | --- | --- | --- | --- | --- | | | length | FAD $\downarrow$ | CLAP $\uparrow$ | OVL $\uparrow$ | REL $\uparrow$ | MUS $\uparrow$ | time (s) $\downarrow$ | | DiffRhythm 2 | 120s | 0.293 | 0.158 | 3.05 $\pm$ 0.94 | 2.10 $\pm$ 1.29 | 2.60 $\pm$ 1.10 | 3.88 | | ACE-Step 1.5 xl-turbo | 120s | 0.193 | 0.321 | 3.35 $\pm$ 1.09 | 3.30 $\pm$ 1.13 | 3.15 $\pm$ 1.31 | 6.23 | | Stable Audio 2.5 | 120s | 0.106 | 0.395 | 3.90 $\pm$ 0.79 | 4.30 $\pm$ 0.66 | 3.70 $\pm$ 0.92 | 0.85 | | small-music | 120s | 0.145 | 0.393 | 3.20 $\pm$ 0.89 | 3.60 $\pm$ 0.94 | 3.15 $\pm$ 0.81 | 0.45 | | medium | 120s | 0.107 | 0.390 | 4.20 $\pm$ 0.89 | 4.25 $\pm$ 0.85 | 4.15 $\pm$ 0.93 | 0.78 | | large | 120s | 0.101 | 0.393 | 3.95 $\pm$ 0.89 | 3.80 $\pm$ 1.11 | 4.30 $\pm$ 0.73 | 0.81 | **Table 4: Instrumental music generation results on SDD dataset with generations of 190s (small generates up to 120s).** | | | | | inference | | --- | --- | --- | --- | --- | | | length | FAD $\downarrow$ | CLAP $\uparrow$ | time (s) $\downarrow$ | | DiffRhythm 2 | 190s | 0.307 | 0.088 | 5.85 | | ACE-Step 1.5 xl-turbo | 190s | 0.184 | 0.273 | 9.21 | | Stable Audio 2.5 | 190s | 0.128 | 0.375 | 0.85 | | medium | 190s | 0.116 | 0.362 | 0.88 | | large | 190s | 0.100 | 0.373 | 0.93 | Tables [3](#S5.T3) and [4](#S5.T4) report results across both settings. Overall, Stable Audio models achieve the strongest performance across metrics. Stable Audio 2.5 remains the best-performing baseline, while Stable Audio 3 medium and large substantially improve musicality. In contrast, the small variant performs noticeably worse than the larger models. However, despite its reduced size and the use of a lightweight autoencoder optimized for CPU inference, it remains competitive with open-weight baselines. Further, Stable Audio models achieve substantially faster inference times than open-weight baselines, generating 190s audio in under one second. Finally, Stable Audio 3 models exhibit minimal degradation when increasing generation length from 120s to 190s. Overall, results highlight the efficiency advantages and improved musicality of Stable Audio 3, enabling high-quality long-form music generation with low latency. ### 5.3 Sound effects generation We evaluate sound effects generation using target durations of 5s, corresponding to the maximum supported duration of Woosh models. This restricted setting enables comparison against a broad set of recent open-weight sound effects generation systems. Additional evaluations at longer durations (up to 120s) are discussed in Section [5.5](#S5.SS5). **Table 5: Sound effects generation results on the BBC Sound Effects Dataset with 5s generations.** | | | | | | | inference | | --- | --- | --- | --- | --- | --- | --- | | | length | FAD $\downarrow$ | CLAP $\uparrow$ | OVL $\uparrow$ | REL $\uparrow$ | time (s) $\downarrow$ | | TangoFlux | 5s | 0.760 | 0.179 | 2.35 $\pm$ 1.04 | 3.25 $\pm$ 1.37 | 1.90 | | Woosh DFlow | 5s | 0.619 | 0.228 | 3.10 $\pm$ 1.25 | 3.20 $\pm$ 1.64 | 0.06 | | Woosh Flow | 5s | 0.580 | 0.277 | 3.45 $\pm$ 1.19 | 3.80 $\pm$ 1.28 | 1.92 | | SAO | 5s | 0.501 | 0.263 | 2.95 $\pm$ 1.32 | 3.30 $\pm$ 1.30 | 12.30 | | SAO-small | 5s | 0.500 | 0.277 | 3.10 $\pm$ 1.12 | 3.55 $\pm$ 1.00 | 0.24 | | small-sfx | 5s | 0.395 | 0.351 | 3.35 $\pm$ 1.39 | 3.25 $\pm$ 1.45 | 0.41 | | medium | 5s | 0.369 | 0.369 | 3.65 $\pm$ 1.14 | 3.95 $\pm$ 1.23 | 0.60 | | large | 5s | 0.358 | 0.370 | 3.60 $\pm$ 0.94 | 3.85 $\pm$ 1.04 | 0.64 | Table [5](#S5.T5) shows that Stable Audio 3 models consistently outperform all baselines. In particular, large and medium achieve the best overall performance while Woosh Flow remains a strong baseline. Interestingly, we observe a discrepancy between FAD and OVL scores for Woosh Flow. Although it attains competitive subjective OVL quality, it is often penalized by FAD due to producing band-limited signals. The results also highlight the efficiency of Stable Audio 3 models. While Woosh DFlow achieves extremely low inference latency, this comes at a quality cost. In contrast, Stable Audio 3 models maintain fast inference while obtaining state-of-the-art results for sound effects generation. Unlike prior open-weight systems that specialize exclusively in either music or sound effects generation, medium and large are trained for both instrumental music and sound effects generation with a single model. Despite this shared training setup, they achieve state-of-the-art performance in both domains. Yet, due to the limited parameter budget of small, we instead train two specialized models: small-music for instrumental music generation and small-sfx for sound effects generation. This design choice enables improved quality despite tight compute and memory constraints. ### 5.4 Variable-length instrumental music generation We evaluate Stable Audio models across multiple generation durations, ranging from 20s clips to 380s full-length generations (when possible). Note that small-music can generate up to 120s and Stable Audio 2.5 up to 190s. Stable Audio 2.5 was trained using fixed-length 190s sequences as in Figure [2](#S1.F2) (a). Shorter training examples were padded with silence to match the 190s maximum duration, enabling the model to learn variable-duration content within a fixed-length generation. Producing shorter clips, therefore, is not efficient as it requires running inference over the full sequence length, with most computation spent generating silence. Yet, running inference directly on shorter sequences than those used during training (as in Figure [2](#S1.F2) (b) despite being trained as (a)) could lead to degraded quality. We evaluate this behavior in the misused Stable Audio 2.5 setting in Table [6](#S5.T6), where inference is performed directly at shorter durations instead of using the original fixed-length generation procedure. The results show that FAD and CLAP degrade, denoting that Stable Audio 2.5 does not generalize to perform *efficient* variable-length inference. **Table 6: Misusing Stable Audio 2.5 to perform efficient variable-length instrumental music generation without success.** | | | | | inference | | --- | --- | --- | --- | --- | | | length | FAD $\downarrow$ | CLAP $\uparrow$ | time (s) $\downarrow$ | | Stable Audio 2.5 (misused) | 20s | 0.731 | 0.285 | 0.41 | | Stable Audio 2.5 | 20s | 0.149 | 0.389 | 0.85 | | Stable Audio 2.5 (misused) | 120s | 0.201 | 0.382 | 0.79 | | Stable Audio 2.5 | 120s | 0.106 | 0.395 | 0.85 | | Stable Audio 2.5 | 190s | 0.128 | 0.375 | 0.85 | Stable Audio 3 models are explicitly designed for native variable-length generations. Our results in Table [7](#S5.T7) highlight the practical advantages of native variable-length generation. Stable Audio 3 inference cost scales naturally with output duration, enabling efficient short-form generation. Across durations, Stable Audio 3 models remain generally strong, with best performance typically observed at intermediate lengths (120–190s). At very short lengths (20s), we observe degradation in both FAD and CLAP, which we attribute to a mismatch between training and evaluation datasets, as most short training samples are loops (not full songs as in the evaluation set). At very long lengths (380s), performance also degrades, particularly the CLAP score. Informal listening suggests that the reduced prompt adherence is because long training examples are predominantly ambient or classical music. As a result, conditioning on long durations can bias the model toward generating ambient or classical music, omitting the provided text prompt. **Table 7: Instrumental music generation results across different lengths.** | | | | | inference | | --- | --- | --- | --- | --- | | | length | FAD $\downarrow$ | CLAP $\uparrow$ | time (s) $\downarrow$ | | small-music | 20s | 0.196 | 0.376 | 0.43 | | small-music | 120s | 0.145 | 0.393 | 0.45 | | medium | 20s | 0.163 | 0.392 | 0.62 | | medium | 120s | 0.107 | 0.390 | 0.78 | | medium | 190s | 0.116 | 0.362 | 0.88 | | medium | 380s | 0.156 | 0.275 | 1.31 | | large | 20s | 0.171 | 0.390 | 0.66 | | large | 120s | 0.101 | 0.393 | 0.81 | | large | 190s | 0.100 | 0.373 | 0.93 | | large | 380s | 0.131 | 0.277 | 1.80 | **Table 8: Sound effects generation results across different lengths.** | | | | | inference | | --- | --- | --- | --- | --- | | | length | FAD $\downarrow$ | CLAP $\uparrow$ | time (s) $\downarrow$ | | small-sfx | 5s | 0.395 | 0.35 | 0.41 | | small-sfx | 10s | 0.367 | 0.348 | 0.44 | | small-sfx | 30s | 0.335 | 0.343 | 0.46 | | small-sfx | 120s | 0.293 | 0.322 | 0.47 | | medium | 5s | 0.369 | 0.369 | 0.60 | | medium | 10s | 0.340 | 0.361 | 0.63 | | medium | 30s | 0.297 | 0.358 | 0.65 | | medium | 120s | 0.266 | 0.321 | 0.77 | | large | 5s | 0.358 | 0.370 | 0.63 | | large | 10s | 0.328 | 0.367 | 0.67 | | large | 30s | 0.286 | 0.361 | 0.69 | | large | 120s | 0.259 | 0.322 | 0.81 | | SAO | 5s | 0.501 | 0.263 | 12.30 | | SAO | 10s | 0.452 | 0.277 | 12.30 | | SAO | 30s | 0.364 | 0.290 | 12.30 | | SAO-small | 5s | 0.500 | 0.277 | 0.24 | | SAO-small | 10s | 0.456 | 0.280 | 0.24 | | TangoFlux | 5s | 0.760 | 0.179 | 1.90 | | TangoFlux | 10s | 0.680 | 0.214 | 1.90 | | TangoFlux | 30s | 0.595 | 0.206 | 1.90 | | Woosh Flow | 5s | 0.580 | 0.277 | 1.92 | | Woosh DFlow | 5s | 0.619 | 0.228 | 0.06 | ### 5.5 Variable-length sound effects generation Table [8](#S5.T8) evaluates the quality of sound effects generation and inference speed for varying output durations and models. Although medium and large support generation up to 380s, we restrict our evaluation to a maximum duration of 120s for simplicity and to easily compare against existing baselines. Across all evaluated durations, the proposed models consistently outperform the baselines while maintaining fast inference times. An interesting trend is that FAD improves monotonically as generation length increases across all proposed variants. We hypothesize that this behavior is partly driven by the nature of longer-duration samples, which are predominantly composed of field recordings and ambient soundscapes with lower acoustic diversity and slower temporal variation. As a result, the generated audio exhibits lower distributional discrepancy with respect to the evaluation data, leading to improved FAD scores. In contrast, CLAP scores decrease for longer generations, likely reflecting the difficulty of maintaining semantic alignment with the text prompt over extended periods of time. ### 5.6 Audio editing capabilities In this section we evaluate the editing capabilities of our models on various tasks: inpainting (single- and double-region) and continuation, for both music (Table [9](#S5.T9)) and sound effects (Table [10](#S5.T10)). Methodological details are explained in Section [5.1](#S5.SS1). Across both domains, small obtains worse FAD results than medium and large, which we attribute to its reduced model capacity and smaller (CPU-optimized) autoencoder. Informal listening also reveals that small produces less smooth transitions, as reflected by the worse FAD full when compared to FAD inpaint. This discrepancy, however, is less pronounced in larger models, denoting that medium and large generate more coherent edits. Further, single and double inpaint numbers are close in both tables. This indicates that our models handle two independent masked regions as well as one. Also, for the continuation setting FAD metrics are generally worse than inpainting. We attribute this to the stronger conditioning constraints in the inpainting setup, where the model is provided with a substantial amount of audio context. This conditioning effectively anchors inpaint edits to the reference distribution, reducing deviation and leading to lower FAD. In contrast, continuation requires extrapolating from a shorter context, which increases variability in long-range structure and results in worse FAD metrics. Continuation results also show that FAD inpaint is worse than FAD full, the opposite of what we see in inpainting, because the generated region is unconstrained on one side and therefore drifts further from the reference distribution. The CLAP gen-orig metric is also worse, reflecting that without the surrounding context to anchor the generation, the continuations are less acoustically similar to the original even when prompt alignment (CLAP text-gen) remains high or improves. **Table 9: Music editing results across models and tasks for inpainting and continuation settings.** | | | FAD | FAD $\downarrow$ | CLAP $\uparrow$ | CLAP $\uparrow$ | | --- | --- | --- | --- | --- | --- | | | | full $\downarrow$ | inpaint | text-gen | gen-orig | | Single Inpaint | small | 0.100 | 0.086 | 0.271 | 0.833 | | medium | 0.046 | 0.040 | 0.289 | 0.875 | | | large | 0.047 | 0.040 | 0.289 | 0.878 | | | Double Inpaint | small | 0.100 | 0.083 | 0.274 | 0.834 | | medium | 0.046 | 0.034 | 0.288 | 0.874 | | | large | 0.047 | 0.035 | 0.289 | 0.876 | | | Continuation | small | 0.121 | 0.130 | 0.404 | 0.649 | | medium | 0.074 | 0.084 | 0.400 | 0.653 | | | large | 0.071 | 0.078 | 0.403 | 0.667 | | **Table 10: Sound effects editing results across models and tasks for inpainting and continuation settings.** | | | FAD | FAD $\downarrow$ | CLAP $\uparrow$ | CLAP $\uparrow$ | | --- | --- | --- | --- | --- | --- | | | | full $\downarrow$ | inpaint | text-gen | gen-orig | | Single Inpaint | small | 0.170 | 0.119 | 0.196 | 0.717 | | medium | 0.086 | 0.068 | 0.220 | 0.748 | | | large | 0.084 | 0.068 | 0.220 | 0.752 | | | Double Inpaint | small | 0.171 | 0.119 | 0.195 | 0.714 | | medium | 0.089 | 0.070 | 0.216 | 0.742 | | | large | 0.086 | 0.069 | 0.217 | 0.746 | | | Continuation | small | 0.208 | 0.214 | 0.351 | 0.600 | | medium | 0.172 | 0.191 | 0.332 | 0.570 | | | large | 0.195 | 0.218 | 0.311 | 0.546 | | ### 5.7 Adversarial Post-Training discussion Tables [11](#S5.T11) and  [12](#S5.T12) compare the pre-trained flow matching (base) models against models further trained using distillation warmup and adversarial post-training (post-trained). The base models require 50 sampling steps at inference time, resulting in substantially higher latency while also yielding inferior generation quality. In contrast, post-trained models enable faster generation and can operate with as few as a single sampling step. However, directly generating audio latents from pure noise ($\epsilon$ $\to$ $\hat{x}_{0}$) in one step remains difficult, leading to degraded FAD and CLAP scores. For this reason, we choose 8-step ping-pong sampling in all our experiments. The iterative denoise-then-renoise schedule of ping-pong sampling allows the model to progressively refine its output, correcting errors from earlier steps while still leveraging the one-step denoising $x_{t}\to\hat{x}_{0}$ capability learned during distillation warmup and adversarial post-training. Under this setting, post-trained models achieve a good balance between generation quality and efficiency, delivering improved FAD and CLAP scores while still requiring substantially lower inference time than the base models. **Table 11: Comparison of pre-trained and post-trained music models at various sampling steps.** | | | | | | | inference | | --- | --- | --- | --- | --- | --- | --- | | | | length | steps | FAD $\downarrow$ | CLAP $\uparrow$ | time (s) $\downarrow$ | | small | base model | 120s | 50 | 0.162 | 0.370 | 2.89 | | medium | base model | 120s | 50 | 0.143 | 0.352 | 3.87 | | large | base model | 120s | 50 | 0.116 | 0.355 | 3.90 | | small | post-trained | 120s | 1 | 0.439 | 0.300 | 0.09 | | medium | post-trained | 120s | 1 | 0.258 | 0.355 | 0.27 | | large | post-trained | 120s | 1 | 0.273 | 0.331 | 0.28 | | small | post-trained | 120s | 8 | 0.145 | 0.393 | 0.45 | | medium | post-trained | 120s | 8 | 0.107 | 0.390 | 0.78 | | large | post-trained | 120s | 8 | 0.101 | 0.393 | 0.81 | **Table 12: Comparison of pre-trained and post-trained sound effects models at various sampling steps.** | | | | | | | inference | | --- | --- | --- | --- | --- | --- | --- | | | | length | steps | FAD $\downarrow$ | CLAP $\uparrow$ | time (s) $\downarrow$ | | small | base model | 120s | 50 | 0.336 | 0.284 | 2.89 | | medium | base model | 120s | 50 | 0.312 | 0.298 | 3.87 | | large | base model | 120s | 50 | 0.282 | 0.299 | 3.90 | | small | post-trained | 120s | 1 | 0.626 | 0.228 | 0.09 | | medium | post-trained | 120s | 1 | 0.596 | 0.186 | 0.27 | | large | post-trained | 120s | 1 | 0.604 | 0.188 | 0.28 | | small | post-trained | 120s | 8 | 0.293 | 0.322 | 0.45 | | medium | post-trained | 120s | 8 | 0.266 | 0.321 | 0.78 | | large | post-trained | 120s | 8 | 0.259 | 0.322 | 0.81 | ### 5.8 VRAM Memory Usage In Table [13](#S5.T13) we report peak VRAM consumption across different models and generation durations. Peak memory usage increases with both model size and sequence length. small exhibits the lowest memory footprint, remaining below 2.5 GB even at 120s. medium and large require approximately 6.5 GB and 9.0 GB respectively at longer durations. These memory requirements are compatible with a wide range of modern consumer-grade GPUs. In particular, small can comfortably run on entry-level GPUs such as the RTX 3050 (typically 4–8 GB VRAM) and laptop GPUs with comparable memory capacity, while supporting generation lengths of up to 120s. Meanwhile, medium requires approximately 6.5 GB of VRAM for long-form audio generation and remains compatible with widely available consumer GPUs such as the RTX 3060 (12 GB VRAM), RTX 4060 (8 GB VRAM), and RTX 4070 (12 GB VRAM). **Table 13: Peak VRAM usage across model sizes and generation durations.** | | Duration | Peak VRAM | | --- | --- | --- | | small | 30s | 1.89 GB | | small | 120s | 2.40 GB | | medium | 30s | 5.49 GB | | medium | 120s | 6.49 GB | | medium | 380s | 6.52 GB | | large | 30s | 8.01 GB | | large | 120s | 9.01 GB | | large | 380s | 9.04 GB | ### 5.9 Inference Times Across Hardware Platforms We compare inference times across multiple hardware platforms. All runs use a fixed configuration of 8 ping-pong sampling steps. We consider four different experiments: (i) MacBook Pro M4 with CPU-only; (ii) MacBook Pro M4 with CoreML acceleration employing CPU, GPU, and neural engine; (iii) NVIDIA H200 GPU with standard PyTorch execution; and (iv) NVIDIA H200 GPU with TensorRT acceleration. We report end-to-end generation latency across different generation durations and model scales. The results in Table [14](#S5.T14) show that CoreML significantly improves performance over CPU-only execution on the MacBook Pro M4. Second, on the H200 GPU, TensorRT further accelerates inference, reducing times by an order of magnitude in most configurations. #### Implementation details CPU-only results on the MacBook Pro M4 are obtained through accelerating small with CoreML and accelerating SAME-S (decoder) with TFLite. The SAME-L decoders used by medium and large are accelerated in PyTorch with torch.compile instead of TensorRT. TensorRT does not support acceleration for the sliding-window attention used in SAME-L, whereas torch.compile can better exploit this setting. **Table 14: Inference times across hardware platforms, acceleration settings, model sizes, and generation lengths.** | | | | generation length | | | | | --- | --- | --- | --- | --- | --- | --- | | Hardware | Acceleration | Model | 5s | 30s | 120s | 380s | | MacBook Pro M4 | CPU only | small | 0.70s | 1.72s | 5.92s | – | | CoreML | small | 0.23s | 0.63s | 3.09s | – | | | H200 | – | small | 0.41s | 0.46s | 0.45s | – | | – | medium | 0.60s | 0.65s | 0.78s | 1.31s | | | – | large | 0.63s | 0.69s | 0.81s | 1.80s | | | TensorRT | small | 0.017s | 0.022s | 0.044s | – | | | TensorRT | medium | 0.02s | 0.05s | 0.13s | 0.43s | | | TensorRT | large | 0.03s | 0.07s | 0.19s | 0.63s | | ## 6 Conclusion Stable Audio 3 is a family of fast latent diffusion models (small, medium, large) for instrumental music and sound effects generation and editing. The models pair a semantic-acoustic autoencoder ($4096\times$ downsampling) with a diffusion transformer trained via flow matching, distillation warmup, and adversarial post-training. With only 8 ping-pong sampling steps at inference, they produce up to 6m 20s of stereo audio at 44.1 kHz in under 2s on an H200 GPU. On instrumental music, Stable Audio 3 models improve musicality over prior work and outperform existing open-weight baselines. On sound effects, they likewise set a new state-of-the-art among open-weight systems. Our models natively support variable-length generation and inpainting-based editing, covering single- and multi-segment edits as well as continuation. Stable Audio 3 models are trained exclusively on licensed and Creative Commons data, and we release small and medium weights. small runs on a MacBook Pro M4 CPU, and medium fits on consumer GPUs with as little as 8 GB of VRAM, putting both models within reach of typical research and creative workflows. ## References - [1] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank (2023) MusicLM: generating music from text. arXiv preprint. Cited by: [§1](#S1.p1.1). - [2] M. S. Albergo and E. Vanden-Eijnden (2023) Building normalizing flows with stochastic interpolants. In ICLR, Cited by: [§1](#S1.p5.1). - [3] M. S. Burtsev, Y. Kuratov, A. Peganov, and G. V. Sapunov (2020) Memory Transformer. arXiv preprint. Cited by: [6th item](#S1.I1.i6.p1.1), [§2.2](#S2.SS2.p1.1). - [4] J. Chen, C. Ge, E. Xie, Y. Wu, L. Yao, X. Ren, Z. Wang, P. Luo, H. Lu, and Z. Li (2024) Pixart-$\sigma$: weak-to-strong training of diffusion transformer for 4k text-to-image generation. In ECCV, Cited by: [§1.1](#S1.SS1.SSS0.Px2.p1.1), [§2.2](#S2.SS2.SSS0.Px1.p2.1), [§2.2](#S2.SS2.SSS0.Px2.p1.34), [§2.2](#S2.SS2.p1.1). - [5] J. Chen, S. Xue, Y. Zhao, J. Yu, S. Paul, J. Chen, H. Cai, E. Xie, and S. Han (2025) SANA-sprint: one-step diffusion with continuous-time consistency distillation. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [6] K. Chen, Y. Wu, H. Liu, M. Nezhurina, T. Berg-Kirkpatrick, and S. Dubnov (2024) MusicLDM: enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. In ICASSP, Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1](#S1.p1.1). - [7] T. Chen (2023) On the importance of noise scheduling for diffusion models. arXiv preprint. Cited by: [§3.1](#S3.SS1.SSS0.Px2.p1.5). - [8] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez (2023) Simple and controllable Music Generation. In NeurIPS, Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1](#S1.p1.1). - [9] M. Cuturi (2013) Sinkhorn distances: lightspeed computation of optimal transport. In NeurIPS, Cited by: [§3.2](#S3.SS2.SSS0.Px2.p1.7). - [10] T. Dao (2024) FlashAttention-2: faster attention with better parallelism and work partitioning. In ICLR, Cited by: [§3.1](#S3.SS1.SSS0.Px1.p1.1). - [11] T. Darcet, M. Oquab, J. Mairal, I. Misra, and H. Jegou (2024) Vision Transformers need registers. In ICLR, Cited by: [6th item](#S1.I1.i6.p1.1). - [12] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi (2023) High fidelity neural audio compression. Transactions on Machine Learning Research. Cited by: [§1](#S1.p4.4). - [13] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, et al. (2024) Scaling Rectified Flow Transformers for high-resolution image synthesis. In ICML, Cited by: [1st item](#S3.I1.i1.p1.3), [§3.1](#S3.SS1.SSS0.Px2.p1.5), [§3.2](#S3.SS2.SSS0.Px3.p1.5). - [14] Z. Evans, C. J. Carr, J. Taylor, S. H. Hawley, and J. Pons (2024) Fast timing-conditioned Latent Audio Diffusion. In ICML, Cited by: [§1.1](#S1.SS1.SSS0.Px2.p1.1), [§1](#S1.p1.1), [§1](#S1.p2.1), [§3.1](#S3.SS1.p1.1). - [15] Z. Evans, J. D. Parker, C. J. Carr, Z. Zukowski, J. Taylor, and J. Pons (2024) Long-form music generation with latent diffusion. In ISMIR, Cited by: [§1.1](#S1.SS1.SSS0.Px2.p1.1), [§1](#S1.p1.1), [§1](#S1.p2.1), [§3.1](#S3.SS1.p1.1), [1st item](#S5.I2.i1.p1.1). - [16] Z. Evans, J. D. Parker, C. J. Carr, Z. Zukowski, J. Taylor, and J. Pons (2024) Stable audio open. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1.1](#S1.SS1.SSS0.Px3.p1.1), [2nd item](#S5.I2.i2.p1.1), [3rd item](#S5.I2.i3.p1.1). - [17] H. F. García, O. Nieto, J. Salamon, B. Pardo, and P. Seetharaman (2025) Sketch2Sound: controllable audio generation via time-varying signals and sonic imitations. In ICASSP, Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [18] H. F. García, P. Seetharaman, R. Kumar, and B. Pardo (2023) VampNet: music generation via masked acoustic token modeling. In ISMIR, Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [19] D. Ghosal, N. Majumder, A. Mehrish, and S. Poria (2023) Text-to-audio generation using instruction-tuned LLM and latent diffusion model. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1). - [20] J. Gong, Y. Song, W. Zhao, S. Wang, S. Xu, J. Guo, and X. Yang (2026) ACE-Step 1.5: pushing the boundaries of open-source music generation. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1.1](#S1.SS1.SSS0.Px4.p1.1), [§1](#S1.p1.1), [6th item](#S5.I2.i6.p1.1). - [21] A. Gui, H. Gamper, S. Braun, and D. Emmanouilidou (2024) Adapting Frechet Audio Distance for generative music evaluation. In ICASSP, Cited by: [1st item](#S5.I3.i1.p1.1). - [22] M. Gutmann and A. Hyvärinen (2010) Noise-contrastive estimation: a new estimation principle for unnormalized statistical models. In AISTATS, Cited by: [§3.4](#S3.SS4.SSS0.Px4.p2.1). - [23] G. Hadjeres, M. Ferras, K. Koutini, B. Weck, A. Bittar, T. Hummel, Z. Lahrichi, H. Missoum, J. Serrà, and Y. Mitsufuji (2026) Woosh: a sound effects foundation model. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1), [§2.2](#S2.SS2.SSS0.Px1.p2.1), [8th item](#S5.I2.i8.p1.1), [9th item](#S5.I2.i9.p1.1). - [24] B. Han, J. Dai, W. Hao, X. He, D. Guo, J. Chen, Y. Wang, Y. Qian, and X. Song (2024) InstructME: an instruction guided music edit framework with latent diffusion models. In IJCAI, Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [25] A. Henry, P. R. Dachapally, S. Pawar, and Y. Chen (2020) Query-key normalization for transformers. arXiv preprint. Cited by: [§2.2](#S2.SS2.SSS0.Px1.p2.1), [§2.2](#S2.SS2.SSS0.Px1.p3.1), [§2.2](#S2.SS2.SSS0.Px5.p3.2). - [26] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In NeurIPS, Cited by: [§1](#S1.p5.1). - [27] E. Hoogeboom, J. Heek, and T. Salimans (2023) Simple diffusion: end-to-end diffusion for high resolution images. In ICML, Cited by: [§3.1](#S3.SS1.SSS0.Px2.p1.5). - [28] N. Huang, A. Gokaslan, V. Kuleshov, and J. Tompkin (2024) The gan is dead; long live the gan! a modern baseline gan. In ICML Workshop on Structured Probabilistic Inference and Generative Modeling, Cited by: [§3.4](#S3.SS4.SSS0.Px3.p2.1). - [29] C.-Y. Hung, N. Majumder, Z. Kong, A. Mehrish, A. A. Bagherzadeh, C. Li, R. Valle, B. Catanzaro, and S. Poria (2024) TangoFlux: super fast and faithful text to audio generation with flow matching and clap-ranked preference optimization. arXiv preprint. Cited by: [10th item](#S5.I2.i10.p1.1). - [30] Y. Jiang, H. Chen, Z. Ning, J. Yao, Z. Han, D. Wu, M. Meng, J. Luan, Z. Fu, and L. Xie (2025) DiffRhythm 2: efficient and high fidelity song generation via block flow matching. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1.1](#S1.SS1.SSS0.Px2.p1.1), [7th item](#S5.I2.i7.p1.1). - [31] A. Jolicoeur-Martineau (2018) The relativistic discriminator: a key element missing from standard gan. arXiv preprint. Cited by: [§3.4](#S3.SS4.SSS0.Px3.p2.1). - [32] K. Jordan (2024) Muon: an optimizer for hidden layers in neural networks. Note: [https://kellerjordan.github.io/posts/muon/](https://kellerjordan.github.io/posts/muon/) Cited by: [§3.5](#S3.SS5.SSS0.Px6.p1.6). - [33] M. Kang, R. Zhang, C. Barnes, S. Paris, S. Kwak, J. Park, E. Shechtman, J.-Y. Zhu, and T. Park (2024) Distilling diffusion models into conditional gans. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [34] K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi (2019) Fréchet Audio Distance: a reference-free metric for evaluating music enhancement algorithms. In Interspeech, Cited by: [1st item](#S5.I3.i1.p1.1). - [35] C. D. Kim, B. Kim, H. Lee, and G. Kim (2019) AudioCaps: generating captions for audios in the wild. In NAACL, Cited by: [§5.1](#S5.SS1.p13.1). - [36] D. Kim, C.-H. Lai, W.-H. Liao, N. Murata, Y. Takida, T. Uesaka, Y. He, Y. Mitsufuji, and S. Ermon (2023) Consistency trajectory models: learning probability flow ODE trajectory of diffusion. In ICLR, Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [37] S. Kim, G. Kim, S. Yagishita, D. Han, J. Im, and Y. Sung (2025) Enhancing diffusion-based music generation performance with lora. Applied Sciences. Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [38] D. Kingma, T. Salimans, B. Poole, and J. Ho (2021) Variational diffusion models. Advances in neural information processing systems. Cited by: [§4](#S4.p3.6). - [39] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley (2020) Panns: large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing. Cited by: [§3.5](#S3.SS5.SSS0.Px1.p1.1). - [40] F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y. Taigman, and Y. Adi (2023) AudioGen: textually guided audio generation. In ICLR, Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1.1](#S1.SS1.SSS0.Px3.p1.1). - [41] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar (2023) High-fidelity audio compression with improved RVQGAN. In NeurIPS, Cited by: [§1](#S1.p4.4). - [42] B. F. Labs (2024) FLUX. Note: [https://github.com/black-forest-labs/flux](https://github.com/black-forest-labs/flux) Cited by: [§2.2](#S2.SS2.SSS0.Px2.p1.34), [1st item](#S3.I1.i1.p1.3), [§3.2](#S3.SS2.SSS0.Px3.p1.5). - [43] G. L. Lan, B. Shi, Z. Ni, S. Srinivasan, A. Kumar, B. Ellis, D. Kant, V. Nagaraja, E. Chang, W.-N. Hsu, et al. (2024) High fidelity text-guided music editing via single-stage flow matching. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1), [§3.2](#S3.SS2.SSS0.Px2.p1.7). - [44] M. Levy, B. Di Giorgi, F. Weers, A. Katharopoulos, and T. Nickson (2023) Controllable music production with diffusion models and guidance gradients. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [45] P. Li, B. Chen, Y. Yao, Y. Wang, A. Wang, and A. Wang (2024) JEN-1: text-guided universal music generation with omnidirectional diffusion models. In IEEE CAI, Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [46] Z. Li, J. Zhang, Q. Lin, J. Xiong, Y. Long, X. Deng, Y. Zhang, X. Liu, M. Huang, Z. Xiao, et al. (2024) Hunyuan-dit: a powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px2.p1.1). - [47] S. Lin, X. Xia, Y. Ren, C. Yang, X. Xiao, and L. Jiang (2025) Diffusion adversarial post-training for one-step video generation. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [48] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow Matching for generative modeling. In ICLR, Cited by: [§1](#S1.p5.1), [§3.2](#S3.SS2.p1.3). - [49] A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo, et al. (2024) Deepseek-v2: a strong, economical, and efficient mixture-of-experts language model. arXiv preprint. Cited by: [§2.2](#S2.SS2.SSS0.Px1.p2.1), [§2.2](#S2.SS2.SSS0.Px1.p3.1), [§2.2](#S2.SS2.SSS0.Px5.p2.1), [§2.2](#S2.SS2.SSS0.Px5.p3.2). - [50] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley (2023) AudioLDM: text-to-audio generation with latent diffusion models. In ICML, Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1](#S1.p1.1). - [51] H. Liu, R. Huang, Y. Liu, H. Cao, J. Wang, X. Cheng, S. Zheng, and Z. Zhao (2024) AudioLCM: text-to-audio generation with latent consistency models. In ACM MM, Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [52] H. Liu, Y. Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y. Wang, W. Wang, Y. Wang, and M. D. Plumbley (2024) AudioLDM 2: learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing. Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1](#S1.p1.1). - [53] R. Liu, C.-Y. Hung, N. Majumder, T. Gautreaux, A. A. Bagherzadeh, C. Li, D. Herremans, and S. Poria (2025) JAM: a tiny flow-based song generator with fine-grained controllability and aesthetic alignment. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1.1](#S1.SS1.SSS0.Px4.p1.1), [§5.1](#S5.SS1.p4.1). - [54] X. Liu, C. Gong, and Q. Liu (2022) Flow straight and fast: learning to generate and transfer data with rectified flow. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1), [§1](#S1.p5.1), [§3.2](#S3.SS2.SSS0.Px2.p1.7), [§3.2](#S3.SS2.p1.3), [§3.3](#S3.SS3.p3.7), [§3.3](#S3.SS3.p4.8). - [55] C. Lu and Y. Song (2024) Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [56] E. Luhman and T. Luhman (2021) Knowledge Distillation in iterative generative models for improved sampling speed. arXiv preprint. Cited by: [§1](#S1.p5.1), [§3.3](#S3.SS3.p2.9). - [57] N. Majumder, C.-Y. Hung, D. Ghosal, W.-N. Hsu, R. Mihalcea, and S. Poria (2024) Tango 2: aligning diffusion-based text-to-audio generations through direct preference optimization. In ACM MM, Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1). - [58] I. Manco, B. Weck, S. Doh, M. Won, Y. Zhang, D. Bogdanov, et al. (2023) The Song Describer Dataset: a corpus of audio captions for music-and-language evaluation. In Machine Learning for Audio Workshop, NeurIPS, Cited by: [1st item](#S5.I4.i1.p1.1). - [59] Z. Ning, H. Chen, Y. Jiang, C. Hao, G. Ma, S. Wang, J. Yao, and L. Xie (2025) DiffRhythm: blazingly fast and embarrassingly simple end-to-end full-length song generation with latent diffusion. arXiv preprint. Note: Weights available at [https://huggingface.co/ASLP-lab/DiffRhythm-full](https://huggingface.co/ASLP-lab/DiffRhythm-full) Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1). - [60] Z. Novack, Z. Evans, Z. Zukowski, J. Taylor, C. J. Carr, J. Parker, A. Al-Sinan, G. M. Iodice, J. McAuley, T. Berg-Kirkpatrick, and J. Pons (2025) Fast text-to-audio generation with Adversarial Post-Training. In WASPAA, Note: WASPAA 2025 Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1.1](#S1.SS1.SSS0.Px4.p1.1), [§1.1](#S1.SS1.SSS0.Px5.p1.1), [§1](#S1.p5.1). - [61] Z. Novack, J. Mcauley, T. Berg-Kirkpatrick, and N. J. Bryan (2024) DITTO: diffusion inference-time t-optimization for music generation. In ICML, Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [62] Z. Novack, J. McAuley, T. Berg-Kirkpatrick, and N. Bryan (2024) DITTO-2: distilled diffusion inference-time t-optimization for music generation. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [63] Z. Novack, G. Zhu, J. Casebeer, J. McAuley, T. Berg-Kirkpatrick, and N. J. Bryan (2025) Presto! distilling steps and layers for accelerating music generation.. In ICLR, Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1), [§3.4](#S3.SS4.SSS0.Px4.p2.1). - [64] Z. Novack, Z. Zukowski, C. J. Carr, J. Parker, Z. Evans, J. Taylor, T. Berg-Kirkpatrick, J. McAuley, and J. Pons (2026) Low-resource guidance for controllable latent audio diffusion. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [65] J. D. Parker, Z. Evans, C. J. Carr, Z. Zukowski, J. Taylor, M. Rice, and J. Pons (2025) SAME: a semantically-aligned music autoencoder. Technical report Cited by: [§1.1](#S1.SS1.SSS0.Px3.p1.1), [Figure 5](#S2.F5), [§2.1](#S2.SS1.p1.2), [§2.1](#S2.SS1.p4.1), [§2.1](#S2.SS1.p5.1). - [66] J. D. Parker, J. Spijkervet, K. Kosta, F. Yesiler, B. Kuznetsov, J.-C. Wang, M. Avent, J. Chen, and D. Le (2024) Stemgen: a music generation model that listens. In ICASSP, Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [67] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In ICCV, Cited by: [6th item](#S1.I1.i6.p1.1), [§2.2](#S2.SS2.SSS0.Px1.p2.1), [§2.2](#S2.SS2.SSS0.Px2.p1.34), [§2.2](#S2.SS2.p1.1). - [68] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach (2023) Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px2.p1.1). - [69] J. Pons, Z. Zukowski, J. D. Parker, C. J. Carr, J. Taylor, and Z. Evans (2025) Music and artificial intelligence: artistic trends. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [70] Y. Ren, X. Xia, Y. Lu, J. Zhang, J. Wu, P. Xie, X. Wang, and X. Xiao (2024) Hyper-SD: trajectory segmented consistency model for efficient image synthesis. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [71] S. Rouard, Y. Adi, J. Copet, A. Roebel, and A. Défossez (2024) Audio conditioning for music generation via discrete bottleneck features. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [72] T. Salimans and J. Ho (2022) Progressive distillation for fast sampling of diffusion models. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1), [§1](#S1.p5.1). - [73] A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, and R. Rombach (2024) Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [74] A. Sauer, D. Lorenz, A. Blattmann, and R. Rombach (2024) Adversarial diffusion distillation. In ECCV, Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [75] F. Schneider, O. Kamal, Z. Jin, and B. Schölkopf (2024) Moûsai: text-to-music generation with long-context latent diffusion. In ACL, Cited by: [§1](#S1.p1.1). - [76] P. Seetharaman, O. Nieto, and J. Salamon (2026) Generative audio extension and morphing. In ICASSP, Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [77] N. Shazeer (2020) Glu variants improve transformer. arXiv preprint. Cited by: [§2.2](#S2.SS2.SSS0.Px1.p5.4). - [78] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever (2023) Consistency models. In ICML, Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1), [§3.3](#S3.SS3.p5.20). - [79] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2021) Score-based generative modeling through stochastic differential equations. In ICLR, Cited by: [§1](#S1.p5.1). - [80] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu (2024) Roformer: enhanced transformer with rotary position embedding. Neurocomputing. Cited by: [§2.1](#S2.SS1.p1.2), [§2.2](#S2.SS2.SSS0.Px1.p2.1). - [81] O. Tal, A. Ziv, I. Gat, F. Kreuk, and Y. Adi (2024) Joint audio and symbolic conditioning for temporally controlled text-to-music generation. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [82] S. Tong, B. Zheng, Z. Wang, B. Tang, N. Ma, E. Brown, J. Yang, R. Fergus, Y. LeCun, and S. Xie (2026) Scaling text-to-image diffusion transformers with representation autoencoders. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px3.p1.1). - [83] F.-Y. Wang, Z. Huang, A. W. Bergman, D. Shen, P. Gao, M. Lingelbach, K. Sun, W. Bian, G. Song, Y. Liu, H. Li, and X. Wang (2024) Phased consistency model. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [84] J. Wang (2025) Audio palette: a diffusion transformer with multi-signal conditioning for controllable foley synthesis. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [85] K. Wang, Z. Wu, D. Zhou, R. Lin, J. Dai, and T. Jiang (2025) Back to ear: perceptually driven high fidelity music reconstruction. arXiv preprint. Cited by: [§1](#S1.p4.4). - [86] Y. Wang, Z. Ju, X. Tan, L. He, Z. Wu, J. Bian, and S. Zhao (2023) AUDIT: audio editing by following instructions with latent diffusion models. In NeurIPS, Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [87] S.-L. Wu, C. Donahue, S. Watanabe, and N. J. Bryan (2024) Music ControlNet: multiple time-varying controls for music generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing. Cited by: [§1.1](#S1.SS1.SSS0.Px4.p1.1). - [88] Z. Xiao, K. Kreis, and A. Vahdat (2022) Tackling the generative learning trilemma with denoising diffusion GANs. In ICLR, Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [89] Y. Xu, W. Nie, and A. Vahdat (2025) One-step diffusion models with $f$-divergence distribution matching. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [90] Y. Xu, Y. Zhao, Z. Xiao, and T. Hou (2024) Ufogen: you forward once large scale text-to-image generation via diffusion gans. In CVPR, Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [91] D. Yang, Y. Xie, Y. Yin, Z. Wang, X. Yi, G. Zhu, et al. (2026) HeartMuLa: a family of open sourced music foundation models. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1.1](#S1.SS1.SSS0.Px4.p1.1), [§1](#S1.p1.1), [§5.1](#S5.SS1.p4.1). - [92] T. Ye, L. Li, G. Huang, S. Xia, D. Li, and F. Wei (2025) Differential Transformer. In ICLR, Cited by: [6th item](#S1.I1.i6.p1.1), [§2.1](#S2.SS1.p1.2), [§2.2](#S2.SS2.SSS0.Px5.p1.4), [§2.2](#S2.SS2.p1.1), [§2.2](#S2.SS2.p4.1). - [93] T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, and W. T. Freeman (2024) Improved distribution matching distillation for fast image synthesis. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px5.p1.1). - [94] R. Yuan, H. Lin, S. Guo, G. Zhang, J. Pan, Y. Zang, et al. (2025) YuE: scaling open foundation models for long-form music generation. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1](#S1.p1.1), [§5.1](#S5.SS1.p4.1). - [95] C. Zhang, Y. Ma, Q. Chen, W. Wang, S. Zhao, Z. Pan, H. Wang, C. Ni, T. H. Nguyen, K. Zhou, Y. Jiang, C. Tan, Z. Gao, Z. Du, and B. Ma (2025) InspireMusic: integrating super resolution and large language model for high-fidelity long-form music generation. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px1.p1.1), [§1](#S1.p1.1). - [96] B. Zheng, N. Ma, S. Tong, and S. Xie (2025) Diffusion Transformers with Representation Autoencoders. arXiv preprint. Cited by: [§1.1](#S1.SS1.SSS0.Px3.p1.1). --- ## [Arize Phoenix Runbook](https://fitxers.usebrioso.com/internal/phoenix-runbook/) (Internal) # Arize Phoenix Runbook Distributed trace backend for the e2e generation pipeline (`apps/web` → `apps/worker`/`apps/runpod` → back), exposed to AI agents via the official Phoenix MCP server. ## Why this exists The pipeline that generates a track is distributed across three processes and a job's `traceId` is already stashed on claim for cross-service correlation (`packages/db/src/schema.ts`), but until this migration there was nowhere to actually look at a correlated trace: prod was PostHog-only, and the previous deep-span sink (`otelite`) was local-dev-only and never ran on the VPS. Phoenix replaces `otelite` as the OTLP/OpenInference-native trace backend for **both local dev and prod**, so a failing e2e run shows exactly which hop broke. PostHog is unchanged — product analytics and prod error/lifecycle reporting still go through it; Phoenix is additive for distributed/LLM tracing. This is a deliberate reversal of the earlier "no OTLP in prod" design (see `.agents/plans/2026-07-21-pr1-observability-foundation.md`) — OTLP now has a real prod backend. ## Architecture ``` apps/web ──┐ apps/worker ├──▶ OTLP/HTTP :6006/v1/traces ──▶ Arize Phoenix (podman quadlet) apps/runpod ┘ │ │ SQLite volume ▼ /mnt/data (brioso-phoenix-data) Agent (Claude Code, Cursor, ...) │ HTTP (Streamable HTTP, native to Phoenix) ▼ http://localhost:6006/mcp (dev) or https://ops.usebrioso.com/mcp (prod, via Caddy — see deploy/README.md's "Custom subdomains" section) ``` Phoenix serves the web UI, the OTLP-HTTP trace collector (`POST /v1/traces`), the REST/GraphQL API, and (as of Phoenix 19.0.0+) a **native MCP server at `/mcp`** — all on the same port (`6006` in-container, `4317` for OTLP-gRPC — unused here). Our pinned image tag (`version-19.6.0`, see `deploy/phoenix/brioso-phoenix.container`) already meets that floor. There is no separate MCP port or sidecar process. Earlier revisions of this doc had agents spawn `@arizeai/phoenix-mcp` locally over stdio (npx); that package now receives bug fixes only, per Arize — new capability lands in the Remote MCP Server built into Phoenix itself. Use the native `/mcp` HTTP endpoint instead: it's zero-install (any MCP-capable client speaks Streamable HTTP directly, no local process to spawn) and it's already reachable wherever the UI is (`:8001` in prod). ## VPS deployment (rootless Podman quadlet) Podman is the first container runtime on this host — everything else here is bare systemd units / binaries. Rootless + `loginctl enable-linger web-services` + loopback-only `PublishPort` keeps it consistent with the existing non-root/tailnet-gated posture (mirrors Drizzle Gateway / MCP SQLite). - `deploy/phoenix/brioso-phoenix.container` — quadlet, installed to `~web-services/.config/containers/systemd/`, managed via `systemctl --user` as `web-services` (not root — quadlets under a user's config dir run under that user's own systemd instance). - `deploy/phoenix/brioso-phoenix-data.volume` — named volume backing Phoenix's SQLite trace store at `/mnt/data` in the container. - `deploy/scripts/install-phoenix.sh` — installs podman + uidmap, provisions subuid/subgid ranges, enables lingering, installs the quadlets, seeds `/etc/brioso/phoenix.env` from the template, reloads and starts the unit. Idempotent — wired into `deploy/scripts/setup-observability.sh`. Persistence starts as SQLite-on-volume (matches the repo's SQLite-everywhere posture, no new Postgres dependency). `PHOENIX_SQL_DATABASE_URL` (Postgres) is the documented upgrade path if trace volume grows — add a second quadlet container and point Phoenix at it. ### Install / update ```bash sudo /opt/brioso/repo/deploy/scripts/install-phoenix.sh ``` Re-run to reconcile state (e.g. after bumping the pinned image tag in `brioso-phoenix.container`). To restart directly: ```bash sudo runuser -u web-services -- env XDG_RUNTIME_DIR=/run/user/$(id -u web-services) \ systemctl --user restart brioso-phoenix.service ``` ### Verify ```bash # Service status (as web-services) sudo runuser -u web-services -- env XDG_RUNTIME_DIR=/run/user/$(id -u web-services) \ systemctl --user status brioso-phoenix.service # Local health curl -sI http://localhost:6006/ # Over Caddy (prod) curl -sI https://ops.usebrioso.com/ ``` ### Custom subdomain (Caddy) Phoenix is reached in prod at `https://ops.usebrioso.com` via the shared Caddy instance — see `deploy/README.md`'s "Custom subdomains (Caddy)" section for the full DNS/Caddy/ACL setup (shared with Drizzle Gateway, MCP SQLite, and `apps/admin`). No path-splitting needed here: the UI, OTLP collector, and `/mcp` all share the same backend port (`6006`), so `ops.usebrioso.com` reverse-proxies the whole thing. ## Auth Primary control is loopback bind (`PublishPort=127.0.0.1:6006:6006`) + the Tailscale ACL on the shared Caddy IP (`autogroup:member` — see `deploy/README.md`). Additionally, `PHOENIX_ENABLE_AUTH=true` + `PHOENIX_SECRET` (from `/etc/brioso/phoenix.env`, mode 0600, owned by `web-services`) are set as defense-in-depth. Generate the secret with `openssl rand -hex 32`. ## Project routing by deployment environment `DEPLOYMENT_ENV` (`"prod"` | `"beta"`, default `"prod"`) selects which Phoenix project an app's traces land in: the prod Phoenix project is `default`, the beta project is `beta`. Beta servers must have `DEPLOYMENT_ENV=beta` set (beta's env file is `/opt/brioso/shared/beta.env`); prod needs nothing — absence is prod. If a beta box is missing the variable, its traces silently land in the `default` (prod) project. Filter accordingly when comparing environments. `PHOENIX_SECRET` and `PHOENIX_API_KEY` both live in Infisical under `/infra/phoenix` (prod env, same project as the app secrets) — nothing to write on the VPS by hand. `deploy-vps.sh`'s `sync_secrets()` syncs that folder into `/etc/brioso/phoenix.env` on every deploy and restarts `brioso-phoenix` (as `web-services`, via `systemctl --user`) only if a value actually changed, since env vars are read once at container start. See `deploy/README.md`'s "Custom subdomains (Caddy)" section for the full mechanics. The `/mcp` endpoint itself supports two auth modes on top of that: an interactive OAuth flow (Phoenix acts as its own authorization server, browser login on first connect — fine for local dev, first-tab-of-a-session UX) or a `PHOENIX_API_KEY` passed as an `Authorization: Bearer` header for headless/ scripted clients. Create the key from the UI (Settings → API Keys) once `PHOENIX_ENABLE_AUTH` is on, then store it in Infisical under `/infra/phoenix` — needed for the prod config below since agents there aren't sitting behind an interactive browser. ## Tailscale ACL setup (one-time, out-of-repo) Shared with Drizzle Gateway, MCP SQLite, and `apps/admin` — all four sit behind the same Caddy instance on one IP:port (`100.105.198.36:443`), and Tailscale ACLs are IP:port-scoped, not per-hostname. The ACL is `autogroup:member` (this tailnet is just the two engineer accounts); see `deploy/README.md`'s "Custom subdomains (Caddy)" section for the full rationale and the exact ACL JSON. Phoenix's own `PHOENIX_ENABLE_AUTH` + `PHOENIX_API_KEY` (below) is the actual access-control boundary for `/mcp`, same as Drizzle Gateway's passcode is for its own UI. ## Client configuration (agent-first) Phoenix's `/mcp` endpoint speaks Streamable HTTP directly — every client config below is a remote `url`, no local process to spawn or version to pin. ### Local dev ```bash claude mcp add --transport http phoenix http://localhost:6006/mcp ``` Or, for Cursor/VS Code-style JSON configs: ```json { "mcpServers": { "phoenix": { "url": "http://localhost:6006/mcp" } } } ``` First connection triggers a browser OAuth prompt (`PHOENIX_ENABLE_AUTH` is prod-only, so this is a no-op locally unless you've turned it on there too). ### Remote (prod, via Caddy) ```bash claude mcp add --transport http phoenix-prod \ https://ops.usebrioso.com/mcp \ --header "Authorization: Bearer " ``` Or as JSON: ```json { "mcpServers": { "phoenix-prod": { "url": "https://ops.usebrioso.com/mcp", "headers": { "Authorization": "Bearer " } } } } ``` The Bearer header skips the interactive OAuth flow, which matters for non-interactive/scripted agent use. The Tailscale hop happens transparently inside the client's own HTTPS request to the `url` (gated by the `autogroup:member` ACL on Caddy's shared IP, see `deploy/README.md`) — no `mcp-remote` bridge needed, since Phoenix already speaks HTTP natively. Works from any MCP-capable client (Claude Code, Cursor, Claude Desktop, VS Code). ## Querying a failing e2e trace 1. Get the job's `trace_id` from `jobs.trace_id` (Drizzle Studio or MCP SQLite, both under `db.usebrioso.com`). 2. Call `getSpans(project_identifier="default", trace_id=[])` on the Phoenix MCP tools — pulls every span of that one trace in a single call. `listProjectTraces` is the complementary tool for browsing traces when you don't have a `trace_id` up front (e.g. by time range). 3. Read each span's `parent_id`/`start_time`/`end_time` to reconstruct hop order: `POST /jobs` (web) → `chat`/`invoke_agent` (ai-sdk refine) → `dispatch.runpod` → `runpod.handler` (or the home-worker equivalent, `worker.process_job`). The hop with the error status or longest duration is the one to dig into first. ## Troubleshooting | Symptom | Check | |---|---| | Quadlet won't start | `journalctl --user -u brioso-phoenix -n 50` (as `web-services`, with `XDG_RUNTIME_DIR` set). Common causes: subuid/subgid not provisioned, linger not enabled, image pull failed. | | `/run/user/` missing | Linger enabled but the user manager never started — `systemctl start user@.service`. | | No spans arriving | `TRACING_ENABLED=false` or `PHOENIX_COLLECTOR_ENDPOINT` unset/wrong in the emitting service's env. Check app boot logs for `[obs-js] otel registered`. | | Generic spans instead of rich LLM spans | `packages/obs-js/src/otel.ts` isn't wrapping the exporter in `OpenInferenceSimpleSpanProcessor` — worker/runpod spans are generic by design (no LLM SDK there), web spans should carry model/token attributes. | | runpod/worker span looks blank in UI | `attributes: {}` and `span_kind: "UNKNOWN"` on `runpod.handler`/`worker.process_job`/`dispatch.runpod` render as empty rows with no preview text, and get filtered out of any LLM/typed-span view since their kind is UNKNOWN — the raw trace is still intact underneath. Check whether the call site still passes `attributes=`/`attributes` to `with_span`/`withSpan` (input/output `.value` + `openinference.span.kind`) — see `apps/runpod/handler.py`, `apps/worker/src/apps/worker/main.py`, `packages/dispatch/src/runpodClient.ts`. | | MCP client gets connection refused | Wrong `url` (missing `/mcp` suffix), Phoenix not running, `PHOENIX_ENABLE_MCP_SERVER=false`, or (remote) the querying device isn't on the tailnet / ACL not applied. | | MCP client 401s (prod) | `PHOENIX_ENABLE_AUTH=true` but the `Authorization: Bearer` header is missing/stale, or you hit the endpoint without a header and declined the OAuth prompt — mint a new key from the Phoenix UI. | | MCP client 404s on `/mcp` | Phoenix image predates 19.0.0 — check the pinned tag in `deploy/phoenix/brioso-phoenix.container`. | --- ## [Activation KPI — PostHog](https://fitxers.usebrioso.com/internal/posthog-activation-kpi/) (Internal) # Activation KPI — PostHog ## Deployment status The existing dashboard is in PostHog EU. The configuration below is the required operator change: create and save its new headline insight, replace the existing headline tile, and add the annotation. Until that work is completed in PostHog, the external dashboard is not yet the thread-level satisfaction measure described here. - Dashboard URL: https://eu.posthog.com/project/219634/dashboard/895474 - Headline thread-completion insight: required operator action — create/save the HogQL insight described below, then replace the prior cross-path headline tile with it. - Anonymous-path insight URL: https://eu.posthog.com/project/219634/insights/dbP85eR5 - Existing signed-in insight URL: https://eu.posthog.com/project/219634/insights/OWNKKg8p (requires replacement or reconfiguration before it represents the `otp.verified` → `prompt.submitted` → generation → download diagnostic). - Reliable headline data begins: the release containing this change. The secondary anonymous path begins at BRI-276; strict-order prompt paths begin at the dual-path release. ## Dashboard configuration Required PostHog operator actions: 1. Create and save the headline HogQL insight specified below. 2. Replace the dashboard's current headline tile with that saved insight. 3. Add the exact dashboard annotation specified below at the release boundary. 4. Reconfigure or replace both existing secondary insights to the exact sequences specified below before presenting them as diagnostics. - Dashboard name: `Activation KPI` - Headline insight: `Prompt → generation → audio delivery started` per thread - Supporting insights: `Anonymous prompt → OTP → generation → download` and `OTP → signed-in prompt → generation → download` (secondary acquisition and identity diagnostics only) - Headline insight type: saved HogQL insight; use a native strict-order funnel only if its property aggregation can faithfully deduplicate by `thread_id` - Headline aggregation: completed threads, deduplicated by event property `thread_id` — never unique persons - Date range: last 30 days - Conversion window: 7 days - Conversion reference: total completion from the first step, retaining per-step conversion/drop-off where supported The headline thread-completion flow is: 1. `prompt.submitted` 2. `generation.started` 3. `audio.delivery_started` Every event must have the same non-empty `thread_id`. The strict sequence must complete within seven days of `prompt.submitted`. `slug` identifies the take; `thread_id` identifies the root conversation that owns that take. A person may have several threads, and a thread may have several take slugs, so neither person nor slug is the headline deduplication key. Use this as the fallback saved HogQL insight query until native funnel property aggregation has been verified to preserve the same strict per-`thread_id` behavior. It anchors candidate `prompt.submitted` events to the last 30 days and returns one row containing the number of completed threads: ```sql WITH eligible_events AS ( SELECT properties.thread_id AS thread_id, timestamp, event FROM events WHERE timestamp >= now() - INTERVAL 30 DAY AND event IN ('prompt.submitted', 'generation.started', 'audio.delivery_started') AND properties.thread_id IS NOT NULL AND properties.thread_id != '' AND ( person.properties.email IS NULL OR ( person.properties.email NOT IN ('usebrioso@gmail.com', 'carloshpiedra@gmail.com') AND person.properties.email NOT LIKE '%@brioso.test' AND person.properties.email NOT LIKE '%@usebrioso.com' ) ) ), prompt_candidates AS ( SELECT thread_id, timestamp AS prompt_at FROM eligible_events WHERE event = 'prompt.submitted' ), candidate_generations AS ( SELECT p.thread_id, p.prompt_at, min(e.timestamp) AS generation_at FROM prompt_candidates AS p INNER JOIN eligible_events AS e ON e.thread_id = p.thread_id WHERE e.event = 'generation.started' AND e.timestamp > p.prompt_at GROUP BY p.thread_id, p.prompt_at ), completed_candidates AS ( SELECT g.thread_id, g.prompt_at, min(e.timestamp) AS delivery_at FROM candidate_generations AS g INNER JOIN eligible_events AS e ON e.thread_id = g.thread_id WHERE e.event = 'audio.delivery_started' AND e.timestamp > g.generation_at AND e.timestamp <= g.prompt_at + INTERVAL 7 DAY GROUP BY g.thread_id, g.prompt_at ) SELECT count(DISTINCT thread_id) AS completed_threads FROM completed_candidates ``` `prompt_candidates` evaluates every eligible prompt rather than only the first prompt in a thread. `candidate_generations` and `completed_candidates` preserve strict `>` ordering for each candidate, while the final `count(DISTINCT thread_id)` deduplicates a thread that has any valid sequence. This supports later takes without counting a completed thread more than once. Required annotation text: “Thread completion data begins with the thread-level delivery release (2026-08-14); count strict prompt → generation → audio delivery completion once per `thread_id` within seven days. Earlier download-based activation data is not comparable.” The exact secondary person-level sequences are: 1. Anonymous acquisition/identity diagnostic: `prompt.otp_sent` → `otp.verified` → `generation.started` → `download`. The existing anonymous saved insight requires operator reconfiguration or replacement before it is treated as this exact sequence. 2. Signed-in acquisition/identity diagnostic: `otp.verified` → `prompt.submitted` → `generation.started` → `download`. The existing signed-in saved insight requires operator reconfiguration or replacement before it is treated as this exact sequence. These are separate because their entry events are alternative journeys, not sequential steps. Exclude internal/test people whose email is `usebrioso@gmail.com` or matches either `@brioso.test` or `@usebrioso.com`; retain the dashboard exclusion for `carloshpiedra@gmail.com`. ## Event contract `prompt.submitted` is emitted after durable job insertion and before dispatch, so a fast worker claim cannot place `generation.started` before the prompt in a strict-order funnel. It has `slug`, `runner`, the resolved user ID, and must carry the root conversation `thread_id` for headline measurement. `generation.started` is emitted once from a successful `JobsQueue.claimForRunner` claim for both `home` and `runpod`. It has `slug`, `runner`, `traceId`, the resolved `userId` when the job email matches a user, and must carry `thread_id`. `audio.delivery_started` is the delivery-start contract used by the headline. It must carry `thread_id` and the take `slug`, so the full sequence can be strictly matched to one conversation thread. `download` is a retired, browser-originated event; do not combine it with `audio.delivery_started` in a headline funnel or completion count. It remains useful only for the explicitly labelled browser/acquisition diagnostics. `prompt.otp_sent` is emitted server-side only after a new anonymous job has been created and its OTP issued. It uses the same user ID as `otp.verified`, so browser tracking blockers cannot break the person funnel. It is forward-looking from this release. `prompt.otp_required` remains a best-effort browser-only UI-friction diagnostic and is intentionally not a required funnel step. ## Signed-in path `OTP → signed-in prompt → generation → download` measures users who authenticate before submitting a prompt. Together with the anonymous path, it remains a person-level, browser/acquisition and identity diagnostic: strict order, unique persons, seven-day window, 30-day range, and internal-account exclusions. It is not the headline satisfaction metric and must not be compared directly with thread completion. --- ## [PostHog KPI dashboards](https://fitxers.usebrioso.com/internal/posthog-kpi-dashboards/) (Internal) # PostHog KPI dashboards PostHog Cloud is hosted in the EU. Use `posthog-cli --host https://eu.posthog.com ...` for every CLI command. Authentication is local to the operator and must never be committed. ## Activation KPI - Required operator actions: create/save the HogQL headline insight described in `posthog-activation-kpi.md`, replace the existing headline tile with it, and add that document's required release-boundary annotation. This document specifies the target state; it does not assert those external PostHog changes have already been made. - Dashboard: https://eu.posthog.com/project/219634/dashboard/895474 - Headline target: saved HogQL thread-completion insight (`prompt.submitted` → `generation.started` → `audio.delivery_started`), grouped and deduplicated by event property `thread_id`, not unique person. See `posthog-activation-kpi.md` for the required strict sequence and query logic. - Existing anonymous insight: https://eu.posthog.com/project/219634/insights/dbP85eR5 (requires replacement or reconfiguration before it represents `prompt.otp_sent` → `otp.verified` → `generation.started` → `download` as the secondary acquisition/identity diagnostic). - Existing signed-in insight: https://eu.posthog.com/project/219634/insights/OWNKKg8p (requires replacement or reconfiguration before it represents `otp.verified` → `prompt.submitted` → `generation.started` → `download` as the secondary acquisition/identity diagnostic). - Headline configuration: strict order within one `thread_id`, 7-day conversion window, last 30 days, and total completion from the first step. Use saved HogQL when native property aggregation cannot faithfully deduplicate threads; do not assume native funnel property support has this behavior. - Secondary-person-funnel configuration: strict order, unique persons, 7-day conversion window, last 30 days, and total conversion from the first step. - Exclusions: person email does not contain `@brioso.test` or `@usebrioso.com`, and is neither `usebrioso@gmail.com` nor `carloshpiedra@gmail.com`. - Data note: headline thread-completion data is reliable only from the 2026-08-14 thread-level delivery release. The anonymous path is reliable from BRI-276; strict-order prompt submission person funnels are reliable after the dual-path release, which emits `prompt.submitted` before dispatch. Browser `download` is retired from the headline and must not be combined with `audio.delivery_started`. ## Generation Operations KPI - Dashboard: https://eu.posthog.com/project/219634/dashboard/895497 - Generation outcomes: https://eu.posthog.com/project/219634/insights/bNWeOjCs (`job.completed` and `job.failed`, daily event totals, last 30 days). - P95 generation duration: https://eu.posthog.com/project/219634/insights/5IUaOpMy (`job.completed.duration_s`, daily p95, seconds, last 30 days). - Generation starts by runner: https://eu.posthog.com/project/219634/insights/2NTXvVZe (`generation.started`, daily event totals broken down by `runner`, last 30 days). ## Business & Engagement KPI - Dashboard: https://eu.posthog.com/project/219634/dashboard/895500 - Completed checkouts: https://eu.posthog.com/project/219634/insights/JkcG9YCg (`checkout.completed`, daily event totals, last 30 days). - Unique prompt submitters: https://eu.posthog.com/project/219634/insights/ZSbbwCFx (`prompt.submitted`, daily unique persons, last 30 days). - Browser download initiations: https://eu.posthog.com/project/219634/insights/XM2Rkjl9 (`download`, daily event totals, last 30 days; a legacy browser diagnostic, reliable only from BRI-276 and not a satisfaction/activation headline). - All three saved insights use PostHog's test-account filter. ## Known gaps The current event schema does not include a checkout-start event, subscription state, or transaction amount/currency. Do not infer checkout conversion, revenue, MRR, or retention revenue from these dashboards. Add those contracts in separately planned instrumentation work before adding such tiles. --- ## [Brioso — Product Definition](https://fitxers.usebrioso.com/internal/prd/) (Internal) # Brioso — Product Definition Brioso is **an audio generation platform**: open-weight models (Stable Audio 3) on GPU cloud, priced to be a trivial line item rather than a negotiated cost, producing royalty-free output with real legal affordances (a rights-clear license, not a EULA maze), delivered in the formats a buyer actually needs, through a studio-like UI that works for people who don't think in prompts. Four product lines run on that platform today. See [`roadmap.md`](roadmap.md) for the engineering Iteration/Quarter plan. ## The four product lines | Line | Sells | Price | Status | |---|---|---|---| | Creator custom generation | bespoke track from a brief | free (commercial use, credit required) or Pro-unlocked (exclusive, tag-free) | live | | Royalty-free catalog | pre-generated browsable library | paid tracks: $4.90/track; selected free-only tracks: attribution-required download | live | | Pro subscription | 300 credits/mo (the only way to unlock a creator track) + unlimited catalog, $4.90/200-gen top-up | Stripe-driven, ~$9.90/mo · $94.90/yr | live | | Agency Audio Brand Kit | white-label kit + guidelines | $99/kit | **built end to end, not launched** | ## Platform capabilities 1. **Open-weight generation, GPU cloud.** Stable Audio 3 via `packages/inference-core`'s `SubprocessEngine`, on a home GPU (macOS/Linux, `apps/worker`) or RunPod serverless (`apps/runpod`) — both paths asset-agnostic, both speak the same wire protocol. 2. **Affordable, fixed pricing.** $99 flat for a kit, $4.90 flat for a catalog track. A creator track is free (commercial use, credit required); a Brioso Pro subscription — not a per-track purchase, that path is retired (BRI-253) — is the only way to unlock the exclusive, tag-free version, plus a Pro subscription is also the volume-buyer path for the catalog: unlimited downloads, subject to a published fair-usage limit on automated or bulk retrieval (see Messaging constraints). Priced to remove cost as a reason to say no, not to maximize per-unit margin. PWYW is retired (BRI-152/153); see "Where the numbers live" below. 3. **Legal affordances.** Four rights-clear licenses, not two, each its own reviewable document: `docs/track-license.md` (creator, single-sale exclusivity, sold once, then retired), `docs/catalog-license.md` (catalog, **non-exclusive**, a stock library, the same track sold to any number of buyers), `docs/customer-license.md` (kit, sublicensable white-label), and `docs/free-license.md` (free download, commercial use with attribution, BRI-253). The right to modify, adapt, and create derivative works is now common to the three paid licenses (it used to exist only in the kit license); the free grant is narrower and doesn't include it. Rendered as a real PDF (`packages/licensing`) from a document a buyer can actually read, not boilerplate. 4. **Multi-format delivery.** FLAC is the single master format platform-wide — lossless, bit-exact, ~50% of WAV in bytes. The creator and catalog flows ship FLAC + MP3 320 + MP3 128 (`DeliveryPanel`); the kit flow ships FLAC only (`assemble.ts:52-58` appends nothing else). 5. **A studio tool, not a form.** The agency or creator refines the output — knobs, previews, regenerate — rather than filling in a wizard and waiting. This is the product's moat as much as the generation itself: Suno, ElevenLabs, and Epidemic Sound ship generation; they don't ship a white-label, platform-agnostic deliverable an agency or creator can put their name on. 6. **Protected previews, a free download, shareable proposals.** Previews are watermarked and served through a gated proxy so unpaid audio can't leak as a clean file. The live creator/studio generation path's in-app preview carries a single tag near the start (BRI-253), not the dense/looped placement the catalog's offline-seeded previews still use for anti-rip protection. The free tier also gets one real downloadable file per track (a sparse, single-tag watermarked MP3 320, BRI-261, same placement as the preview above), so a creator session that never subscribes still ends with something to keep. The free grant is public and commercial: postable anywhere, monetized projects included, provided the required "Music by Brioso" credit line and link are used and the audible tag stays on the file. Subscribing to Brioso Pro replaces it with the exclusive, tag-free master and release from the credit requirement — the $4.90 per-track unlock is retired (BRI-253); a Pro subscriber's freshly generated track now auto-unlocks at generation completion, no separate purchase step. A link-shareable `packages/sharing` share token (no login required, matching the product's forward-a-link workflow) lets a customer or agency forward a status page or single proposal without exposing the account's full job history to anyone else who might guess the URL. 7. **Provenance and QC.** AudioSeal per-track forensic watermark (`packages/inference-core/src/inference_core/watermark.py`, catalog-seeding path only; `apps/runpod/handler.py` and `apps/worker` never call it, so Studio tracks carry no forensic watermark, just the audible tag below), the audible spoken "usebrioso.com" tag on public previews (BRI-198) and free-tier downloads (BRI-261), and the deterministic ffmpeg/LUFS QC gate (`packages/inference-core/src/inference_core/qc.py`, BRI-143). Note the `watermark.py` module docstring's "every delivered tier" describes that module's own call chain, not the whole platform: it's reachable only from the catalog seeding scripts, so it's accurate to say every *catalog* tier carries the forensic watermark, not every product. Extending AudioSeal into the live generation pipeline is tracked as a follow-up ticket under BRI-253, not yet scheduled. 8. **Discovery surface.** The catalog taxonomy pages are the SEO product; plus the blog (`apps/web/src/data/blog/`, rules in `.agents/docs/seo/blog-authoring-rules.md`) and the Remotion YouTube/Shorts pipeline (`videos/youtube/`). 9. **First-session profiling.** A blocking, skippable five-step `/onboarding` flow runs right after a user's first OTP verify — free and Pro alike, gated only on `users.onboardingCompletedAt IS NULL`, not plan. It writes six columns on `users` (`name`, `country`, `aiExperience`, `useCase`, `discoveryChannel`, `discoveryChannelOther`) via `apps/web/src/server/onboarding/onboarding-fn.ts`. `country` is BRI-270's hard dependency; `discoveryChannel` is the acquisition-channel attribution signal, otherwise unavailable since the product has no ad pixels. Every step is skippable and the flow itself is dismissible ("I'll do this later"), so the data is directional signal from whoever chooses to answer, not a census. ### Why not WAV FLAC is bit-exact lossless, so nothing is given up for a creator or an agency — WAV is not shipped. The one case that would reopen it: a game-engine target, because Unity has no FLAC importer and the RIFF `smpl` loop chunk that carries loop points is WAV-only — revisit if and when a game-dev vertical is scoped, not before. ## Messaging constraints Global, not agency-specific — every product line's copy has to satisfy these: - Say **royalty free**. Never *copyright free* / *no copyright*. (BRI-76 verdict; license §4 concedes AI audio may have limited or no copyright, so the claim is close to literally true and tells buyers they need no permission.) - "No claims **from us**" — never an absolute "no claims" or "cleared for Content ID." Platforms throw false positives against similar-sounding audio, outside Brioso's control; no product line can promise no-claim. This isn't a pure prohibition. Lead with what we can say instead: no one else will generate the exact same file (every generation is randomly seeded), and the model is trained on commercially licensed and Creative Commons audio, not scraped catalogs (the model provider's published position). On Studio and the catalog (not the free tier or the kit), close with the remedy: if a claim hits authorized use and we can't get it cleared, we refund the price paid, capped at that amount and stated as our entire liability, not a legal indemnity. Never name the model itself (BRI-253, `blog.rules.test.ts`). - Price next to the promise; no "free" token in the first 100 characters. - Never "unique" as a **musical quality** claim. Capability §2 concedes similar-sounding audio can exist elsewhere. Uniqueness of the **file** is fine to state, because generation is randomly seeded and no one else will generate that exact file: phrase it as "no one else will generate this file," never "a unique track." - Attribution is a **license condition**, not a request, and only on the free tier: the credit line ("Music by Brioso" + link, canonical strings in `apps/web/src/lib/studio/creditSnippets.ts`) is required on every free download. No paid tier, on any product line, ever requires attribution. The "Powered by Stability AI" footer line is Brioso's own obligation to the model provider and never flows down to a buyer. - Pro's "unlimited" catalog downloads may ship only with the concrete fair-usage caveat: unmetered for normal creative use, Brioso reserves the right to apply a published fair-usage limit to automated or bulk retrieval, any limit is published before it applies, and licenses already granted on tracks already downloaded stay valid regardless. BRI-162 lands the actual number without contradicting this. - The pitch is **"sold once, then retired" plus made-to-brief** — true of the creator line only, **false for the catalog**, which is a non-exclusive stock library. `routes/_marketing/license.tsx:24-28` already flags this; keep both lines honest about which promise applies to which product. - Agency line keeps its existing do/don't ("Audio Brand Kit," not "AI music generator"). ## Line 1: creator custom generation (live) **ICP:** creators making video, podcasts, games, or any project that needs a track built to their brief rather than picked off a shelf. Live and shipping (`/studio`): a tag-based brief loop, single-sale exclusivity licensing (`docs/track-license.md`), FLAC + MP3 320 + MP3 128 delivery, free-preview and free-trial quotas ahead of unlocking. The per-track $4.90 Stripe checkout is retired (BRI-253): every generated track gets a watermarked MP3 320 download, free, licensed for commercial use provided the required credit line stays attached and the audible tag stays on the file (BRI-261) — a real artifact for the free tier, not just a playable stream. Subscribing to Brioso Pro is now the only way to unlock the exclusive, tag-free master and release from the credit requirement; a Pro subscriber's track auto-unlocks at generation completion rather than requiring a separate purchase. ## Line 2: royalty-free catalog (live) **What it is:** a pre-generated, browsable royalty-free music library — the "Epidemic Sound, not ElevenLabs" bet — sold as single tracks rather than commissioned per brief. - **Browse axes:** genre, mood, instrument, use-case — four independent taxonomies (`packages/catalog/src/taxonomy.ts`), path-based URLs (`/music/genre/...`, `/music/mood/...`) rather than query params, chosen to mirror Epidemic's URL architecture because real slug collisions exist across axes (e.g. "Jungle," "Lounge" are both a genre and a mood) that a flatter scheme can't disambiguate. - **Curation:** 12 priority genres and 12 priority moods (`packages/catalog/src/ taxonomy.ts:50,465`, `featured: true`) pinned first in the UI — a subjective curation call, not search-volume-driven. - **Coverage:** ~3,400 tracks is the full-coverage target across every taxonomy leaf; the live launch is a smaller seeded subset (`seeded: true` leaves only) of roughly 32 categories, expanding per BRI-133 as demand is proven. Unseeded leaves stay `noindex` until they have both real intro copy and published tracks (`apps/web/src/lib/catalog-seo.ts`'s `shouldIndexTaxonomyPage`). - **License:** `docs/catalog-license.md` — **non-exclusive**. The same track can be sold to any number of buyers, before and after any given purchase, and stays in the catalog. This is the one place the platform's "sold once, then retired" pitch does not apply. - **Pricing:** $4.90/track. Source of full reasoning: `docs/royalty-free-music-decisions.md`. ## Line 3: agency Audio Brand Kit (built, not launched) **ICP:** branding and creative agencies, roughly 3–50 staff, who deliver visual identity projects (logo, color, typography, brand guidelines) to clients and are asked — or want to be able to offer — an audio identity as part of that engagement. Buyers, in order of likely entry point: the founder/owner of a small branding studio, the creative director who owns deliverable quality, the brand strategist who'd write the audio brief if it didn't require hiring a specialist. Explicitly not the customer: restaurants/retailers buying audio directly, freelance composers/sound designers (a substitute, not a buyer), or consumers looking for stock/library music. **The problem:** a branding agency wins an identity engagement, ships the visual system, and hears "can you also do our audio branding?" Today the honest answer is "we outsource it" — to a sonic branding studio (MassiveMusic, Sixième Son, Made Music Studio, amp, DLMDD, Antfood), which costs $5–15k and weeks of turnaround, cuts the agency out of the delivery relationship, and risks the client going to the studio directly next time. Brioso lets the agency say yes, keep the relationship, and deliver an Audio Brand Kit the client believes came from the agency. **The loop:** ``` logo + website URL ──► brand analysis ──► AUDIO STRATEGY (free) (required) (vision model) personality · emotion · + optional brand instrumentation · tempo · answers texture · the "Why" paragraph │ agency edits + approves │ $99 ▼ kit generation ──► white-label ZIP + guidelines ``` The **Audio Strategy** is this use case's moat, not the audio itself. Agencies are experts in brands, not audio — asking them "what BPM?" makes Brioso an AI music toy. Instead Brioso infers a brand's audio identity from what already exists (site, logo) and proves it understands the brand *before* generating a single note. The agency edits and approves; the approved strategy is exactly the plan Brioso generates against — one artifact earns trust and specifies the kit. **What ships:** 1. **Strategy loop (free).** Brand intake (logo + URL required; brand-language questions optional) → automated brand analysis → an editable Audio Strategy document with a rationale paragraph an agency would recognize as accurate. 2. **Paid generation ($99).** Approving a strategy fans out into per-asset generation jobs (sonic logo, brand theme, stingers, intro/outro, UI sounds, social pack) on the shared generation pipeline. 3. **White-label delivery.** A ZIP of FLAC masters plus Audio Brand Guidelines (Markdown) derived from the strategy, carrying zero Brioso branding, so the agency's client never sees Brioso. **The $99 price and its risk:** chosen to be a trivial line item inside a ~$6k+ engagement rather than a negotiated cost, deliberately below what a full economic analysis might justify. Margin risk, stated openly: a kit spans multiple generated assets plus a vision-model brand-analysis call. If true generation + inference cost per kit exceeds roughly $25–30, the $99 price does not clear margin at the assumed asset-plan size, and either the cost ceiling on the asset plan tightens or the price moves. Measured cost per kit: not yet measured — populate once Milestone 2.5 (cost-per-kit instrumentation) has real data from test/design-partner kits. **Status and launch gate.** The pipeline above is real end to end — intake, SSRF-guarded capture, vision analysis, editable strategy, $99 Stripe checkout, per-asset fan-out, white-label ZIP + guidelines delivery, all with tests — but it sits behind `ComingSoonChip` on `/agencies`, in the header, in the footer, and on the `/pricing` card while GTM focuses on the creator and catalog lines instead. **Messaging do / don't:** do say "Audio Brand Kit," "Audio Strategy," "white-label," "your client never sees Brioso," "deliver without hiring composers or sound designers." Don't say "AI music generator," "generate songs," "make a track" — anything that frames the output as a novelty rather than a brand deliverable. The AI is the engine that makes the economics work; it is never the pitch. ## Not building - A multi-tenant agency platform. Agency accounts, client-project hierarchies, and a dashboard are deferred until agencies have actually paid — see "Agency workspace" in [`roadmap.md`](roadmap.md). ## Where the numbers live One place to look before the next price change: - `packages/verticals/{catalog,kits}/src/*-checkout.ts` — the cents actually charged, server-side. Creator has no per-track checkout anymore (BRI-253 retired it); its `track-checkout.ts` only guards that Stripe is configured for the flows that remain. - `apps/web/src/lib/{creator-pricing,catalog-seo}.ts` — the labels shown in copy, tested against the checkout constants above. - `packages/billing/src/pro-price-display.ts` — Pro's price, read live from Stripe. - `.agents/reports/2026-07-31-pricing-model-decision.md` — the reasoning behind all of it. - `packages/db/src/onboarding.ts` — the onboarding enum value lists (`AiExperience`, `UseCase`, `DiscoveryChannel`) and the `COUNTRIES`/`COUNTRY_CODES` allowlist; single source for the schema column types, the server validator, and the route's option labels. ## Engineering roadmap Feature sequencing and Iteration/Quarter mapping now live in [`roadmap.md`](roadmap.md), which also tracks the underlying monorepo/infra migration this product is built on. --- ## [R2 Lifecycle Rules: \`brioso-jobs\` Bucket](https://fitxers.usebrioso.com/internal/r2-lifecycle-rules/) (Internal) # R2 Lifecycle Rules: `brioso-jobs` Bucket > **Never add a lifecycle rule touching the `litestream/` prefix.** That > prefix holds Litestream's DB WAL replication stream. A lifecycle rule that > expires or deletes objects under `litestream/` will corrupt backups. This > is not a hypothetical — treat it as a hard boundary when editing rules in > the Cloudflare dashboard. ## Why this doc exists The `brioso-jobs` bucket is on Cloudflare R2's 10GB free tier and was observed at ~16GB. Rather than deleting paid/delivered content, we expire only artifacts that are either rebuildable, orphaned, or explicitly time-boxed. **Paid or delivered content is never deleted by this system.** ## Dashboard lifecycle rules These two rules are hand-configured via the Cloudflare dashboard (**Bucket → Settings → Object lifecycle rules**). There is no Terraform/IaC coverage for this bucket, so if it is ever recreated, these rules must be re-applied by hand from this doc. ### Rule 1 — `kits/` prefix, expire after 7 days Kit ZIPs are rebuildable from their source tracks on demand, so they don't need to survive indefinitely. Objects under the `kits/` prefix expire 7 days after upload. ### Rule 2 — `logos/` prefix, expire after 30 days Logo uploads that are never attached to a finished kit are orphaned data with no other reference pointing at them. Objects under the `logos/` prefix expire 30 days after upload. ## What is NOT a dashboard rule Two more retention categories exist but are **not** representable as Cloudflare dashboard lifecycle rules, because the dashboard feature has no awareness of database state (job status, payment status): - **`failed-job`** — artifacts belonging to failed or cancelled jobs are swept after 14 days, excluding artifacts that are children of paid kits **and any job with a paid or free-unlocked track** (BRI-374: cancelJob is legal from any state, so a cancelled job can still hold delivered content — payment status outranks the sweep). After a whole-prefix sweep the groomer stamps the job row's `artifacts_purged_at` tombstone, and the studio renders those generations as "no longer available" (terminal state, no player) instead of serving broken playback. - **`watermarked-master`** — a track's watermarked master is swept immediately once the track is marked paid (no grace period). Free-tier `free_unlocked` tracks are never touched by this category, since their watermarked file is their only copy. Both categories require checking job/track state in the database before deleting, which the dashboard's prefix+age model can't express. They are handled by a code-side groomer instead: `packages/dispatch/src/storageGroomer.ts`. ## Keeping this doc in sync The thresholds above must match `packages/storage/src/retention.ts`'s `RETENTION_RULES` array (the `mode: "age"` entries) by hand — there is no automated sync between the dashboard config, this doc, and that file. If you change a threshold in one place, update the other two in the same change. --- ## [Release process (CalVer via release-it)](https://fitxers.usebrioso.com/internal/release-process/) (Internal) # Release process (CalVer via release-it) All workspace packages share one lockstep version `yyyy.ww.patch` (ISO week, e.g. `2026.34.0`). Releases are cut with `release-it` + `@csmith/release-it-calver-plugin` + `@release-it/bumper`, configured in `.release-it.json`. Nothing is published to npm; versioning exists for git tags, GitHub Releases, CHANGELOG.md, and tag-driven deploys. ## Procedure 1. Run `pnpm run release:dry` first — verify the computed ISO week number and that all 26 `package.json` files appear in the bumper's out list. 2. Cut stable from `main`: `pnpm run release` → commits `chore(release): vX`, tag `vX`, GitHub Release, CHANGELOG entry, lockstep version bump. 3. Beta: from a `beta` branch (or main between stable cuts), `pnpm run release:beta` → `yyyy.ww.patch-beta.N` tags (the calver plugin ignores release-it's `--preRelease`, so `scripts/release-beta.ts` computes the suffix explicitly). Merge `beta` → `main` before the next stable cut. 4. All deploys go through `.github/workflows/deploy-pyinfra.yml` — the single deploy path (pyinfra artifact + activate, Infisical env render, report-only drift gate, post-deploy smoke + config-audit). It fires on: - **push to `beta`** (BRI-435 converge-always: merging to beta IS the beta deploy — docs-only paths are ignored), - **CalVer tags**: `vX.Y.Z` -> production, `vX.Y.Z-beta.N` -> beta, - **`workflow_dispatch`**: rollback/redeploy only (`ref` + optional `artifact_sha256` redeploy an existing artifact). It replaces whatever is live for that env. Every trigger is guarded: the `resolve-env` job refuses any sha that is not already on the env's branch (`beta` for beta, `main` for prod), so a feature-branch commit can never be deployed. To test on beta, merge to `beta`. Every run builds the artifact in-run via `build-artifact.yml` (`workflow_call`), so deploy never races a separate builder. The drift gate no longer blocks: host drift warns in the run summary and the deploy converges the host (PYINFRA_ENV_MODE=write lets it reconcile the env file too). 5. Cut a beta tag after each merge to `beta`. The push already deployed; the tag is a stable point-in-time marker: it ties a deploy run's `head_sha` to a named release on `beta.usebrioso.com`, so "what was on staging when X shipped?" stays answerable when the branch tip has moved on. `git fetch origin beta` and inspect the tag's `git log -1` (or the GitHub Release) instead of guessing from commit SHAs. ## Fail-first PR checks (BRI-434) Deploy problems surface on the PR, not on deploy night: - `build-artifact.yml` runs build + boot-smoke + smoke-test-artifact on PRs touching deployable paths (`apps/**`, `packages/**`, `deploy/**`, `scripts/**`, `config/**`, `.github/**`, toolchain manifests). PR builds resolve no GitHub Environment — they compile against the checked-in fixture env `deploy/artifact-env/smoke.env.example` (audit-build-env.ts still enforces required key NAMES) and skip the Release publish (the smoke jobs consume the run-scoped artifact upload). - `deploy-pr-checks.yml` (`pull_request_target`, base-branch code only, PRs into beta/main): `secrets-check` hard-fails when a GitHub secret NAME in `config/deploy-secrets.yaml` is missing from the target environment, audits the Infisical render, and asserts the Modal worker secret's required keys; `drift-report` posts the live `pyinfra --dry --json` report as a warning + job summary (informational). - `mise run deploy:rehearse` locally remains the pre-merge dry run. ## Retired deploy surfaces (BRI-435) Deleted: `deploy-beta.yml` (bash beta deploy; its `deploy-beta.sh` was already gone), `pyinfra-deploy-beta.yml` (Phase-3 soak, superseded by deploy-pyinfra.yml), `modal-deploy.yml`, `deploy-modal-audiocpp.yml`, `beam-deploy.yml` (dispatch-only deploys of retired runtimes), `build-runpod.yml` (retired image build). Still live: `deploy.yml` (frozen manual audit trail — do not dispatch), `deploy-modal-worker.yml`, `deploy-secrets-sync.yml`, `pyinfra-drift.yml` (daily drift schedule). --- ## [Brioso Roadmap](https://fitxers.usebrioso.com/internal/roadmap/) (Internal) # Brioso Roadmap Source of truth for Iteration/Quarter mapping used to refine tasks in [Linear, Brioso team](https://linear.app/brioso/team/BRI). See [`.agents/backlog-policy.md`](../backlog-policy.md) for the refine gate. Iterations/Quarters below mirror Linear's configured `Iteration` and `Quarter` fields exactly (same start dates/durations) — don't invent new period names, extend these when the Iteration field runs out. ## Quarter 1 (2026-06-01 -> 2026-08-30) Goal: monorepo live, libSQL/Turso cutover, TanStack Start strangler underway. | Iteration | Window | Milestone | |---|---|---| | Iteration 1 | 2026-06-01 -> 2026-06-14 | Legacy backlog migrated, project scaffolded — **done** | | Iteration 2 | 2026-06-15 -> 2026-06-28 | Monorepo migration plan finalized — **done** | | Iteration 3 | 2026-06-29 -> 2026-07-13 | Phase 1 (scaffold monorepo) done; Phase 2 (libSQL/Turso cutover) starts — **done** | | Iteration 4+ | 2026-07-14 onward | Phase 2 completes; Phase 3 (TanStack Start strangler) starts. Phase 8 (observability foundation, #9) pulled forward into this iteration — blocks PR #50 merge, see below — extend project's Iteration field when it runs out — **done** | ## Quarter 2 (2026-08-30 -> 2026-11-28) Goal: TanStack Start cutover complete, RunPod inference live, A/B dispatch, systemd + git-pull deploy. - Phase 4: Cutover to TanStack Start — **done, pulled forward ahead of the Q2 window** - Phase 5: inference-core + apps/runpod — **done, pulled forward** - Phase 6: A/B dispatch (home-preferred, RunPod-fallback) — **done, pulled forward** - Phase 7: VPS systemd deploy + CI — **done, pulled forward** Also pulled forward, not originally scoped into any phase above: - **Royalty-free catalog (BRI-125).** Third product line, `packages/catalog` + `packages/verticals/catalog` + `packages/storage`'s catalog bucket, taxonomy-page marketplace under `/music`. Shipped 2026-08-02. See `prd.md`'s "Line 2." - **Pricing overhaul (BRI-152/151/153).** Retired pay-what-you-want across creator and catalog in favor of fixed per-track pricing, plus the Pro subscription (`packages/billing`'s `subscription-checkout`/`pro-credits`/`entitlement`). Shipped 2026-08-02. The model has since moved to freemium + Pro + Enterprise (see `.agents/reports/2026-07-31-pricing-model-decision.md`). - **GTM / discovery track.** The catalog taxonomy pages as an SEO surface, the blog (`apps/web/src/data/blog/`), and the Remotion YouTube/Shorts pipeline (`videos/youtube/`, BRI-140/198) — none of it tracked as a phase above, all of it live or in active use by the time this section was written. - **Free-tier value boost — watermarked Studio downloads (BRI-261, sub-tickets BRI-262/263/264/265/266).** Gives the free tier a real, downloadable artifact — a sparse single-tag watermarked MP3 320 per generated track, personal/non-commercial — in place of "hear it, then nothing," while a paid track-level unlock remains the path to the clean master + commercial license. See `prd.md`'s Line 1 and capability 6. In progress as of 2026-08-07. ## Quarter 3 (2026-11-28 -> 2027-02-26) Goal: observability hardening, backlog cleanup. - Phase 8: Observability foundation (#9) — pulled forward to Iteration 4 (2026-07-21), superseding the Langfuse plan with PostHog (errors/events/AI-spans) + shared `obs-js`/`obs-py` packages; otelite decommissioned, replaced by Phoenix. Design: [`.agents/plans/2026-07-21-pr1-observability-foundation.md`](../plans/2026-07-21-pr1-observability-foundation.md). ## Phase -> task map Phase issues live in `briosoco/brioso` (#1 epic, #2-#9 phases). Full plan: [`.agents/plans/2026-07-10-monorepo-migration.md`](../plans/2026-07-10-monorepo-migration.md). ## Milestone 2.5 — cost-per-kit instrumentation Referenced by `prd.md`'s "kit pricing and its risk": measure real generation + inference cost per Audio Brand Kit from test/design-partner kits, so the assumed asset-plan size and $25-30 cost ceiling can be checked against data instead of estimate. Not scheduled into an Iteration yet — pull into Quarter 2/3 once design-partner kit volume exists to measure. ## Agency workspace Referenced by `prd.md`'s "Not building": a multi-tenant agency platform — agency accounts, client-project hierarchies, a dashboard. Deferred until agencies have actually paid for the current single-tenant kit flow; not scheduled into an Iteration. ## Product capability track Flag-gated survivors of the audio tooling evaluation ([`audio-tooling-evaluation.md`](audio-tooling-evaluation.md) — rationale lives there, not restated here). Not yet scheduled into an Iteration; each becomes one when its flag ships. - Demucs v4 stems (`audio-stems`) - Librosa/Essentia BPM + key (`audio-analysis`) - EBU R128 loudness normalization (`audio-loudness`) - Seamless looping (`audio-seamless-loop`) - Wavesurfer.js inpainting UI (`audio-inpaint-ui`) --- ## [Route census](https://fitxers.usebrioso.com/internal/route-census/) (Internal) # Route census Every user-facing page route in apps/web belongs to exactly one class. The manifest lives in `apps/web/scripts/check-route-census.ts`; the gate runs in `mise run validate` and `validate:quick`. ## Classes - **A: static / prerendered.** Marketing pages, blog, legal. No session dependencies, no request-time auth. The file may not import or call `chromeSession` / `getChromeSession`. - **B: SSR, CDN cacheable.** The `/audio/music` taxonomy tree (genre, mood, instrument, use case, license, checkout-success, index). Deterministic per URL, so responses cache at the edge. Same rule: no `chromeSession` import or call in the route file. - **C: dynamic SSR.** Everything session or request dependent: `_authed` pages, onboarding, `studio/**`, `s/$token`, checkout-success, `login`, `strategy/$slug` (client useQuery), redirects (`status/$slug`, `music/$`, `music`), and the non HTML `robots.txt` / `sitemap.xml`. ## Render strategy first Default to SSG/prerender (class A). Every SSR page is a per-request origin render unless declared otherwise; reaching for SSR by default rebuilds the route bloat the census exists to prevent. 1. **SSG / prerender** (class A) — static content, no per-request data, no session reads. Default for marketing, legal, blog, shells. Add the path to the prerender pages in `apps/web/vite.config.ts` AND the census manifest; a missing declaration fails the build. 2. **CSR** (class A shell if the loader is static, else C) — data loads via `useQuery` after hydration (`strategy/$slug` is the example). Prefer for user-specific or interactive pages that do not need SEO. 3. **SSR + CDN cache** (class B) — deterministic per URL, read-only DB, no personalization. Loaders must stay free of `getChromeSession`; cache headers come from nitro `routeRules` (`s-maxage`). 4. **Dynamic SSR** (class C) — genuinely per-request: authed, personalized, share tokens. The fallback, not the default. Server handlers, API endpoints, internal admin, and the dev-only login handler are not pages and stay out of the census. ## Adding a route 1. Add the route file as usual. 2. Regenerate `routeTree.gen.ts` (run the web dev or generate task once). 3. Declare the new path in the manifest in `apps/web/scripts/check-route-census.ts` with its class. Undeclared routes fail the gate, by design. 4. If the route is A or B, keep session code out of the route file. Session dependent behavior belongs in class C pages or in `_marketing.tsx`'s `beforeLoad`. The gate also fails if a manifest entry no longer matches any route, so deleting a page requires removing its census line. ## Run ```sh bun apps/web/scripts/check-route-census.ts ``` ## Facet split for taxonomy cache headers Each taxonomy leaf and hub index under `/audio/music/**` sets its `Cache-Control` via the route's `headers: () => ({ ... })` callback (router-core merges the returned headers into `match.headers`, then `getStartResponseHeaders` builds the final Response from those). The shared value for unfiltered URLs is `public, s-maxage=300, stale-while-revalidate=86400`. When any filter query param is present (`mood`, `q`, `bpmMin`, `bpmMax`, `key`, `movement`, `character`, `instrument`), the helper returns `private, max-age=0` so a poisoned filtered variant can never persist at the edge. Why route-level and not a nitro `routeRules` header: h3's `toResponse` finalization prefers `event.res.headers` (where nitro writes) over `Response.headers` (where the route's headers callback lands), so a per-route `private` override wrote by the headers callback would lose to a `/audio/music/**` routeRule. Putting the value on the Response (not on `event.res.headers`) keeps it through the merge. There is therefore no longer a `/audio/music/**` entry in `vite.config.ts`'s `routeRules`. The helper is `taxonomyLeafCacheControl()` in `apps/web/src/lib/taxonomy-cache.ts`. It reads the incoming URL via `getRequestUrl()` (only available inside the request middleware chain — TanStack Start wraps the headers callback invocation inside `runWithStartContext`). It returns `null` outside a request context (client hydration re-renders the headers callback; the route's `cc ? { "Cache-Control": cc } : {}` handles that). ## Verifying the matrix Start the dev stack, then run the check script: ```sh mise run dev:fake bun apps/web/scripts/check-taxonomy-cache.ts [baseUrl] ``` It parses taxonomy leaf URLs from the live `/sitemap.xml`, asserts every unfiltered leaf returns 200 with `s-maxage` in Cache-Control, asserts faceted variants of one genre leaf return exactly `private, max-age=0`, and asserts `/music/` 301s to `/audio/music/`. Nonzero exit on any failure. On demand only; not wired into `mise run validate`. ## Cloudflare Cache Rule + canary (slice 3, out of repo) Pending: a Cloudflare Cache Rule for `/audio/music/*` that respects origin cache headers. Canary: curl the same unfiltered leaf twice and watch `cf-cache-status` go MISS then HIT. The purge token must be a NEW Cloudflare token scoped to Zone.Cache Purge, stored in Infisical. Never reuse the existing DNS:Edit token for purges. ## Adding a new faceted leaf 1. Add the route with `validateSearch` declaring its filter params. 2. Add those param names to `FILTER_PARAMS` in `apps/web/src/lib/filter-params.ts` (a tiny pure-data module so the `check-taxonomy-cache.ts` script can import it too). 3. Add a `headers: () =>` option to the route that returns the result of `taxonomyLeafCacheControl()`: ```ts import { taxonomyLeafCacheControl } from "#src/lib/taxonomy-cache"; export const Route = createFileRoute("...")({ validateSearch: ..., headers: () => { const cc = taxonomyLeafCacheControl(); return cc ? { "Cache-Control": cc } : {}; }, loader: ..., }); ``` ## Purge hook on admin publish `apps/web/src/server/admin/catalog-handlers.ts` (sole mutation path for catalog rows — `updateCatalogTrackHandler`) calls `clearLeafCountsCache()` from `packages/catalog/src/list-tracks-fn.ts` to reset the module-scope memo for this blue-green slot, then fires `purgeTaxonomyCache()` from `apps/web/src/lib/cdnPurge.ts`. The purge sends `POST /zones/{CF_ZONE_ID}/purge_cache` with `{"prefixes":["usebrioso.com/audio/music/"]}` (a prefix purge needs no URL enumeration). When `CF_PURGE_TOKEN` or `CF_ZONE_ID` are unset the function returns immediately with `{ purged: false }`, so local dev and preview slots stay no-ops. A purge failure must NOT fail the admin update. The hook is fire-and-forget and logs success/failure via `@brioso/obs-js`. Cloudflare zone sharing: beta and prod share one CF zone, so a prefix purge from beta clears prod's edge entries too. That is acceptable for correctness — the origins re-serve per-host content and a beta cache miss just re-fetches. Documented so future engineers don't read purge logs as prod signals. --- ## [RunPod Bootstrap — manual setup runbook](https://fitxers.usebrioso.com/internal/runpod-bootstrap/) (Internal) > Retired in BRI-439: the RunPod runtime is gone. Pre-v2 setup; for current RunPod ops see `runpod-v2-cutover.md` (BRI-357). # RunPod Bootstrap — manual setup runbook Companion to `.agents/plans/2026-07-18-runpod-bootstrap.md`. Walks through every manual step to get the first SA3 job dispatched and returning tracks. > **Status (2026-07-18): Phases A–C are done.** The endpoint is live and returning tracks — > see `apps/runpod/README.md` for what was actually deployed (endpoint id, image tag, GPU, > region) and for the container gotchas discovered while building it. Phase D (web app wiring > and the end-to-end test) is still outstanding. > > Two corrections from doing it for real: > - The image is **PyTorch-only**; the TensorRT path was stripped because it cannot be built > without a GPU present (tracked in BRI-18). The "TRT only serves small-music" limitation > below therefore does not apply yet — the endpoint runs `stable-audio-3-medium`. > - Measured cost is higher than the ~$0.01/job estimated below: a 3x30s job takes ~43s of > RTX 4090 time plus platform fee. Re-measure on a full `[[3,180],[2,160]]` job before > trusting `RUNPOD_EST_COST_CENTS_PER_JOB`. ## Prerequisites - RunPod account (https://www.runpod.io) - Docker locally with buildx - A container registry account (Docker Hub or GHCR recommended) - `WORKER_SHARED_SECRET` value (must match what's on the VPS web app — ask yourself / check VPS env) - GlitchTip DSN — get from https://eu.glitchtip.com/brioso/settings/keys/ (placeholder only in `apps/runpod/.env.example`) ## Phases - **Phase A** — RunPod account prep (parallelizable now, no code dependency) - **Phase B** — Image build + push (after W1–W4 from the plan are merged) - **Phase C** — Endpoint creation + first job (after Phase B + image pushed) - **Phase D** — Web app wiring + verification --- ## Phase A — RunPod account prep ### A1. Add credits Console → **Billing** → **Add credits**. - Recommended starting balance: **$50** (covers ~150 hr of RTX 4090 Community at $0.34/hr — plenty for first-week testing) - Payment: credit card (Stripe), crypto, or wire for >$5k - Enable **Low balance alert** at $20 - Optional: **Auto-pay** with $50 reload at $10 threshold Default spend cap is $80/hr across all resources. Fine for now; raise via support ticket later. Doc: https://docs.runpod.io/accounts-billing/billing ### A2. Create network volume for HF cache Console → **Storage** → **Network Volumes** → **Create**. | Field | Value | |---|---| | Name | `brioso-sa3-cache` | | Datacenter | Pick a US region with RTX 4090 availability (`US-GA-2`, `US-OR-2`, etc.) — note the region; the endpoint MUST be in the same region | | Size | **100 GB** (SA3 model ~6–10 GB + HF cache grows) | Mount path inside container is automatic at `/runpod-volume` — do not override. ⚠️ **Region locks the endpoint.** Once a volume is attached, the endpoint can only spin up workers in that volume's datacenter. Pick a region known to have RTX 4090 + A6000 inventory. Cost: $0.07/GB/month ≈ $7/month for 100 GB. Doc: https://docs.runpod.io/storage/network-volumes ### A3. Pick a container registry RunPod has no first-party registry. We use Docker Hub: - **Docker Hub**: `docker.io/usebrioso/brioso-runpod:` — auth via `DOCKER_USERNAME` / `DOCKER_TOKEN` (root `.env`); `build-and-push.sh` logs in automatically. For private images, you'll need to register credentials in RunPod later (Phase C5). --- ## Phase B — Image build + push (after W1–W4 merged) ### B1. Authenticate to your registry ```bash # Docker Hub — build-and-push.sh logs in automatically when # DOCKER_USERNAME/DOCKER_TOKEN are set in root .env. Manual login only for one-off pushes: # echo "$DOCKER_TOKEN" | docker login -u "$DOCKER_USERNAME" --password-stdin ``` ### B2. Build the image From repo root: ```bash docker build -f apps/runpod/Dockerfile.sa3 -t docker.io/usebrioso/brioso-runpod:$(git rev-parse --short HEAD) . ``` Expected: long build (~20–40 min on first run due to `git clone` of stable-audio-3 + TRT engine compile via `install.sh`). Subsequent builds cached. ### B3. Push ```bash docker push docker.io/usebrioso/brioso-runpod:$(git rev-parse --short HEAD) # also tag :latest for convenience docker tag docker.io/usebrioso/brioso-runpod:$(git rev-parse --short HEAD) docker.io/usebrioso/brioso-runpod:latest docker push docker.io/usebrioso/brioso-runpod:latest ``` The image is **private** on Docker Hub, so wire up registry auth (Phase C5) before endpoints can pull it. --- ## Phase C — Endpoint creation + first job ### C1. Create the serverless endpoint Console → **Serverless** → **New Endpoint**. #### Container image | Field | Value | |---|---| | Container Image | `docker.io/usebrioso/brioso-runpod:` (or `:latest`) | | Registry Auth | required (image is private) — pick the auth created in C5 | #### GPU Configuration | Field | Value | |---|---| | GPU type | **RTX 4090** first, **RTX A6000** as fallback (select both, in priority order) | | GPUs per worker | 1 | #### Worker scaling | Field | Value | |---|---| | Active workers (min) | **0** (scale-to-zero when idle) | | Max workers | **3** | #### Advanced | Field | Value | Why | |---|---|---| | Idle timeout | **60s** | Keep worker warm between burst requests | | Execution timeout | **600s** | Matches `LEASE_SECONDS=600` in `sweep.ts` — do not change one without the other | | FlashBoot | **Enabled** (default) | Sub-200ms revival for warm workers | | Container disk | **20 GB** | SA3 medium + headroom | | Network Volume | **attach `brioso-sa3-cache`** | Same region as volume | Doc: https://docs.runpod.io/serverless/endpoints/endpoint-configurations ### C2. Set environment variables on the endpoint Endpoint → **Settings** → **Environment Variables**. | Variable | Value | Secret | |---|---|---| | `JOBSTORE_API_URL` | `https://usebrioso.com` | no | | `WORKER_SHARED_SECRET` | (same value as VPS web app) | yes | | `PHOENIX_COLLECTOR_ENDPOINT` | `https://usebrioso.com/otel` (OTLP forward route on the web app; auth'd via worker secret) | no | | `OTEL_RELAY_TOKEN` | (same value as `WORKER_SHARED_SECRET` — the route validates the bearer against it) | yes | | `TRACING_ENABLED` | `true` | no | | `HF_HOME` | `/runpod-volume/huggingface-cache` (already in Dockerfile — confirm only) | no | Doc: https://docs.runpod.io/serverless/development/environment-variables ### C3. Capture endpoint ID After creation, copy the **Endpoint ID** from the endpoint detail page URL or panel. Looks like `abcd1234efgh`. This becomes `RUNPOD_ENDPOINT_ID` on the web app. ### C3a. Beta staging endpoint `beta.usebrioso.com` (see `deploy/README.md`'s "Beta / staging environment" section) gets its own duplicated **medium** endpoint, same image — sharing prod's endpoint would let a beta job's GPU worker call back into the prod app, since `JOBSTORE_API_URL` is set per-endpoint here, not in Infisical. Repeat C1–C2 with: | Variable | Value | |---|---| | `JOBSTORE_API_URL` | `https://beta.usebrioso.com` | | `WORKER_SHARED_SECRET` / `OTEL_RELAY_TOKEN` | beta's own value (staging Infisical env), not prod's | Put the resulting endpoint id in staging's `RUNPOD_ENDPOINT_ID_MEDIUM`. Leave `RUNPOD_ENDPOINT_ID_SMALL_*` unset in staging — `runpodClient.ts` returns a clean 501 for unconfigured variants rather than throwing. Serverless scales to zero, so an idle beta endpoint costs nothing. ### C4. Generate API key Console → **Settings** → **API Keys** → **Create API Key**. | Field | Value | |---|---| | Name | `brioso-webapp-runpod` | | Permissions | **Restricted** | | Endpoint access | Grant to this endpoint only | | GraphQL access | None | New key is prefixed `rpa_`. Copy immediately — RunPod does not store it. This becomes `RUNPOD_API_KEY` on the web app. Doc: https://docs.runpod.io/get-started/api-keys ### C5. (Only if image is private) Register registry credentials Console → **Settings** → **Container Registry Auths** → **Create**. | Field | Value | |---|---| | Provider | Custom (for GHCR) or Docker Hub | | Username | GitHub username or Docker Hub username | | Password / token | GHCR PAT with `read:packages`, or Docker Hub access token | Generates a `containerRegistryAuthId`. Attach it to the endpoint template under Advanced. ### C6. Smoke test from the console Endpoint → **Requests** tab → modify the JSON body: ```json { "input": { "slug": "smoke-test", "prompt": "warm acoustic guitar loop, 90 bpm, gentle fingerpicking", "customer_email": "smoke@brioso.co", "batches": [[1, 30]], "steps": 8, "cfg_scale": 1.0, "seed": 42 } } ``` Click **Run**. Watch the **Logs** tab for: - Container boot + TRT engine load (~30–60s cold start) - `[runpod-handler]` log lines from `handler.py` - Generation progress - Upload to R2 via presigned URLs (requires web app's `/jobs/{slug}/upload-urls` route to accept the request — authenticate with `WORKER_SHARED_SECRET`) Expected outcome: `{"tracks": [{"idx": 0, "preview_key": "...", ...}]}`. If you see `{"error": "..."}`, check Logs first, then GlitchTip for the captured exception. ### C7. Smoke test via curl (alternative) ```bash curl -X POST https://api.runpod.ai/v2/$RUNPOD_ENDPOINT_ID/runsync \ -H "Authorization: Bearer $RUNPOD_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "input": { "slug": "smoke-test", "prompt": "warm acoustic guitar loop, 90 bpm", "customer_email": "smoke@brioso.co", "batches": [[1, 30]], "steps": 8, "cfg_scale": 1.0, "seed": 42 } }' ``` Doc: https://docs.runpod.io/serverless/endpoints/send-requests --- ## Phase D — Web app wiring + verification ### D1. Set env vars on the VPS web app Add to the web app's env (whatever mechanism — systemd EnvironmentFile, dotenv, etc.): ``` RUNPOD_API_KEY=rpa_... RUNPOD_ENDPOINT_ID=abcd1234efgh DISPATCH_WAIT_SECONDS=45 DISPATCH_QUEUE_DEPTH_THRESHOLD=5 RUNPOD_DAILY_SPEND_CAP_CENTS=2000 RUNPOD_EST_COST_CENTS_PER_JOB=15 ``` Restart the web app. Dispatch sweep is triggered externally (systemd timer / GH Actions cron per `apps/web/src/routes/dispatch/sweep.ts`) — confirm that scheduler is running. ### D2. Verify `runpod_spend` table exists on target DB The migration SQL exists at `packages/db/src/migrations/20260711092351_swift_mastermind/migration.sql` but `packages/db/src/migrations/meta/_journal.json` is missing, so `mise run db:migrate` cannot apply it. Check the table: ```bash # against your prod libsql/SQLite sqlite3 $DATABASE_URL "SELECT name FROM sqlite_master WHERE type='table' AND name='runpod_spend';" ``` If empty, apply manually: ```bash sqlite3 $DATABASE_URL < packages/db/src/migrations/20260711092351_swift_mastermind/migration.sql ``` Without this table, `sweep.ts:25-42` throws on every dispatch attempt. ### D3. End-to-end test With the home worker **stopped** (so it doesn't claim first): 1. Create a job via the normal UI / API (cfg=1.0, sm-music — the defaults). 2. Wait `DISPATCH_WAIT_SECONDS + ~30s` for the sweep to fire. 3. Sweep logs should show: `sweep started` → `dispatched to runpod`. 4. RunPod logs should show the handler running. 5. Job in DB should end in `state: "paid"` with `tracks` populated. 6. R2 (brioso-jobs bucket) should have preview + master objects under the slug key namespace. ### D4. Monitor spend - Console → **Billing** → live balance + Billing Explorer for history. - Per-endpoint cost: Console → Serverless → endpoint → cost/run. - The `runpod_spend` SQLite table records daily aggregated spend in cents — queryable from the web app. At $0.34/hr (RTX 4090 Community) and ~60s per generation after warm-up, real cost per job is ~$0.01. The plan's default `RUNPOD_EST_COST_CENTS_PER_JOB=15` (15¢) is conservative on purpose — covers cold starts. --- ## Rollback - **Stop the endpoint**: Console → Serverless → endpoint → **Stop**. Workers spin down, no further cost. Already-running requests complete. - **Disable dispatch**: unset `RUNPOD_API_KEY` on the web app — `runpodClient.ts:33-34` returns 501, sweep marks job as failed-attempt-1 and the home worker can still claim it. - **Revert code**: W1–W4 are independent; revert any single one without affecting the others. --- ## Known limitations (do not file as bugs) - **TRT only serves `stable-audio-3-small-music`.** Medium and SFX requests will be gated at dispatch (W2 in the plan) and routed to the home worker. Multi-variant TRT support is deferred until `install.sh --variant medium` is verified upstream. - **PyTorchEngine fallback** is fixed (W1) but rarely exercised — only triggers when cfg≠1.0 / init / inpaint / non-pingpong, none of which the product currently emits. - **OTel tracing is active.** `init_obs("brioso-runpod")` is called in the handler; spans export to Phoenix via the `https://usebrioso.com/otel` forward route (BRI-63). - **Manual spans and `jobs.trace_id` propagation for RunPod jobs are deferred to Phase 8.** RunPod-dispatched jobs will have `trace_id = NULL` until then. --- ## Doc links (canonical) - Billing: https://docs.runpod.io/accounts-billing/billing - Network volumes: https://docs.runpod.io/storage/network-volumes - Endpoint config: https://docs.runpod.io/serverless/endpoints/endpoint-configurations - Env vars: https://docs.runpod.io/serverless/development/environment-variables - API keys: https://docs.runpod.io/get-started/api-keys - Send requests: https://docs.runpod.io/serverless/endpoints/send-requests - Pricing: https://www.runpod.io/pricing --- ## [RunPod v2 endpoint cutover runbook](https://fitxers.usebrioso.com/internal/runpod-v2-cutover/) (Internal) > Retired in BRI-439: the RunPod runtime is gone. # RunPod v2 endpoint cutover runbook Move production dispatch from the legacy medium endpoint (`xjke3k5jsm966n`, network-volume bound, regional HA risk) to the three v2 endpoints (multi-region, scale-to-zero, FlashBoot, cu128 fixed). ## State (updated 2026-07-24) The v2 endpoints **pin the image directly at the endpoint level** — there is no RunPod template. The old template IDs below (`gajgxcbzpp` / `c1ftlcbzc1` / `03wo6f930b`) are gone (a `get-template` on them 404s); ignore them. The real image repo is the private `usebrioso/brioso-runpod` on Docker Hub (**not** `espetrinho/brioso-sa3-runpod`, which was never correct). | Endpoint | Variant | Image (endpoint-level) | Registry auth | Role | |---|---|---|---|---| | `xjke3k5jsm966n` | medium (v1) | template `rfesur7g6m`, netvol `o8uv2ik5u0` | — | legacy prod | | `oqgxuymb9odwsf` | medium (v2) | `usebrioso/brioso-runpod:v5` ✅ live | `brioso-docker-hub-runpod` (`cmryqcsj80054mycphgt1tiou`) | prod | | `awuc2vcxamnjzs` | small-music (v2) | old tag — **needs repoint to `:v5`** | old (deleted) — **needs reattach** | not serving | | `eqmrfos88whzly` | small-sfx (v2) | old tag — **needs repoint to `:v5`** | old (deleted) — **needs reattach** | not serving | All v2 endpoints: `networkVolumeId=null`, `containerDiskInGb=30`, model cache configured with pinned SHAs. ### 2026-07-24 incident — stale registry auth Prod SA3 (medium, `oqgxuymb9odwsf`) went down. It *looked* like the BRI-46 `variant=` TypeError, but that code bug was already fixed in `usebrioso/brioso-runpod:v5`. Real cause: **both stored RunPod container-registry credentials had been rotated/revoked**, so RunPod could no longer pull the private repo (`IMAGE_AUTH_ERROR: pull access denied`). The old image kept limping only because its workers had it cached; repointing to `:v5` forced a fresh pull that exposed the dead creds. Fix: new Docker Hub PAT (Read scope) on the `usebrioso` account → RunPod container-registry auth `brioso-docker-hub-runpod` → attach to the endpoint → repoint image to `:v5`. Diagnose pull-vs-boot failures from the dashboard **Logs tab**, not worker status (both show `INITIALIZING→UNHEALTHY`). Web app reads endpoint IDs via `cfg.get()` (`apps/web/src/server/dispatch/runpodClient.ts:25-34`). Variant map: ``` medium → RUNPOD_ENDPOINT_ID_MEDIUM (falls back to RUNPOD_ENDPOINT_ID) small-music → RUNPOD_ENDPOINT_ID_SMALL_MUSIC small-sfx → RUNPOD_ENDPOINT_ID_SMALL_SFX ``` Currently `RUNPOD_ENDPOINT_ID=xjke3k5jsm966n` (legacy). The per-variant vars are unset, so medium falls through to the legacy endpoint. ## Prerequisites - [ ] PR #77 merged to main (preload_only warmup — prevents regression if deployed before cutover; old endpoint silently ignores preload_only) - [ ] All v2 endpoints healthy: `GET /v2//health` shows ready+idle workers, zero unhealthy - [ ] `gh` authenticated, Infisical prod access ## Cutover steps ### 1. Update Infisical prod environment Set these in Infisical (env=prod, projectId=INFISICAL_PROJECT_ID): ``` RUNPOD_ENDPOINT_ID_MEDIUM=oqgxuymb9odwsf RUNPOD_ENDPOINT_ID_SMALL_MUSIC=awuc2vcxamnjzs RUNPOD_ENDPOINT_ID_SMALL_SFX=eqmrfos88whzly ``` Blank or remove the legacy var (only used as medium fallback once the per-variant var is unset): ``` RUNPOD_ENDPOINT_ID= ``` Leave `RUNPOD_API_KEY`, `WORKER_SHARED_SECRET`, and other dispatch knobs unchanged. ### 2. Deploy ```bash gh workflow run deploy.yml ``` This triggers `.github/workflows/deploy.yml` which: Tailscale SSH to VPS → `deploy/scripts/deploy-vps.sh` → `sync-secrets.sh` (pulls Infisical snapshot into `/opt/brioso/shared/web.env`) → build → systemd restart → health check. Watch the run: `gh run watch`. ### 3. Verify traffic shifted Poll both endpoint health counters (workersMax on old endpoint means it will still hold one worker even with zero traffic): ```bash APIKEY= # Old endpoint — completed should flatten, inQueue should stay 0 curl -s -H "Authorization: Bearer $APIKEY" https://api.runpod.ai/v2/xjke3k5jsm966n/health # New medium — completed should start incrementing curl -s -H "Authorization: Bearer $APIKEY" https://api.runpod.ai/v2/oqgxuymb9odwsf/health ``` Send a real test job via the web UI (anonymous, OTP-verified) and confirm it dispatches to `oqgxuymb9odwsf` (check `leased_by` in `jobs` row or RunPod dashboard). ### 4. Retire the old endpoint Only after confident the new endpoints carry prod for at least one full day: ```bash runpodctl serverless update xjke3k5jsm966n --workers-max 0 # Then delete via dashboard (CLI delete is unreliable for endpoints with volumes) ``` Do not delete the network volume `o8uv2ik5u0` until you are sure nothing else references it. ## Rollback Set Infisical back to: ``` RUNPOD_ENDPOINT_ID_MEDIUM= RUNPOD_ENDPOINT_ID=xjke3k5jsm966n ``` Leave the small-music/sfx vars alone (they have no legacy fallback, so they keep routing to v2 — which is fine, those variants never had a prod endpoint before). Redeploy via `gh workflow run deploy.yml`. ## Known gotchas - **Cold start**: v2 endpoints scale to zero. First job after idle pays 30-60s spawn + image pull. The OTP-time warmup (PR #77) and FlashBoot mitigate this. If latency is unacceptable, set `workersMin=1` per endpoint (costs idle rate, ~$0.28/hr/endpoint with FlashBoot discount). - **Image size**: v3 is 22.4GB. Cold host pulls take 3-5min. This is the real cold-start bottleneck; the model cache eliminates HF download but not image pull. - **`runpodctl serverless update --model-reference`** has a flashBootType bug (sends empty string). Use raw GraphQL `saveEndpoint(input:{id,name})` or the dashboard for model cache config. - **Old endpoint stuck state observed 2026-07-22**: 0 workers, 1 job queued, workersMax=1. Likely a capacity or template degradation on the legacy path — another reason to cut over promptly. --- ## [Screens registry guide (\`.agents/screens/\`)](https://fitxers.usebrioso.com/internal/screens-registry-guide/) (Internal) # Screens registry guide (`.agents/screens/`) `.agents/screens/` is the canonical ASCII map of every rendered screen and route. Agents (QA, dogfood, navigation, planning) rely on it instead of loading each page, so it must stay accurate at every commit. **Stale ASCII is worse than none** — it misleads the next agent into acting on a UI that no longer exists. The registry has two layers: - `screen-INDEX.txt` — master list: user flows, screen list (name / route / source file), layout shells, component roots, retired routes, API-only routes, styling, analytics, known gaps. Bump its `generated:` line on edit. - `screen--.txt` — one per screen. Frontmatter (`screen` / `name` / `route` / `file`), an ASCII diagram in a fenced block, a `## Elements` table (`Element` / `State` / `Description`), and a `## Behavior` section. Flow buckets: `discovery`, `create`, `studio`, `auth`, `admin`. `scripts/check-screen-staleness.ts` (wired into `mise run validate` and `.github/workflows/security.yml`) automates three checks: freshness (`generated:` date in `screen-INDEX.txt` is ≤30 days old), coverage (every route in `routeTree.gen.ts`'s `FileRoutesByFullPath` interface is referenced in the INDEX), and a known stale-copy pattern (`Sign in -> /brand-kit`). It cannot check ASCII/Elements/Behavior fidelity — that's on the agent making the change. ## When you MUST update the screens Update the affected `screen--.txt` (ASCII + Elements + Behavior) AND `screen-INDEX.txt` when a change alters anything a user or agent would see or hit. That includes: - visible layout, copy, elements, states, or interactions on any screen - a route's path, auth gate, redirect target, or post-action destination - adding, removing, or repurposing a screen or route - a layout shell or shared chrome component (`Header`, `Footer`, `StudioChrome`, layout shells) that changes multiple screens at once ## When you do NOT need to update - pure backend / API-only route changes with no rendered UI (the INDEX lists these under "API-only routes") - refactors with zero visual or routing change - dependency bumps, build config, tests ## New screen 1. Add `screen--.txt` (next free number in that flow) with frontmatter, an ASCII diagram matching the rendered page, an Elements table, and Behavior. 2. Add a row to `screen-INDEX.txt`'s "screen list" with name / route / source file. 3. If it introduces a new flow bucket, add it under "user flows". ## Removed or repurposed route 1. Move its `screen-INDEX.txt` row from "screen list" to "retired routes", noting the redirect target or a `DELETED` marker, matching the existing format. 2. Delete the `screen--.txt` file unless the screen is still reachable. ## Fidelity rules - The ASCII diagram must match the actual rendered screen at the current commit. Re-read the component source; never copy an old diagram. - Use callout arrows (`←`) for non-obvious regions, as the existing files do. - Keep the Elements table `State` values precise (`required`, `conditional`, `action`, `ambient`, `static`) — agents reason about affordances from these. - If a change can't be fully captured in ASCII, update the Elements/Behavior prose and add an entry under "known gaps" in `screen-INDEX.txt`. ## Pre-finish check If you touched anything under `apps/web/src/routes/` or a shared chrome component, ask: did the rendered screen change? If yes, the matching `screen-*.txt` and `screen-INDEX.txt` must change in the same commit. --- ## [Worker secrets sync (BRI-427)](https://fitxers.usebrioso.com/internal/secrets-sync/) (Internal) # Worker secrets sync (BRI-427) Infisical is the only source of truth for worker-facing runtime secrets. `deploy-secrets-sync.yml` renders the configured key set from Infisical and pushes it into each configured runtime's remote secret store on every beta tag, then redeploys the runtime's app so the new values actually take effect. ## What syncs where | From | To | Keys | |---|---|---| | Infisical `staging` (the beta scope — no `beta` env exists) | Modal secret `brioso-worker` (account `espetro`, env `beta`) | the six keys in `config/secrets-sync.yaml` | The key set and each key's source (`infisical` vs `literal` ops config) live in `config/secrets-sync.yaml` — change policy there, never in the workflow. `literal` is only for non-secret config (runtime ids, model names, URLs); the script refuses a `literal` under a secret-shaped name. ## When it runs - Every `v*-beta.*` tag push (same trigger family as the deploy pipeline). - Manually: Actions → *Worker secrets sync* → `target=beta`, `dry_run` to preview (render + names-only drift report; no writes, no redeploy). ## What a run does (per configured runtime) 1. Render `staging` from Infisical into `$RUNNER_TEMP` (0600) — same universal-auth export as `deploy-pyinfra.yml`. 2. `secrets_sync.py render` resolves the runtime's keys → JSON in `$RUNNER_TEMP` (names only on stdout). 3. `modal environment create beta` (idempotent). 4. Pre-sync `assert-live` — **warn-only**: drift is the input to a sync, not a failure. 5. `modal secret create -e beta --force brioso-worker --from-json …` — replaces the whole secret (extra live keys are dropped). 6. Post-sync `assert-live` — **hard gate**: every rendered name must now be live or the job fails. 7. `modal deploy -e beta apps/modal-worker/modal_app.py` — **required**: the deployed app pins a secret snapshot, so a `--force`d secret is invisible until redeploy (this was incident 9 of the 2026-10-06 beta-tag chain). 8. Job summary: names + `sha256(value)` digests (D3 `EnvKeyFingerprint` convention — digests cross the wire, values never do). ## Drift detection limits (deliberate) Modal exposes **no** API that returns a live secret's key names or values (`modal secret list` prints secret names only). `assert-live` therefore uses the server-side `required_keys` assert on `SecretGetOrCreate` — it catches a live secret that is *missing* rendered keys, and nothing else. The other direction (live has extra keys) self-heals: `--force` replaces the whole secret. Value drift is unobservable through the API by design — the sync itself is the convergence (an authoritative push each beta tag). ## Ops prerequisites (one-time) - GitHub secrets (already used by `deploy-modal-worker.yml`): `MODAL_ESPETRO_TOKEN_ID` / `MODAL_ESPETRO_TOKEN_SECRET` — deploy-scope token for the `espetro` Modal workspace. - Infisical `staging` (project `brioso`) must carry: - `WORKER_SHARED_SECRET` — **required** (the existing beta value; sync fails loudly if absent). - `CF_SERVICE_TOKEN_ID` / `CF_SERVICE_TOKEN_SECRET` — **optional** (`required: false`): synced when present, warned-and-omitted when not. Add them if beta sits behind Cloudflare Access. - `JOBSTORE_API_URL`, `BRIOSO_WORKER_RUNTIME`, `BRIOSO_WORKER_MODEL` are checked-in literals in `config/secrets-sync.yaml` — flip any to `source: infisical` to drive it from Infisical instead. ## RunPod Not wired. The repo has no endpoint env-update mechanism (`runpod-v2-cutover.md` updates endpoint env via dashboard/`saveEndpoint`), the only RunPod runtime in `gpu-gateway.beta.yaml` is disabled/unmerged (`kind: runpod-pod`, BRI-357), and RunPod serverless endpoints are push-runners that don't consume this pull-worker key set. Extension point: add a `kind: runpod-endpoint` runtime once the endpoint-id / `RUNPOD_API_KEY` provisioning decision is made; `scripts/secrets_sync.py` fails loudly on unknown kinds until then. --- ## [Security Runbook: API Schema Exposure Prevention](https://fitxers.usebrioso.com/internal/security-runbook/) (Internal) # Security Runbook: API Schema Exposure Prevention ## The Incident (2026-07) `/openapi.json` was publicly accessible because it relied solely on edge-level (Cloudflare Access) auth, which was misconfigured. This happened **twice**. The route has been deleted, but the underlying class of bug — sensitive routes accidentally exposed without auth — is easy to repeat. This runbook documents the layered defenses now in place to keep it from happening a third time. ## Defense Layers ### 1. Runtime: Nitro plugin (`apps/web/plugins/sensitive-route-guard.ts`) Intercepts every request at the server level. If the path matches a sensitive prefix (`/admin`, `/dispatch`, `/openapi`, `/graphql`, `/schema`, `/swagger`, `/internal`) and is not in the public exception list (`/health`, `/admin/login`, `/admin/logout`), the plugin requires either: - a valid `WORKER_SHARED_SECRET` Bearer token, OR - a valid admin session cookie. Bypasses route-level guards. **Cannot be removed by deleting a route handler.** ### 2. Commit gate: lefthook `lefthook.yml` pre-commit runs `scripts/check-sensitive-routes.ts` against staged route files, alongside `check-workspace-manifests.ts`. Raw observability imports and cross-package boundaries are enforced separately, as `.oxlintrc.json` overrides and ruff's `TID251` banned-api rule, both run inside `mise run validate`. `check-sensitive-routes.ts` reads the route path from `createFileRoute("...")` and requires an auth marker (`requireWorkerSecret`, `requireAdmin`, `middleware`, or `sensitiveRouteGuard`) on any sensitive path. `mise run validate` (pre-push and CI) runs the same script with `--all` for a repo-wide sweep. ### 3. CI gate: GitHub Actions `.github/workflows/security.yml` runs three jobs on PRs to `main` and pushes to `main`: - **semgrep-scan**: runs the rules in `.semgrep/rules/sensitive-routes.yml` - **route-auth-audit**: runs `scripts/check-sensitive-routes.ts --all` to scan every route file in the repo - **canary-check** (push only): curls `https://usebrioso.com/openapi.json` and fails the job if it returns 200. This catches the case where a route was deleted but the response is cached at the edge. ### 4. Documentation `AGENTS.md` has a "Security Invariants" section that agents must read before adding new server routes. ### 5. Server function auth Account-level server functions (e.g. `delete-account-fn.ts`) use `requireUser()` from `apps/web/src/server/auth/session.ts`. This resolves the session cookie and throws 401 if the user is not authenticated. These functions are not file routes, so they are not caught by `check-sensitive-routes.ts`, but the `requireUser()` gate is mandatory for any function that mutates user data. ## If a Sensitive Endpoint Returns 200 in Production 1. Verify the `sensitive-route-guard` plugin is registered in `vite.config.ts` (`plugins: ["./plugins/logging.ts", "./plugins/sensitive-route-guard.ts"]`). 2. Verify the endpoint path is in the `SENSITIVE_PREFIXES` list. 3. Check if the endpoint was added to the `PUBLIC_PATHS` exception list by mistake (it shouldn't have been — exception list reviews are required). 4. Investigate which commit removed or bypassed the route-level guard. 5. Invalidate any edge cache for the path. ## Adding a New Sensitive Endpoint 1. Pick the runner: worker (requires `WORKER_SHARED_SECRET`) or admin (requires signed session cookie). 2. Add an auth check at the top of the route handler: ```ts const gate = requireWorkerSecret(request); if (gate) return gate; ``` 3. The Nitro plugin will automatically cover the endpoint if the path matches a sensitive prefix. No additional work needed. 4. If the prefix is not in `SENSITIVE_PREFIXES`, add it. ## Adding a New Public Endpoint (rare) This is **discouraged**. Only do it for: - webhook endpoints that authenticate via a provider signature (e.g. `/stripe/webhook`) - monitoring endpoints that intentionally expose health data - explicit allowlist with documented justification If you must: 1. Get explicit sign-off from the project owner. 2. Add the path to `PUBLIC_PATHS` in the plugin. 3. Add a comment in the plugin explaining why this exception exists. ### Existing exceptions #### `GET /dev/login?persona=<...>|email=<...>@brioso.test&redirect=` (local test login) Mints a real session cookie for a `@brioso.test` user — either a named `persona` (see `PERSONAS` in `apps/web/src/routes/dev/login.ts`) or an ad-hoc `email` — so an agent or developer can authenticate against a local dev server without pasting `document.cookie = ...` into devtools. Not registered with the Nitro `sensitive-route-guard` plugin — that plugin gates on `WORKER_SHARED_SECRET`, which is the wrong credential for a route whose entire purpose is minting a *user* session before any secret exchange has happened. **Guard is request-time, not a route-level auth marker:** the handler's first statement returns `404` when `process.env.NODE_ENV === "production"`, and it is also gated behind `import.meta.env.DEV` so it tree-shakes out of the production bundle entirely. `scripts/check-sensitive-routes.ts`'s `AUTH_MARKERS` list includes the literal `NODE_ENV === "production"` string so the commit gate recognizes this pattern instead of silently skipping the route — see that file's comment. **Further narrowing:** even if both gates above were bypassed, the handler refuses any email not ending in `@brioso.test`, so it can never mint a session for a real account. #### `GET /cosmos/$fixture` (React Cosmos component sandbox) Routes to a fixture in the standalone React Cosmos dev server (`bunx cosmos`) via an iframe mount — see `apps/web/src/routes/cosmos/$fixture.tsx`. **Why it is in `SENSITIVE_PREFIXES` even though `/dev/login` is not:** `/cosmos` is registered with the Nitro `sensitive-route-guard` plugin (it takes the `WORKER_SHARED_SECRET`-Bearer credential, like `/dispatch/*`, `/openapi`, `/graphql`). Unlike `/dev/login` — whose purpose is to mint a *user* session before any secret exchange — `/cosmos` is downstream of an authenticated deploy, and the worker secret is the right credential to gate it. A misconfigured deploy that fails the guard will 503, not 200. **Component-level `import.meta.env.DEV` short-circuit** tree-shakes the iframe JSX out of production bundles; the server handler additionally returns 404 when `process.env.NODE_ENV === "production"`. The commit gate `scripts/check-sensitive-routes.ts` recognizes both patterns (see the `AUTH_MARKERS` list in that file). **Param narrowing:** the `$fixture` segment is validated against `/^[A-Za-z0-9._-]{1,128}$/` at both the server handler and the component, so path traversal or oversized input 404s before reaching the iframe. ## Rotation - `WORKER_SHARED_SECRET`: rotate quarterly. The secret lives on **three surfaces that do not sync** — rotate them in sequence: 1. Generate new value: `openssl rand -hex 32`. 2. `infisical secrets set WORKER_SHARED_SECRET= --env=dev` and `--env=prod` (apps/web picks it up on next `sync-secrets.sh` run / deploy). 3. Update the home worker daemon env the same way. 4. `gh secret set WORKER_SHARED_SECRET` (GitHub Actions deploy + stripe-doctor cron call `/internal/*` with it). 5. Smoke: a green `Deploy web` run confirms the GitHub↔apps/web pair (the deploy job curls `/internal/config-audit` with the secret). The `WORKER_SHARED_SECRET_ROLLOVER` overlap-window idea (second env var accepted during a 24h window) applies across all three surfaces. **Coupled secret:** `OTEL_RELAY_TOKEN` (worker side) MUST equal `WORKER_SHARED_SECRET` (apps/web side). The relay route `/otel/v1/traces` validates the inbound Bearer against `WORKER_SHARED_SECRET`. This is a deliberate reuse (BRI-63) to avoid a 4th secret. When rotating `WORKER_SHARED_SECRET`, rotate `OTEL_RELAY_TOKEN` on the home daemon to match. Drift = silent trace darkness (relay 403, exporter non-fatal). - `SESSION_SECRET`: rotate annually. Rotate immediately on suspected compromise. - API keys (Anthropic, Stripe, Modal proxy keys): rotate on personnel changes and suspected exposure. ## Audio Protection (2026-07) Three linked defenses against preview/master audio leaking as clean, ownership-free files or against one customer's job details leaking to another guesser. Tracked as BRI-103/104/105. ### Preview watermarking Preview streams (never masters) are watermarked at generation time — a Pocket TTS voice line (`developpeuse-3`), band-specific ducking, jittered per-track offsets so the watermark can't be reliably gated out by timing alone. Masters sent to a paying customer are never watermarked; this is verified by test, not just by convention. ### Preview streaming: proxy + Fetch-Metadata + playback token `GET /jobs/:slug/preview` proxies the R2 object through the app (never a redirect to a presigned URL, which would leak a bearer-style link anyone could replay or share) and requires two things: - a Fetch-Metadata check (`packages/auth-core/src/fetch-metadata.ts`) rejecting requests whose `Sec-Fetch-*` headers don't look like an in-page `