RunPod Bootstrap — manual setup runbook

Retired in BRI-439: the RunPod runtime is gone.

Pre-v2 setup; for current RunPod ops see runpod-v2-cutover.md (BRI-357).

RunPod Bootstrap — manual setup runbook

Companion to .agents/plans/2026-07-18-runpod-bootstrap.md. Walks through every manual step to get the first SA3 job dispatched and returning tracks.

Status (2026-07-18): Phases A–C are done. The endpoint is live and returning tracks —
see apps/runpod/README.md for what was actually deployed (endpoint id, image tag, GPU,
region) and for the container gotchas discovered while building it. Phase D (web app wiring
and the end-to-end test) is still outstanding.

Two corrections from doing it for real:

• The image is PyTorch-only; the TensorRT path was stripped because it cannot be built
  without a GPU present (tracked in BRI-18). The “TRT only serves small-music” limitation
  below therefore does not apply yet — the endpoint runs stable-audio-3-medium.
• Measured cost is higher than the ~$0.01/job estimated below: a 3x30s job takes ~43s of
  RTX 4090 time plus platform fee. Re-measure on a full [[3,180],[2,160]] job before
  trusting RUNPOD_EST_COST_CENTS_PER_JOB.

Prerequisites

• RunPod account (https://www.runpod.io)
• Docker locally with buildx
• A container registry account (Docker Hub or GHCR recommended)
• WORKER_SHARED_SECRET value (must match what’s on the VPS web app — ask yourself / check VPS env)
• GlitchTip DSN — get from https://eu.glitchtip.com/brioso/settings/keys/ (placeholder only in apps/runpod/.env.example)

Phases

• Phase A — RunPod account prep (parallelizable now, no code dependency)
• Phase B — Image build + push (after W1–W4 from the plan are merged)
• Phase C — Endpoint creation + first job (after Phase B + image pushed)
• Phase D — Web app wiring + verification

---

Phase A — RunPod account prep

A1. Add credits

Console → Billing → Add credits.

• Recommended starting balance: $50 (covers ~150 hr of RTX 4090 Community at $0.34/hr — plenty for first-week testing)
• Payment: credit card (Stripe), crypto, or wire for >$5k
• Enable Low balance alert at $20
• Optional: Auto-pay with $50 reload at $10 threshold

Default spend cap is $80/hr across all resources. Fine for now; raise via support ticket later.

Doc: https://docs.runpod.io/accounts-billing/billing

A2. Create network volume for HF cache

Console → Storage → Network Volumes → Create.

Mount path inside container is automatic at /runpod-volume — do not override.

⚠️ Region locks the endpoint. Once a volume is attached, the endpoint can only spin up workers in that volume’s datacenter. Pick a region known to have RTX 4090 + A6000 inventory.

Cost: $0.07/GB/month ≈ $7/month for 100 GB.

Doc: https://docs.runpod.io/storage/network-volumes

A3. Pick a container registry

RunPod has no first-party registry. We use Docker Hub:

• Docker Hub: docker.io/usebrioso/brioso-runpod:<tag> — auth via DOCKER_USERNAME / DOCKER_TOKEN (root .env); build-and-push.sh logs in automatically.

For private images, you’ll need to register credentials in RunPod later (Phase C5).

---

Phase B — Image build + push (after W1–W4 merged)

B1. Authenticate to your registry

# Docker Hub — build-and-push.sh logs in automatically when
# DOCKER_USERNAME/DOCKER_TOKEN are set in root .env. Manual login only for one-off pushes:
#   echo "$DOCKER_TOKEN" | docker login -u "$DOCKER_USERNAME" --password-stdin

B2. Build the image

From repo root:

docker build -f apps/runpod/Dockerfile.sa3 -t docker.io/usebrioso/brioso-runpod:$(git rev-parse --short HEAD) .

Expected: long build (~20–40 min on first run due to git clone of stable-audio-3 + TRT engine compile via install.sh). Subsequent builds cached.

B3. Push

docker push docker.io/usebrioso/brioso-runpod:$(git rev-parse --short HEAD)

# also tag :latest for convenience
docker tag docker.io/usebrioso/brioso-runpod:$(git rev-parse --short HEAD) docker.io/usebrioso/brioso-runpod:latest
docker push docker.io/usebrioso/brioso-runpod:latest

The image is private on Docker Hub, so wire up registry auth (Phase C5) before endpoints can pull it.

---

Phase C — Endpoint creation + first job

C1. Create the serverless endpoint

Console → Serverless → New Endpoint.

Container image

GPU Configuration

Worker scaling

Advanced

Doc: https://docs.runpod.io/serverless/endpoints/endpoint-configurations

C2. Set environment variables on the endpoint

Endpoint → Settings → Environment Variables.

Doc: https://docs.runpod.io/serverless/development/environment-variables

C3. Capture endpoint ID

After creation, copy the Endpoint ID from the endpoint detail page URL or panel. Looks like abcd1234efgh. This becomes RUNPOD_ENDPOINT_ID on the web app.

C3a. Beta staging endpoint

beta.usebrioso.com (see deploy/README.md’s “Beta / staging environment”
section) gets its own duplicated medium endpoint, same image — sharing
prod’s endpoint would let a beta job’s GPU worker call back into the prod
app, since JOBSTORE_API_URL is set per-endpoint here, not in Infisical.
Repeat C1–C2 with:

Put the resulting endpoint id in staging’s RUNPOD_ENDPOINT_ID_MEDIUM.
Leave RUNPOD_ENDPOINT_ID_SMALL_* unset in staging — runpodClient.ts
returns a clean 501 for unconfigured variants rather than throwing. Serverless
scales to zero, so an idle beta endpoint costs nothing.

C4. Generate API key

Console → Settings → API Keys → Create API Key.

New key is prefixed rpa_. Copy immediately — RunPod does not store it. This becomes RUNPOD_API_KEY on the web app.

Doc: https://docs.runpod.io/get-started/api-keys

C5. (Only if image is private) Register registry credentials

Console → Settings → Container Registry Auths → Create.

Generates a containerRegistryAuthId. Attach it to the endpoint template under Advanced.

C6. Smoke test from the console

Endpoint → Requests tab → modify the JSON body:

{
  "input": {
    "slug": "smoke-test",
    "prompt": "warm acoustic guitar loop, 90 bpm, gentle fingerpicking",
    "customer_email": "smoke@brioso.co",
    "batches": [[1, 30]],
    "steps": 8,
    "cfg_scale": 1.0,
    "seed": 42
  }
}

Click Run. Watch the Logs tab for:

• Container boot + TRT engine load (~30–60s cold start)
• [runpod-handler] log lines from handler.py
• Generation progress
• Upload to R2 via presigned URLs (requires web app’s /jobs/{slug}/upload-urls route to accept the request — authenticate with WORKER_SHARED_SECRET)

Expected outcome: {"tracks": [{"idx": 0, "preview_key": "...", ...}]}. If you see {"error": "..."}, check Logs first, then GlitchTip for the captured exception.

C7. Smoke test via curl (alternative)

curl -X POST https://api.runpod.ai/v2/$RUNPOD_ENDPOINT_ID/runsync \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "slug": "smoke-test",
      "prompt": "warm acoustic guitar loop, 90 bpm",
      "customer_email": "smoke@brioso.co",
      "batches": [[1, 30]],
      "steps": 8,
      "cfg_scale": 1.0,
      "seed": 42
    }
  }'

Doc: https://docs.runpod.io/serverless/endpoints/send-requests

---

Phase D — Web app wiring + verification

D1. Set env vars on the VPS web app

Add to the web app’s env (whatever mechanism — systemd EnvironmentFile, dotenv, etc.):

RUNPOD_API_KEY=rpa_...
RUNPOD_ENDPOINT_ID=abcd1234efgh
DISPATCH_WAIT_SECONDS=45
DISPATCH_QUEUE_DEPTH_THRESHOLD=5
RUNPOD_DAILY_SPEND_CAP_CENTS=2000
RUNPOD_EST_COST_CENTS_PER_JOB=15

Restart the web app. Dispatch sweep is triggered externally (systemd timer / GH Actions cron per apps/web/src/routes/dispatch/sweep.ts) — confirm that scheduler is running.

D2. Verify runpod_spend table exists on target DB

The migration SQL exists at packages/db/src/migrations/20260711092351_swift_mastermind/migration.sql but packages/db/src/migrations/meta/_journal.json is missing, so mise run db:migrate cannot apply it. Check the table:

# against your prod libsql/SQLite
sqlite3 $DATABASE_URL "SELECT name FROM sqlite_master WHERE type='table' AND name='runpod_spend';"

If empty, apply manually:

sqlite3 $DATABASE_URL < packages/db/src/migrations/20260711092351_swift_mastermind/migration.sql

Without this table, sweep.ts:25-42 throws on every dispatch attempt.

D3. End-to-end test

With the home worker stopped (so it doesn’t claim first):

1. Create a job via the normal UI / API (cfg=1.0, sm-music — the defaults).
2. Wait DISPATCH_WAIT_SECONDS + ~30s for the sweep to fire.
3. Sweep logs should show: sweep started → dispatched to runpod.
4. RunPod logs should show the handler running.
5. Job in DB should end in state: "paid" with tracks populated.
6. R2 (brioso-jobs bucket) should have preview + master objects under the slug key namespace.

D4. Monitor spend

• Console → Billing → live balance + Billing Explorer for history.
• Per-endpoint cost: Console → Serverless → endpoint → cost/run.
• The runpod_spend SQLite table records daily aggregated spend in cents — queryable from the web app.

At $0.34/hr (RTX 4090 Community) and ~60s per generation after warm-up, real cost per job is ~$0.01. The plan’s default RUNPOD_EST_COST_CENTS_PER_JOB=15 (15¢) is conservative on purpose — covers cold starts.

---

Rollback

• Stop the endpoint: Console → Serverless → endpoint → Stop. Workers spin down, no further cost. Already-running requests complete.
• Disable dispatch: unset RUNPOD_API_KEY on the web app — runpodClient.ts:33-34 returns 501, sweep marks job as failed-attempt-1 and the home worker can still claim it.
• Revert code: W1–W4 are independent; revert any single one without affecting the others.

---

Known limitations (do not file as bugs)

• TRT only serves stable-audio-3-small-music. Medium and SFX requests will be gated at dispatch (W2 in the plan) and routed to the home worker. Multi-variant TRT support is deferred until install.sh --variant medium is verified upstream.
• PyTorchEngine fallback is fixed (W1) but rarely exercised — only triggers when cfg≠1.0 / init / inpaint / non-pingpong, none of which the product currently emits.
• OTel tracing is active. init_obs("brioso-runpod") is called in the handler; spans export to Phoenix via the https://usebrioso.com/otel forward route (BRI-63).
• Manual spans and jobs.trace_id propagation for RunPod jobs are deferred to Phase 8. RunPod-dispatched jobs will have trace_id = NULL until then.

---

Doc links (canonical)

• Billing: https://docs.runpod.io/accounts-billing/billing
• Network volumes: https://docs.runpod.io/storage/network-volumes
• Endpoint config: https://docs.runpod.io/serverless/endpoints/endpoint-configurations
• Env vars: https://docs.runpod.io/serverless/development/environment-variables
• API keys: https://docs.runpod.io/get-started/api-keys
• Send requests: https://docs.runpod.io/serverless/endpoints/send-requests
• Pricing: https://www.runpod.io/pricing