RunPod v2 endpoint cutover runbook

Retired in BRI-439: the RunPod runtime is gone.

RunPod v2 endpoint cutover runbook

Move production dispatch from the legacy medium endpoint (xjke3k5jsm966n, network-volume bound, regional HA risk) to the three v2 endpoints (multi-region, scale-to-zero, FlashBoot, cu128 fixed).

State (updated 2026-07-24)

The v2 endpoints pin the image directly at the endpoint level — there is no RunPod
template. The old template IDs below (gajgxcbzpp / c1ftlcbzc1 / 03wo6f930b) are gone
(a get-template on them 404s); ignore them. The real image repo is the private
usebrioso/brioso-runpod on Docker Hub (not espetrinho/brioso-sa3-runpod, which was
never correct).

All v2 endpoints: networkVolumeId=null, containerDiskInGb=30, model cache configured with
pinned SHAs.

2026-07-24 incident — stale registry auth

Prod SA3 (medium, oqgxuymb9odwsf) went down. It looked like the BRI-46 variant=
TypeError, but that code bug was already fixed in usebrioso/brioso-runpod:v5. Real cause:
both stored RunPod container-registry credentials had been rotated/revoked, so RunPod
could no longer pull the private repo (IMAGE_AUTH_ERROR: pull access denied). The old image
kept limping only because its workers had it cached; repointing to :v5 forced a fresh pull
that exposed the dead creds. Fix: new Docker Hub PAT (Read scope) on the usebrioso account →
RunPod container-registry auth brioso-docker-hub-runpod → attach to the endpoint → repoint
image to :v5. Diagnose pull-vs-boot failures from the dashboard Logs tab, not worker
status (both show INITIALIZING→UNHEALTHY).

Web app reads endpoint IDs via cfg.get() (apps/web/src/server/dispatch/runpodClient.ts:25-34). Variant map:

medium       → RUNPOD_ENDPOINT_ID_MEDIUM (falls back to RUNPOD_ENDPOINT_ID)
small-music  → RUNPOD_ENDPOINT_ID_SMALL_MUSIC
small-sfx    → RUNPOD_ENDPOINT_ID_SMALL_SFX

Currently RUNPOD_ENDPOINT_ID=xjke3k5jsm966n (legacy). The per-variant vars are unset, so medium falls through to the legacy endpoint.

Prerequisites

• PR #77 merged to main (preload_only warmup — prevents regression if deployed before cutover; old endpoint silently ignores preload_only)
• All v2 endpoints healthy: GET /v2/<id>/health shows ready+idle workers, zero unhealthy
• gh authenticated, Infisical prod access

Cutover steps

1. Update Infisical prod environment

Set these in Infisical (env=prod, projectId=INFISICAL_PROJECT_ID):

RUNPOD_ENDPOINT_ID_MEDIUM=oqgxuymb9odwsf
RUNPOD_ENDPOINT_ID_SMALL_MUSIC=awuc2vcxamnjzs
RUNPOD_ENDPOINT_ID_SMALL_SFX=eqmrfos88whzly

Blank or remove the legacy var (only used as medium fallback once the per-variant var is unset):

RUNPOD_ENDPOINT_ID=

Leave RUNPOD_API_KEY, WORKER_SHARED_SECRET, and other dispatch knobs unchanged.

2. Deploy

gh workflow run deploy.yml

This triggers .github/workflows/deploy.yml which: Tailscale SSH to VPS → deploy/scripts/deploy-vps.sh → sync-secrets.sh (pulls Infisical snapshot into /opt/brioso/shared/web.env) → build → systemd restart → health check.

Watch the run: gh run watch.

3. Verify traffic shifted

Poll both endpoint health counters (workersMax on old endpoint means it will still hold one worker even with zero traffic):

APIKEY=<runpod api key>
# Old endpoint — completed should flatten, inQueue should stay 0
curl -s -H "Authorization: Bearer $APIKEY" https://api.runpod.ai/v2/xjke3k5jsm966n/health
# New medium — completed should start incrementing
curl -s -H "Authorization: Bearer $APIKEY" https://api.runpod.ai/v2/oqgxuymb9odwsf/health

Send a real test job via the web UI (anonymous, OTP-verified) and confirm it dispatches to oqgxuymb9odwsf (check leased_by in jobs row or RunPod dashboard).

4. Retire the old endpoint

Only after confident the new endpoints carry prod for at least one full day:

runpodctl serverless update xjke3k5jsm966n --workers-max 0
# Then delete via dashboard (CLI delete is unreliable for endpoints with volumes)

Do not delete the network volume o8uv2ik5u0 until you are sure nothing else references it.

Rollback

Set Infisical back to:

RUNPOD_ENDPOINT_ID_MEDIUM=
RUNPOD_ENDPOINT_ID=xjke3k5jsm966n

Leave the small-music/sfx vars alone (they have no legacy fallback, so they keep routing to v2 — which is fine, those variants never had a prod endpoint before). Redeploy via gh workflow run deploy.yml.

Known gotchas

• Cold start: v2 endpoints scale to zero. First job after idle pays 30-60s spawn + image pull. The OTP-time warmup (PR #77) and FlashBoot mitigate this. If latency is unacceptable, set workersMin=1 per endpoint (costs idle rate, ~$0.28/hr/endpoint with FlashBoot discount).
• Image size: v3 is 22.4GB. Cold host pulls take 3-5min. This is the real cold-start bottleneck; the model cache eliminates HF download but not image pull.
• runpodctl serverless update --model-reference has a flashBootType bug (sends empty string). Use raw GraphQL saveEndpoint(input:{id,name}) or the dashboard for model cache config.
• Old endpoint stuck state observed 2026-07-22: 0 workers, 1 job queued, workersMax=1. Likely a capacity or template degradation on the legacy path — another reason to cut over promptly.