GPU gateway beta rollout runbook (BRI-420 / plan 3 v2 §5)

GPU gateway beta rollout runbook (BRI-420 / plan 3 v2 §5)

The human-executed checklist that takes the beta GPU gateway from
“stack merged” to “Modal runtime registered and importable”. Post-BRI-439
the gateway is push-only — no weights or ramp stages; dispatch order is
the fallbackChain. Day-2 operations live in
gpu-gateway-canary.md.

Who runs this: a human holding Infisical + Modal + GitHub credentials.
No VPS shell is needed beyond what the deploy pipeline already does; every
command below runs from your own shell/browser. Nothing in this doc is
executed by CI or an agent session.

Safety: set +o history; umask 077 before touching secrets, set -o history when done. Secrets go straight into Infisical/GitHub — never into
files, issues, or chat.

CLI verification status (vs pinned modal==1.3.5, D11 exact pin)

Anything not on this list is “verify before use” — check
modal <group> --help on the pinned version (uv tool run --from "modal==1.3.5" modal …) before scripting it.

Prerequisites

• U1–U4, U6, U7a and U9 are merged to beta (BRI-407, runtime ids,
  BRI-413 dispatch→wake, BRI-411 modal-worker, BRI-412 spend ledger,
  BRI-417 admin plumbing, BRI-418 local runtime) and deployed.
• Plan-2 gates green: the last deploy-pyinfra beta run passed
  drift-gate → validate → deploy → origin /health smoke →
  /internal/config-audit, and /ready reports ready.
• GitHub secrets exist: MODAL_ESPETRO_TOKEN_ID /
  MODAL_ESPETRO_TOKEN_SECRET (deploy token for the espetro account —
  the existing MODAL_TOKEN_ID/MODAL_TOKEN_SECRET pair serves
  account 1 if it holds a deploy token; see §5.1 of the plan).
• The home GPU box runs the BRI-437 push daemon (brioso-worker serve
  on :7311 — see home-worker-wake.md) and its
  tailnet hostname is known: config/gpu-gateway.beta.yaml ships
  push.url as http://REPLACE_WITH_HOME_TAILNET_HOST:7311/jobs, a
  placeholder you fill in before importing. HOME_WAKE_TOKEN is
  optional — uncomment push.tokenEnv only if a non-tailnet caller
  must submit; tailnet peers (tailscale whois) and loopback are
  authorized without it. BRIOSO_WAKE_ALLOW_NODES narrows which nodes
  may submit.

Step 1 — sanity: beta still works today

curl -fsS https://beta.usebrioso.com/health
curl -fsS https://beta.usebrioso.com/ready

Expected: both 200; /ready’s gateway block shows the effective
source and a valid config. If it fails: stop — fix beta first; the
gateway rollout rides on the deploy pipeline. Triage: hop 1–2 of the §7
table in gpu-gateway-canary.md.

Step 2 — per-Modal-account environment + secrets

Repeat for each Modal account runtime (today: espetro → modal-espetro).
Export that account’s deploy token first so the CLI acts on the right
workspace:

export ACCT=espetro RUNTIME=modal-espetro
export MODAL_TOKEN_ID=… MODAL_TOKEN_SECRET=…   # this account's deploy token
modal environment create beta || true          # verified: 1.3.5; "exists" is fine
modal secret create --env beta --force brioso-worker \
  BRIOSO_WORKER_MODEL=audiocpp
modal secret create --env beta --force brioso-registry \
  REGISTRY_USERNAME="$REGISTRY_USERNAME" \
  REGISTRY_PASSWORD="$REGISTRY_PASSWORD"

Post-BRI-439 the Modal app never calls the backend — generate writes to
R2 via the payload’s presigned URLs, so the pull-era keys
(JOBSTORE_API_URL, WORKER_SHARED_SECRET, BRIOSO_WORKER_RUNTIME) are
gone from brioso-worker; config/secrets-sync.yaml is the source of
truth for what the app still needs.

Then create the proxy-auth token pair (wk-…/ws-…): Modal dashboard →
Settings → Proxy Auth Tokens (the secret is shown once). Store it straight
into Infisical — new keys, not a rotation:

infisical secrets set MODAL_ESPETRO_PROXY_KEY=<wk-…> \
  MODAL_ESPETRO_PROXY_SECRET=<ws-…> --env=staging

Expected: modal secret create prints the created/updated secret; the
dashboard shows the new proxy token; Infisical lists both new keys.
If it fails: environment create error other than “already exists” →
check MODAL_TOKEN_* is that account’s deploy token (modal token info,
modal profile current). Secret-create failure on --env beta → the
environment wasn’t created; rerun the secret-create above. Wrong-workspace
surprise → modal token info names the workspace you’re actually authed
as.

Step 3 — deploy the Modal worker (per account/env)

Actions → deploy-modal-worker → account=espetro, environment=beta,
confirm=yes. Human action — the workflow spends third-party compute.

Expected: deploy log ends with Created web function submit => https://espetro-beta--brioso-worker-submit.modal.run (plus status and
cancel); the step summary prints them. Record the printed submit URL —
it must match the {workspace}-{env}--{app}-submit.modal.run shape that
the beta YAML’s modal: {workspace, environment, app} block produces
(espetro + beta → espetro-beta--brioso-worker-submit.modal.run).
If the real URL differs, fix the YAML’s modal block before importing.
If it fails: missing MODAL_<ACCT>_TOKEN_* secrets → add them under
repo/org secrets and re-run; no submit => line → the app file didn’t
expose the endpoint — check the run log against the workflow’s
URL-extraction step.

Step 4 — redeploy beta through the plan-2 pipeline

So CI re-renders the env (Infisical hash check) with the new
MODAL_ESPETRO_PROXY_* names present:

bun run release:beta        # cuts vYYYY.WW.N-beta.M → deploy-pyinfra.yml
# or: Actions → "Deploy web (pyinfra)" → env=beta, ref=<sha>

(deploy-beta.yml is dead audit-trail only — do not dispatch it.)

Expected: deploy-pyinfra run is green end-to-end: drift-gate →
validate → deploy → origin /health smoke → /internal/config-audit.
If it fails: drift-gate red → real VPS drift, resolve before deploying;
config-audit 503 → a REQUIRED key is missing from the rendered env — its
output names the key; smoke sha-pin mismatch → the wrong artifact deployed.

Step 5 — zero-GPU probe checks

curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  "https://espetro-beta--brioso-worker-submit.modal.run" -d '{}'          # 401
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  -H "Modal-Key: <wk-…>" -H "Modal-Secret: <ws-…>" \
  "https://espetro-beta--brioso-worker-submit.modal.run" -d '{}'          # 400 (validated, not spawned)

An empty body with the proxy pair returns 400 (missing … from
validate_push_job) — it exercises auth + validation without spawning a
GPU call. A real end-to-end submit is covered by the admin canary job
below, not curled by hand.

Expected: 401 without the pair; 400 with it. If it fails: 401
with the pair → the keys are deploy tokens, not proxy tokens — create a
proxy-auth pair in the dashboard and update Infisical. 404 → wrong URL or
app name (compare the recorded deploy URL from step 3). Timeout → Modal
control plane — retry once, then hop 3–4 of the ops table in
gpu-gateway-canary.md. Also verify in the admin
console (tailnet): beta → GPU Gateway → modal-espetro shows health
cold/ready and budget $10.

Step 6 — import the beta config

1. Export first — import is replace semantics (rows absent from the
   document are deleted with their key rings):
   GET /api/admin/beta/gpu-gateway/export (admin console → export, or the
   hook route) and save the YAML as the rollback file.
2. Import config/gpu-gateway.beta.yaml from this repo
   (POST /api/admin/beta/gpu-gateway/import with
   { "document": "<file text>" }, or the UI’s import affordance — pass
   ifMatchVersion from the export when posting by hand).
3. Confirm the registry shows home then modal-espetro, chain
   [home, modal-espetro] — both enabled. The chain order IS the policy:
   home serves when healthy, modal-espetro is the fallback. There are no
   weights to park.

Expected: applied: true, the two rows in inference_routes with
chain_position 0/1. If it fails: 400 → schema rejection (the error
names the field — the checked-in file is CI-verified, so a 400 means the
doc was hand-edited or a deployed schema predates the file’s fields);
412 → someone edited since your export — re-export and re-import. A
runtime row landing but health: unreachable → MODAL_ESPETRO_PROXY_*
names don’t resolve in beta env — back to step 4’s config-audit output.

Done-state

Registry imported with chain [home, modal-espetro], both enabled, the
submit endpoint auth-verified, and the home daemon’s real tailnet host
filled into push.url. Smoke it with an admin job pinned to
runner: modal-espetro (admin console), then watch day-2 operations in
gpu-gateway-canary.md.

Appendix A — admin “still doesn’t work” triage (plan §5.3)

Human on tailnet, read-only:

1. curl -sI https://admin.usebrioso.com/dash/ from a tailnet device. No
   route → you’re not on the tailnet, or Caddy is down.
2. systemctl status pocketbase and grep ADMIN_UPSTREAM /etc/brioso/pocketbase.env. Post-U7a the env keys are
   ADMIN_UPSTREAM_PROD/ADMIN_UPSTREAM_BETA (loopback :8090/:8091 — no
   slot ports); a leftover slot port (:3000/:3002) is BRI-366.
3. The GPU tab loads but edits do nothing on prod: prod’s chain is
   config/gpu-gateway.prod.yaml — import it once prod’s deploy carries
   the push-only schema (same replace-semantics caveat).