Modal + Beam runbook (GPU gateway v2, BRI-359)

Modal + Beam runbook (GPU gateway v2, BRI-359)

Retired in BRI-439: the Beam runtime kind and this runbook’s weight-routing
are gone. The live Modal app is apps/modal-worker (see
gpu-gateway-beta-runbook.md). Kept for history.

Operational guide for running the SA3 medium variant on Modal and Beam,
alongside the existing RunPod path. Both new providers reuse the same
container image and handler as RunPod (apps/runpod/Dockerfile.sa3 ->
apps/runpod/handler.py), so payloads, upload-urls result posting, and obs
are provider-agnostic.

• Modal app: apps/modal-sa3/modal_app.py
• Beam app: apps/beam-sa3/beam_deployment.py
• CI: .github/workflows/modal-deploy.yml, .github/workflows/beam-deploy.yml

One-time secret setup

CI (GitHub repo secrets, before first deploy run):

• MODAL_TOKEN_ID / MODAL_TOKEN_SECRET: Modal workspace -> API tokens,
  scoped to deploy the brioso-sa3 app.
• BEAM_TOKEN: Beam dashboard -> API keys.

Platform secret stores (mirror of the RunPod endpoint env):

• Modal: modal secret create brioso-sa3 JOBSTORE_API_URL=... WORKER_SHARED_SECRET=...
• Beam: beam secret create JOBSTORE_API_URL=... and
  beam secret create WORKER_SHARED_SECRET=...

Values come from Infisical, same entries the RunPod endpoints use. Leave
SA3_MODEL_ID unset so the handler defaults to the medium variant.

Deploy

1. Ensure the base image tag exists: the runpod image workflow
   (build-runpod.yml) must have pushed the tag referenced by
   SA3_IMAGE_TAG (repo variable, defaults to v8). Bump in both places
   together so RunPod, Modal, and Beam stay in lockstep.
2. Trigger deploy-modal-sa3 (or deploy-beam-sa3) via Actions ->
   Run workflow, typing yes in the confirm box. Also available via
   repository_dispatch for automation. Never on push: deploys pay
   third-party compute, and the CI runner policy (ci-runner-policy.md)
   plus the cost guardrails below rely on that gate.
3. Verify in the platform dashboard that the app deployed and one manual
   smoke job completes (track URLs land in the job row).

Local deploy alternative (from a trusted machine):

uv tool run modal deploy apps/modal-sa3/modal_app.py
uv tool run beam deploy apps/beam-sa3/beam_deployment.py

Pointing traffic at a provider

Routing lives in the inference_routes table (BRI-359 phase 2): each
variant has provider rows with a weight and an enabled flag, resolved by
the cached resolver in packages/dispatch. To send jobs to Modal:

• Set the medium variant’s modal row weight (e.g. runpod 90, modal 10) and
  enabled = true.
• Watch the weight-drift obs metric and job outcomes before raising the
  weight.
• Beam works the same way via its own row.

Shadow-test before weighting: dispatch a single seeded job with the route
temporarily pinned to modal/beam (weight 100), confirm track upload works,
then return to blended weights.

Rollback

Set the provider row’s enabled flag to false (or weight 0) in
inference_routes; the resolver stops selecting it within the cache TTL (30s
by default) and new jobs route elsewhere. In-flight jobs keep polling: a
composing job leased as modal:<id>/beam:<id> is completed by the reaper
poll backstop, so draining needs no redeploy. A submit that keeps failing
accumulates attempts like any runner and flips to failed at the queue’s
terminal threshold, unless the operator configures a fallback chain entry
(e.g. JOBS_RUNNER_FALLBACK_CHAIN=modal>runpod) to promote instead. If the
app itself is bad, also stop the platform deployment (Modal dashboard ->
stop app; Beam -> deactivate the deployment) to avoid paying idle GPU time.

Cost guardrails

• Both apps run min_containers=0 / no minimum workers: idle time costs
  nothing; you pay per job plus brief container idle windows.
• GPU classes are deliberately modest (Modal L40S, Beam L4). Changes to
  them require a review of expected monthly spend, same as the RunPod
  scaler work (BRI-359 phase 6).
• Weights are the main leak: a misconfigured modal 100 route burns budget.
  Keep enabled toggles false for any provider not under active evaluation,
  and watch the chosen-vs-configured weight-drift obs metric.
• Deploy workflows gate on a typed yes confirmation; never relax that to
  push triggers.