GPU gateway operations (post-BRI-439 push-only)

GPU gateway operations (post-BRI-439 push-only)

Supersedes the BRI-420 canary/weight-ramp checklist — weights, canary
rules, and ramp stages no longer exist. Dispatch order IS the policy:
fallbackChain in the resolved gateway config, walked top to bottom by
dispatchViaPushChain. This doc is the operator’s reference for how a job
moves and where to look when it doesn’t.

How a job dispatches

1. POST /jobs creates the row (state: new, runner: its assigned
   runtime id or the default) and fires dispatchJob — the event-driven
   path. The 60 s dispatch-sweep loop picks up anything left new.
2. dispatchViaPushChain walks dispatchCandidates(job.runner) — the
   job’s runner first, then fallbackChain, dedup’d — skipping runtimes
   that are disabled, model-mismatched, missing an impl, unreachable,
   or over_budget.
3. For each candidate: promoteRunner the row, take a 600 s dispatcher
   lease (claimForRunner(id, "<slug>-dispatcher")), impl.submit. A
   retryable failure (429 busy, unreachable, timeout) retries once in
   place, then walks to the next candidate; a non-retryable failure
   releases the queue row and runs failOrFallback.
4. Success stamps leased_by = "<runtimeId>:<callId>". Then:
   • kind: home/local — the push daemon (BRI-437) drives state
     transitions itself via the worker-secret routes (PATCH /jobs/:slug,
     /lease, /fail, /claimed). The reaper polls GET push.url/<callId> as a stale-call backstop.
   • kind: modal — the spawned generate FunctionCall writes to R2 via
     the payload’s presigned URLs and returns the result to the
     status(callId) poll; nothing on Modal calls the backend. The row
     goes composing on submit and the reaper’s poll is the only
     completion path.
5. failOrFallback promotes the row to the next fallbackChain entry on
   exhausted attempts; after the last entry the job fails terminally.

Config changes (chain order, enable/disable, budgets) go through the admin
console (GPU Gateway) or the import/export routes — import is
replace-semantics, so export first.

Cost guardrails (always on)

• budget.dailyCents = 1000 per Modal runtime ($10/day soft cap; the
  runtime reads over_budget at 2x and is skipped in the chain walk).
• maxConcurrency: 2 on beta.
• idleExitSeconds: 60.
• A Modal dashboard budget alert at $10.

Stop conditions — disable + drain immediately

• The same slug submitted more than 3 times.
• Stuck count > 3.
• A 401/403 on submit (proxy token wrong/expired).

Drain verb: modal app stop brioso-worker -e <env> (proxy-auth) or
disabling the runtime in the admin console — a disabled runtime is skipped
in the walk and modal:app-stop releases in-flight calls on next poll.
Rollback layers: disable in the UI → modal app stop → restore the
exported gateway file via import.

Observability — where a stuck job is stuck

The admin console’s runtime detail shows hops 2–4 live per runtime: queue
depth, last submit, health state, spend ledger.