Modal audio.cpp runbook (SA3.0-small, BRI-382)

Modal audio.cpp runbook (SA3.0-small, BRI-382)

Superseded by BRI-436/439: apps/modal-audiocpp is gone — the audiocpp
image + handler now run inside apps/modal-worker (brioso-worker Modal
app), push-dispatched via submit. Ops live in
gpu-gateway-beta-runbook.md; the image-assembly sections below (bake
layer, GGUF pin) still apply to apps/modal-worker/modal_app.py.

Operational guide for the brioso-audiocpp Modal app: how the image is
assembled, why the bake layer exists, how to deploy and smoke test, and
the failure modes that cost us time to find. Companion to
apps/modal-audiocpp/README.md (overview) and .agents/docs/modal-beam-runbook.md
(SA3 medium on Modal, the contrasting pattern).

  • Modal app: apps/modal-audiocpp/modal_app.py
  • Handler: apps/runpod/handler_audiocpp.py
  • Smoke: apps/modal-audiocpp/scripts/smoke.py
  • CI: .github/workflows/deploy-modal-audiocpp.yml
Artifact Tag / state
audiocpp image usebrioso/brioso-runpod-audiocpp:v9 (built by .github/workflows/build-runpod.yml; there are no upstream Linux release prebuilts for audio.cpp, so images come only from our Docker Hub builds)
Baked GGUF stable-audio-3-medium-q8_0.gguf (3,582,595,584 bytes, from audio-cpp/audio.cpp-gguf)

Lockstep rule: bump AUDIOCPP_IMAGE_TAG only together with a new image
build, and set AUDIOCPP_FORCE_BUILD=1 for exactly one deploy after each
bump (see Gotchas).

modal_app.py builds the image as an ordered chain:

  1. modal.Image.from_registry(f"{IMAGE_NAME}:{IMAGE_TAG}"), driven by env
    AUDIOCPP_IMAGE / AUDIOCPP_IMAGE_TAG (default :v9), with secret
    brioso-registry for the private Docker Hub pull.
  2. .entrypoint([]) trivial override (keeps the registry layer cacheable
    while letting Modal control container startup).
  3. .run_function(_bake_medium_into_image) as the LAST layer.

Layer ordering matters for cache hits: a tag bump invalidates the
registry pull, but the entrypoint layer still resolves from cache, and
only the GGUF bake layer rebuilds. Ordering the bake last is what makes
bumps cheap.

The bake function downloads
audio-cpp/audio.cpp-gguf restricted to *q8_0*.gguf, then
rglob("*.gguf") + shutil.copyfile (which follows the HF symlink)
into /app/audiocpp-model/<basename>.gguf. Use rglob, not a
single-depth glob: the HF snapshot layout nests as
snapshots/<hash>/<subdir>/<file>.gguf, so snapshots/*/*.gguf finds
nothing.

Functions generate and load_model (warmup) both run
gpu="T4", timeout=1200, with AUDIOCPP_MODEL_PATH pointed at the
baked /app/audiocpp-model/stable-audio-3-medium-q8_0.gguf.

generate carries scaledown_window=300: after a burst the container
stays warm ~5 min, so consecutive jobs within that window reuse it.
Scale-to-zero still applies afterward. min_containers is the
documented-but-not-default lever (a warm T4 is ~$0.59/hr ≈
$425/mo/container); only turn it on if WS-C benchmarks show cold starts
dominate.

  1. Pin the image: AUDIOCPP_IMAGE_TAG in modal_app.py (default :v9).
    Keep lockstep with the built image (rule above).

  2. If the tag changed, export AUDIOCPP_FORCE_BUILD=1 for this deploy
    only (see Gotchas / Cache policy).

  3. Bake layer: confirm the glob pattern matches only the q8_0 file you
    intend (no f16 strays).

  4. Functions: gpu="T4", timeout=1200, AUDIOCPP_MODEL_PATH correct.

  5. Secrets: brioso-audiocpp (jobstore URL + shared secret) attached;
    brioso-registry present for the private pull.

  6. Deploy:

    uv tool run --from "modal==1.3.5" modal deploy apps/modal-audiocpp/modal_app.py

  7. Smoke (below). If you set AUDIOCPP_FORCE_BUILD=1 in step 2, unset it
    for the deploy after: the next deploy reuses Modal’s cached image.

CI deploys the same app via MODAL_TOKEN_ID / MODAL_TOKEN_SECRET repo
secrets in .github/workflows/deploy-modal-audiocpp.yml.

The modal==1.3.5 exact pin (D11) exists because the SA3 app
(apps/modal-sa3/modal_app.py) needs modal 1.3.x (>=1.4 dropped
modal.gpu). This app is version-tolerant (string gpu="T4"), but the
pin matches.

  • Bake into image via Image.run_function: weights ride with the image,
    cold containers pay zero download cost.
  • AUDIOCPP_MODEL_PATH short-circuit in
    handler_audiocpp._resolve_model_path: when set, the handler skips
    the HF cache entirely and uses the baked file directly.
  • Handler _materialize_model: copies HF-cache symlinks to a real
    .gguf file. Required on the RunPod / home worker paths where weights
    come from the HF cache instead of a baked image.

resource_bundle.cpp:36 calls std::filesystem::weakly_canonical(path)
BEFORE tensor_source.cpp:1494-1508 dispatches on path.extension().
The HF snapshot symlink resolves to the extensionless blobs/<sha> path,
so audiocpp_cli throws unsupported tensor source format: .../blobs/<sha>.
Upstream: 0xShug0/audio.cpp#274, fixed on main, unreleased as of v0.7.3.
Workarounds: bake with a copy that dereferences symlinks (Modal, via
shutil.copyfile), or _materialize_model in the handler (RunPod / home).

The Xet backend reads it at import time of huggingface_hub. The image
sets it as an env var, so imports inside the bake function and handlers
see it. Without it, blobs can land as 1 KB LFS pointer files or the
download hangs.

Re-pushing a Docker Hub tag under the same name serves stale cached
layers, because from_registry caches on the definition string, not the
resolved digest. That is why force build exists: AUDIOCPP_FORCE_BUILD=1
makes from_registry pass force_build=True for that deploy. Set it only
for the deploy immediately after a tag bump, deploy, then deploy once
more without it — Modal reuses the cached image. There is no always-on
rebuild left in the code (the earlier temporary force_build=True
leftover is gone).

Covered above: snapshots/<hash>/<subdir>/<file>.gguf needs rglob or
snapshots/**/*.gguf.

apps/modal-audiocpp/scripts/smoke.py has three modes:

  • preload: calls load_model only. No GPU bill beyond the cold start.
  • local: uploads go to synthetic 127.0.0.1:1 stubs, so generation
    runs end to end and the upload step fails as expected. No jobstore
    traffic.
  • generate: full run with real upload; needs a MODAL_SMOKE_SLUG
    present in the jobstore.

Prereqs: MODAL_TOKEN_ID + MODAL_TOKEN_SECRET in the environment, and
the brioso-audiocpp secret created in Modal.

At the end of every real job, Modal’s generate wrapper POSTs the terminal
result to {JOBSTORE_API_URL}/jobs/{slug}/complete?token=<WORKER_SHARED_SECRET>
(best-effort; a missed webhook converges via the reaper). The web app then
bridges the confirmed transition to a Hatchet job:completed:<slug> event,
which wakes the dispatch workflow’s completion waiter, so paid transitions no
longer wait out the reaper window. adhoc- prefixed slugs skip the webhook.

Spans: the Modal wrapper emits modal.boot (cold first call only,
cold_start: true) and modal.generate (cold_start, slug,
modal.container_id) on the job’s existing traceparent. Watch these in
Phoenix to read cold vs warm share before deciding on min_containers.

A 10 s / 4-step t2t job on T4 is roughly $0.005 at $0.59/hr; a full
local smoke run is around $0.01. preload costs only the cold start.

Repoy to the previous image tag: revert AUDIOCPP_IMAGE_TAG, deploy once
with AUDIOCPP_FORCE_BUILD=1 (the old tag’s cached layers may be
evicted or stale), then deploy without it. No DB or schema coupling
exists on the Modal side, so rollback is purely the app deploy. If Modal
itself is the problem, the RunPod endpoints remain the falling-back path
per packages/jobs/AGENTS.md.

  • Revert force_build=True Done: force build is now opt-in via
    AUDIOCPP_FORCE_BUILD=1; the code no longer always rebuilds.
  • No synthetic-GGUF test fixture: a probe against audiocpp_cli v0.7.3
    showed it validates model resources deeply (requires a real
    model_config.json with keys like sample_size, T5 tokenizer
    sidecar files, and real tensor data), so a few-KB stub cannot get far
    enough past arg parsing to be useful in tests. The backend-override
    tests instead assert on argv construction.

apps/modal-sa3/README.md uses the registry-image pattern with no bake
because PyTorch tolerates HF cache symlinks. audio.cpp cannot (first
gotcha above), which is the entire reason this app carries a bake layer.

apps/modal-audiocpp/scripts/bench.py measures wall seconds per job shape
(smoke 10s/4-step, medium 30s/6-step, worst 180s/8-step) against
brioso-audiocpp.generate, for cold (first call after boot) and warm
(median of 3 consecutive calls, calls 2-3) containers.

# dry run: matrix + T4 cost estimate, writes nothing
uv tool run --from "modal==1.3.5" python apps/modal-audiocpp/scripts/bench.py --dry-run

# real run (needs MODAL_TOKEN_ID/SECRET; bills real T4 seconds)
uv tool run --from "modal==1.3.5" python apps/modal-audiocpp/scripts/bench.py

Reports land in .agents/reports/audiocpp-modal-bench-<iso-ts>.json
(same structure as the RunPod bench: meta + results), with a printed
summary table showing cold-start share per shape:
(cold_wall - warm_wall) / cold_wall.

For a clean cold reading, run against a fresh deploy or after
scaledown_window (300s) has expired, so no leftover container skews the
cold number. The cold/warm numbers decide min_containers: a large
cold-start share justifies keeping one T4 warm (~$425/mo); a small share
means scale-to-zero is fine.

Previous Modal + Beam runbook (GPU gateway v2, BRI-359) Next Native audio.cpp home workers