Local Stack — Home Worker + Garage S3

Local Stack — Home Worker + Garage S3

Run the full Brioso stack locally with the home GPU worker and a local Garage S3
bucket, without touching real Cloudflare R2.

This covers BRI-199 (runner routing + Garage S3). Garage is bootstrapped via
Docker directly.

Overview

apps/web (TanStack Start)
  ├── POST /jobs  →  jobs.runner = "home" (when JOBS_DEFAULT_RUNNER=home)
  └── dispatch sweep  →  POST to the daemon's /jobs (push; see
                          home-worker-wake.md)
        │
        ├── SA3 generation (MLX on Apple Silicon, diffusers on CUDA)
        └── PUT to Garage (presigned PUT URLs arrive inside the push
            payload — no worker-side R2 credentials)

Jobs default to the gateway chain head in production. JOBS_DEFAULT_RUNNER=home
routes every new job to the home worker.

Prerequisites

mise trust && bun install   # one-time per worktree
uv pip install -e apps/worker
uv pip install -e packages/stable-audio-3 packages/musicgen-adapters packages/inference-core
# ffmpeg must be on PATH

One-time: Garage S3

Garage on Docker

Start Garage and initialise a bucket and key:

docker run -d --name garage \
  -p 3900:3900 -p 3903:3903 \
  -v garage-data:/data \
  garagehq/garage:v1

# Wait for startup, then initialise a layout (one-time)
docker exec garage garage layout assign /data garage -r 1

# Create the jobs bucket
docker exec garage garage bucket create brioso-jobs

# Create an API key for the web app
docker exec garage garage key create local-dev

# Allow read+write on the bucket
# (replace <key-name> with the name printed by the previous command)
docker exec garage garage bucket allow brioso-jobs \
  --key <key-name> --read --write

The key create command prints an access key ID and secret. Copy them for the
config step below.

Garage on bare-metal or with nono sandbox

You need to have it pre-installed.

Configure apps/web

Infisical’s dev environment already provides Garage-pointed R2_* values —
a team-wide, 2-dev convenience setup. Each developer still needs their own
local Garage running, since the endpoint and key are machine-local; point
Infisical dev’s R2_ACCESS_KEY_ID/R2_SECRET_ACCESS_KEY at your own
garage key create output if they differ. Because of this, apps/web/.env
normally doesn’t need its own R2_* block to run bun run dev:infisical.

Set the following in apps/web/.env (needed either way for runner routing
and worker auth; needed for the full R2_* block only when running
bun run dev, with no Infisical):

# ── Runner routing ───────────────────────────────────────────────
# Every new job inserts with this runner when the caller doesn't specify one.
# "home" → the dispatch sweep pushes jobs to the daemon's POST /jobs.
# Unset → home-capacity bias, then the gateway chain head.
JOBS_DEFAULT_RUNNER=home

# Optional: allow POST /jobs callers to pin a specific runner.
# Default-deny; only intended for local dev. Unset to disable.
# JOBS_ALLOW_RUNNER_OVERRIDE=true

# ── Garage S3 (local) ───────────────────────────────────────────
# Only needed for `bun run dev` (no Infisical) — dev:infisical already gets
# these from Infisical's dev environment (see above).
R2_ENDPOINT=http://localhost:3900
R2_REGION=garage
R2_FORCE_PATH_STYLE=true
R2_BUCKET=brioso-jobs
R2_ACCESS_KEY_ID=<from `garage key create`>
R2_SECRET_ACCESS_KEY=<from `garage key create`>

# ── Worker auth ─────────────────────────────────────────────────
# Must match WORKER_SHARED_SECRET set in the worker's environment.
WORKER_SHARED_SECRET=local-dev-secret

Restart the dev server after changing env vars.

Configure apps/worker

The worker reads plain env vars, not Infisical. Set in its .env or shell:

# Auth — must match WORKER_SHARED_SECRET above
WORKER_SHARED_SECRET=local-dev-secret

# Backend to poll
JOBSTORE_API_URL=http://localhost:3000

# Optional: MLX vs diffusers (auto-detected on macOS vs other platforms)
# SA3_ENGINE=mlx

# Optional: HuggingFace token for first-run SA3 weight download
# HF_TOKEN=

# Optional: refine service (no-op if unset)
# BRIOSO_REFINE_URL=
# BRIOSO_REFINE_API_KEY=

# Optional: observability (no-op if unset)
# PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006
# POSTHOG_API_KEY=

Run the stack (one command)

For the common case (default SA3/MLX model, no flag overrides), skip the
manual two-terminal dance below:

mise run dev:home-worker

This applies pending packages/db migrations, then starts apps/web
(dev:infisical, with JOBS_DEFAULT_RUNNER=home) and apps/worker together,
resolving WORKER_SHARED_SECRET from your shell env
or, failing that, Infisical’s dev project (infisical secrets get WORKER_SHARED_SECRET --env=dev), and reading vite’s actual bound port from
its own stdout (vite auto-increments past :3000 when another worktree
already has a dev server running). It exits immediately if the secret isn’t
set either way — add it to Infisical’s dev project or export
WORKER_SHARED_SECRET yourself first. Logs from both interleave in one
terminal; Ctrl+C stops both. You still need Garage running first (docker start garage, or mise run s3:up). See scripts/dev-home-worker.ts for
exactly what it runs.

If you need to override the model, pin a fixed secret across restarts, or
run Garage/web/worker in separate terminals for closer inspection, use the
manual flow below instead.

Run the stack

# Terminal 1 — Garage S3
docker start garage

# Terminal 1 — web app (from repo root)
JOBS_DEFAULT_RUNNER=home bun run dev:infisical

# Terminal 2 — home GPU worker
uv run brioso-worker worker \
  --model sa3 \
  --worker-id "$(hostname)" \
  --store api

Verifying it works

  1. Open http://localhost:3000/dev/login?persona=creator&redirect=/studio to authenticate.
    (agency is a kept alias for creator; both mint the same session —
    canonical persona for studio QA is creator, see
    .agents/docs/dev-seed-states.md.)
  2. Create a job through the studio UI.
  3. Confirm the jobs.runner column is "home":
 SELECT slug, runner, state FROM jobs ORDER BY created_at DESC LIMIT 5;
  1. Watch the worker terminal — it should log a claim and begin generation.
    (On the push daemon the equivalent log is push_daemon.job_accepted
    plus a POST /jobs 202 — the sweep pushes instead of the box polling.)
  2. After the worker finishes, state should be "paid" and Garage should
    contain preview/master objects under the slug key namespace:
 docker exec garage garage object list brioso-jobs

Running a single job via script (no GPU needed)

To land a runner="home" job without a GPU, for fast funnel testing:

RUNNER=home JOBS_ALLOW_RUNNER_OVERRIDE=true bun scripts/fake-complete.ts

This hits real HTTP endpoints so it exercises auth, schema validation,
idempotency, and completeGeneration — but skips the GPU entirely (track
keys point at objects that don’t exist, so audio players render empty).

Switching back to RunPod

Comment out or unset JOBS_DEFAULT_RUNNER. New jobs insert with
runner="runpod" and the dispatch sweep handles them. Restart the dev server.

Troubleshooting

Worker never claims jobs

  • Is jobs.runner = 'home' in the DB? Check with SELECT runner FROM jobs WHERE slug = '[[ORCA_RICH_MD:907aa2a23f6294e8b43528c898725d33:inline-html:%3Cslug%3E]]'.
  • Is WORKER_SHARED_SECRET the same in both web app and worker env?
  • Is the worker pointing at the right JOBSTORE_API_URL?

Garage uploads return 403

  • Did you run garage bucket allow for the key you created?
  • Is R2_FORCE_PATH_STYLE=true set in the web app env?
  • Is R2_ENDPOINT=http://localhost:3900 (not the real R2 endpoint)?

MLX weights download / venv bootstrap on first run

  • The worker auto-bootstraps the Stable Audio 3 MLX repo on first use:
    ensure_mlx_repo_cloned() clones the repo (if missing) and runs its
    optimized/mlx/install.sh -y (if .venv/bin/python is missing). This is
    non-interactive and can take a few minutes (dependency install + optional uv
    install); the first generation job may appear stalled while it runs.
  • Set HF_TOKEN=[[ORCA_RICH_MD:907aa2a23f6294e8b43528c898725d33:inline-html:%3Cyour%20HuggingFace%20token%3E]] in the worker’s env — sa3_mlx.py lazy-fetches the ~4 GB model weights from HuggingFace on first use after the venv exists.
  • If bootstrap fails (network, disk, permissions), the job fails and retries on the next lease cycle (5 min); fix the underlying error rather than re-running install.sh by hand.
Previous Local stack — env resolution and measured budgets Next Mac worker Cloudflare Tunnel