Home push daemon ("serverless home", BRI-437)

Home push daemon (“serverless home”, BRI-437)

Deploy-simplification step 5b. The home box joins the Runtime registry as
a push runtime: the backend’s dispatch sweep POSTs each job to the box
over the tailnet (submit(job) → call_id) instead of the box polling
/jobs/claim. This doc is the tailnet-host runbook.

History: this daemon grew out of the BRI-410 wake agent — same stdlib HTTP
server, same BRI-422 auth chain — which spawned a claim loop on
POST /wake. BRI-437 deleted the spawn machinery and made the daemon the
worker itself (brioso-worker serve is now THE home worker unit; the old
POST /wake stayed on as a deprecated no-op through rollout; BRI-439
removed it).

What it is

A tiny stdlib HTTP server (apps/worker/src/apps/worker/wake_agent.py)
that runs process_job in-process on a supervised single-flight slot:

  • POST /jobs — authorized per the auth model below.
    Body is the push contract: {slug, params, uploadUrls[{url,key}], deadlineUnix} — params is the job’s full JobResponse dict; the flat
    uploadUrls are the presigned R2 PUT URLs minted at submit, re-nested
    into the tiered shape PresignedUploader expects. Returns 202
    {call_id} on accept, 429 {detail} while the slot is busy (single
    in-flight call, mirroring generation_slot(1)), 409 on a duplicate
    in-flight slug or a passed deadline, 422 on a malformed payload.
  • GET /jobs/{call_id} — same auth. {state, call_id, slug, result?, error?} with state ∈ queued|running|done|failed|cancelled; a call_id
    the daemon never saw answers {"state": "unknown"} — the web side
    treats that as failed per sweep rules.
  • GET /health — unauthenticated (the tailnet is the boundary): {ok, runtime, agent, engine, in_flight, gpu}.

Per-call state persists as JSON under
<state_dir>/jobs/<call_id>.json, so GET /jobs/... survives a restart;
calls still queued/running at boot are marked cancelled (threads
don’t survive — the reaper’s expired-lease arm promotes the row onward).

Completion is unchanged: the call’s worker_id is
<runtime_id>:<call_id> — the same string the sweep writes into
jobs.leased_by — so heartbeat extend_lease, the /fail leased_by
guard, and the paid-PATCH gate all apply exactly as for the old claim
loop (pull-era; removed in BRI-439). Results land via the same PATCH /jobs/{slug} upserts +
presigned-PUT uploads.

Install (tailnet host)

Requires: repo checkout, bun install done, uv + worker deps installed
(uv pip install -e apps/worker + the SA3 venv, same as today), brioso doctor green, Tailscale up and reachable from the backend box.

1. Zero secrets (BRI-422)

No openssl rand step, no shared token: the daemon authorizes tailnet
peers by tailscale whois and loopback callers unconditionally, so a home
GPU joins with nothing provisioned — run brioso-worker serve and
register the runtime with just a push URL.

2. .env

# required
JOBSTORE_API_URL=https://usebrioso.com
WORKER_SHARED_SECRET=<same as the always-on daemon>
# optional
HOME_WAKE_TOKEN=<openssl rand -hex 32>  # only for non-tailnet remote callers
BRIOSO_WAKE_ALLOW_NODES=vps-1          # restrict which tailnet nodes may submit
BRIOSO_WORKER_RUNTIME=home            # runtime id; also the leased_by prefix
BRIOSO_WAKE_PORT=7311                 # must match the runtime row's push URL
BRIOSO_WAKE_STATE_DIR=/path/to/state  # default <repo>/tmp/wake-agent
BRIOSO_PUSH_DRAIN_SECONDS=10          # SIGTERM grace for the in-flight call

Auth (BRI-422)

POST /jobs and GET /jobs/{call_id}
authorize when ANY of these holds — first match wins and is reported as
authorized_by:

  1. Bearer — HOME_WAKE_TOKEN is configured AND the request’s
    Authorization: Bearer <token> matches (constant-time compare).
    Back-compat: provisioned deployments keep working unchanged.
  2. Loopback — the caller’s address is a loopback address. Covers local
    ops AND the local-E2E recipe below — no token, no tailscale.
  3. Tailnet peer — tailscale whois <remote-ip> resolves the caller to
    a tailnet node. With BRIOSO_WAKE_ALLOW_NODES (comma-separated node
    names, e.g. the VPS node) the resolved name must be listed; unset = any
    resolved tailnet node. whois failure / timeout / missing CLI fails
    closed → the rule does NOT authorize.

Otherwise 401 {"error": "unauthorized"}.

The bearer remains the right choice for a remote caller outside the
tailnet. When the runtime row sets push.tokenEnv, the backend sends
Authorization: Bearer <env value> only when the var resolves non-empty —
a declared-but-empty var sends no header (never a 401 from dispatch’s
side).

Register the runtime

Admin console → GPU Gateway → home runtime → set the push URL
(push.url), leave Token env name blank unless calling from off the
tailnet:

- id: home
  kind: home
  push:
    url: "http://<home-gpu-host>.<tailnet>:7311/jobs"
    # tokenEnv omitted — tailnet peer auth

A push-configured runtime is dispatched by submit() — the sweep’s
push arm POSTs the job, writes leased_by="home:<callId>", and a
retryable submit failure (429 busy, 409 stale, 5xx, unreachable) walks
the fallbackChain in order (modal-espetro on beta).

Local home GPU (no token, no tailscale)

Run the daemon on the same host as apps/web and register the runtime
with push.url=http://127.0.0.1:7311/jobs — the loopback rule authorizes
the sweep’s submits. This enables a real home-kind E2E generation
(audiocpp/small model) from dev:fake: the sweep POSTs the job to the
daemon, the daemon generates and PATCHes the result back.

# 1. Stack (the fake runner is the `local` runtime; the real daemon is a
#    separate `home`-kind runtime so they never contest)
mise run dev:fake -- --no-runner

# 2. Register the home runtime + fallback chain via the admin API
#    (DB layer wins over GPU_GATEWAY_CONFIG_FILE — source: "db").
source .brioso/dev-secrets.env
curl -X PUT localhost:3000/internal/admin/gpu-gateway/runtime/home-local \
  -H "Authorization: Bearer $WORKER_SHARED_SECRET" \
  -H 'content-type: application/json' -d '{
    "id":"home-local","kind":"home","enabled":true,
    "push":{"url":"http://127.0.0.1:7311/jobs"},
    "idleExitSeconds":60,"claimTimeoutSeconds":90}'
curl -X PUT localhost:3000/internal/admin/gpu-gateway/chain \
  -H "Authorization: Bearer $WORKER_SHARED_SECRET" \
  -H 'content-type: application/json' -d '{"chain":["home-local"]}'
# GET /internal/admin/gpu-gateway → source "db", chain head home-local.
# With the DB config live, `local` is unregistered, so new jobs land on
# the chain head at insert and the sweep's push arm submits to it
# (audit: submit:home-local → dispatcher lease leased_by
# "home-local:<call_id>" → new→composing→paid).

# 3. Worker env — apps/worker/.env (gitignored). Source dev:fake's
#    generated secrets instead of copying the literal value:
#      source "$(dirname "${BASH_SOURCE[0]}")/../../.brioso/dev-secrets.env"
#      JOBSTORE_API_URL=http://localhost:3000
#      BRIOSO_WORKER_RUNTIME=home-local
#      BRIOSO_WORKER_MODEL=audiocpp
#      AUDIOCPP_BACKEND=cpu
#      AUDIOCPP_BIN=<abs path to audiocpp_cli>
#      AUDIOCPP_MODEL_PATH=<abs path to a stable_audio .gguf>
#      AUDIOCPP_TIMEOUT_S=2400   # CPU is far slower than the 900s GPU default

# 4. audiocpp on CPU-only Linux. The pinned v0.7.3 release's Ubuntu
#    prebuilt needs glibc ≥ 2.38 (24.04) — on 22.04 build it instead:
#      git clone --depth 1 --branch v0.7.3 https://github.com/0xShug0/audio.cpp
#      cd audio.cpp && scripts/build_linux.sh --backend cpu \
#        --model-set custom --models stable_audio --target audiocpp_cli
#    (needs cmake ≥ 3.20 + gcc; binary lands in build/linux-cpu-release/bin/)
#    Lightest real model:
#    Stable-Audio-3-Small-Music-GGUF/stable-audio-3-small-music-q8_0.gguf
#    (~1.7 GB) from HF audio-cpp/audio.cpp-gguf. 15 s stereo 44.1 kHz
#    generates in ~17 s on 8 CPU cores.

# 5. Push daemon — loopback auth, no token
cd apps/worker && uv run brioso-worker serve --port 7311

# 6. Submit (any persona; POST /jobs defaults runner → chain head → swept)
curl -b /tmp/cookies -X POST localhost:3000/jobs -H 'content-type: application/json' \
  -d '{"description":"lo-fi hip hop beat","email":"creator@brioso.test",
       "duration_seconds":15,"track_count":1}'

Expected daemon log for a landed job:
push-daemon: "POST /jobs" 202 → push_daemon.job_accepted (obs) →
PATCH /jobs/<slug> (composing → paid) → push_daemon.call_finished.

pm2 start deploy/pm2/ecosystem.wake-agent.config.cjs
pm2 save && pm2 startup

3-alt. Run — systemd (Linux)

Edit deploy/pm2/brioso-wake-agent.service (User/WorkingDirectory/
EnvironmentFile), then:

sudo cp deploy/pm2/brioso-wake-agent.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now brioso-wake-agent

Foreground/debug

brioso worker serve --port 7311         # host CLI wrapper
brioso-api worker serve --port 7311     # same, via @brioso/cli
uv run brioso-worker serve              # canonical Typer entrypoint

All three land on run-wake-agent.sh (sources .env; HOME_WAKE_TOKEN
optional — without it the daemon warns on stderr and authorizes loopback +
tailnet peers only). brioso-api worker serve needs BRIOSO_REPO_ROOT
when invoked outside a checkout.

Verify

# local (no token needed for /health)
curl -s http://127.0.0.1:7311/health | jq .

# status of a call — unknown is the never-seen/restart answer
curl -s http://127.0.0.1:7311/jobs/<call_id> | jq .

# remote callers still use the bearer when HOME_WAKE_TOKEN is set
curl -s http://<host>.<tailnet>:7311/jobs/<call_id> \
  -H "Authorization: Bearer $HOME_WAKE_TOKEN" | jq .

# over the tailnet, from the backend host
curl -s http://<host>.<tailnet>:7311/health

Expected /health shape:

{
  "ok": true, "runtime": "home",
  "agent": {"pid": 1, "uptime_seconds": 12},
  "engine": {"state": "ready", "load_error": null},
  "in_flight": null,
  "gpu": {"detected": true, "accelerator": "cuda", "name": "RTX 4090",
          "memory_used_mb": 0, "memory_total_mb": 24564}
}

Teardown semantics

  • SIGTERM/SIGINT (pm2 stop/restart, systemctl stop, Ctrl-C): the
    in-flight call’s record is marked cancelled and the runner thread gets
    BRIOSO_PUSH_DRAIN_SECONDS (default 10) to land in-flight PATCHes before
    the process exits. pm2’s kill_timeout (10 s) bounds it; the job row’s
    lease expires and the reaper promotes it onward.
  • Reboot/power loss: queued/running records flip to cancelled on the
    next boot (JobBook reopen); GET /jobs/<id> then reports cancelled.

Troubleshooting

  • 401 on /jobs — caller matched no rule: wrong/absent bearer AND
    non-loopback AND tailscale whois <ip> didn’t resolve (or the node isn’t
    in BRIOSO_WAKE_ALLOW_NODES). Check tailscale whois <caller-ip> from
    the daemon host.
  • 429 on every submit — a call is genuinely in flight
    (GET /health → in_flight), or a wedged thread holds the slot; restart
    the daemon to release it.
  • 409 on submit — duplicate in-flight slug (the web re-dispatched a job
    already running — same call_id is returned) or the payload’s
    deadlineUnix already passed.
  • 422 on submit — payload doesn’t parse as the push contract; the
    detail names the bad field (usually uploadUrls key convention drift —
    keys must be track-<idx>.<tier>).
  • /health unreachable over tailnet — tailscale status on the host, and
    confirm the port isn’t bound to localhost only (BRIOSO_WAKE_HOST).
  • Port already in use — another daemon running? pgrep -af brioso-worker.
Previous Home GPU inference server setup (BRI-249) Next Infisical environment & secrets guide (apps/web)