Arize Phoenix Runbook

Arize Phoenix Runbook

Distributed trace backend for the e2e generation pipeline (apps/web →
apps/worker/apps/runpod → back), exposed to AI agents via the official
Phoenix MCP server.

Why this exists

The pipeline that generates a track is distributed across three processes and
a job’s traceId is already stashed on claim for cross-service correlation
(packages/db/src/schema.ts), but until this migration there was nowhere to
actually look at a correlated trace: prod was PostHog-only, and the previous
deep-span sink (otelite) was local-dev-only and never ran on the VPS. Phoenix
replaces otelite as the OTLP/OpenInference-native trace backend for both
local dev and prod, so a failing e2e run shows exactly which hop broke.
PostHog is unchanged — product analytics and prod error/lifecycle reporting
still go through it; Phoenix is additive for distributed/LLM tracing.

This is a deliberate reversal of the earlier “no OTLP in prod” design (see
.agents/plans/2026-07-21-pr1-observability-foundation.md) — OTLP now has a
real prod backend.

Architecture

apps/web ──┐
apps/worker ├──▶ OTLP/HTTP :6006/v1/traces ──▶ Arize Phoenix (podman quadlet)
apps/runpod ┘                                        │
                                                       │ SQLite volume
                                                       ▼
                                              /mnt/data (brioso-phoenix-data)

Agent (Claude Code, Cursor, ...)
    │  HTTP (Streamable HTTP, native to Phoenix)
    ▼
http://localhost:6006/mcp (dev) or
https://ops.usebrioso.com/mcp (prod, via Caddy — see deploy/README.md's
"Custom subdomains" section)

Phoenix serves the web UI, the OTLP-HTTP trace collector (POST /v1/traces),
the REST/GraphQL API, and (as of Phoenix 19.0.0+) a native MCP server at
/mcp — all on the same port (6006 in-container, 4317 for OTLP-gRPC —
unused here). Our pinned image tag (version-19.6.0, see
deploy/phoenix/brioso-phoenix.container) already meets that floor.

There is no separate MCP port or sidecar process. Earlier revisions of this
doc had agents spawn @arizeai/phoenix-mcp locally over stdio (npx); that
package now receives bug fixes only, per Arize — new capability lands in the
Remote MCP Server built into Phoenix itself. Use the native /mcp HTTP
endpoint instead: it’s zero-install (any MCP-capable client speaks
Streamable HTTP directly, no local process to spawn) and it’s already
reachable wherever the UI is (:8001 in prod).

VPS deployment (rootless Podman quadlet)

Podman is the first container runtime on this host — everything else here is
bare systemd units / binaries. Rootless + loginctl enable-linger web-services

• loopback-only PublishPort keeps it consistent with the existing
  non-root/tailnet-gated posture (mirrors Drizzle Gateway / MCP SQLite).

• deploy/phoenix/brioso-phoenix.container — quadlet, installed to
  ~web-services/.config/containers/systemd/, managed via
  systemctl --user as web-services (not root — quadlets under a user’s
  config dir run under that user’s own systemd instance).
• deploy/phoenix/brioso-phoenix-data.volume — named volume backing
  Phoenix’s SQLite trace store at /mnt/data in the container.
• deploy/scripts/install-phoenix.sh — installs podman + uidmap, provisions
  subuid/subgid ranges, enables lingering, installs the quadlets, seeds
  /etc/brioso/phoenix.env from the template, reloads and starts the unit.
  Idempotent — wired into deploy/scripts/setup-observability.sh.

Persistence starts as SQLite-on-volume (matches the repo’s SQLite-everywhere
posture, no new Postgres dependency). PHOENIX_SQL_DATABASE_URL (Postgres) is
the documented upgrade path if trace volume grows — add a second quadlet
container and point Phoenix at it.

Install / update

sudo /opt/brioso/repo/deploy/scripts/install-phoenix.sh

Re-run to reconcile state (e.g. after bumping the pinned image tag in
brioso-phoenix.container). To restart directly:

sudo runuser -u web-services -- env XDG_RUNTIME_DIR=/run/user/$(id -u web-services) \
  systemctl --user restart brioso-phoenix.service

Verify

# Service status (as web-services)
sudo runuser -u web-services -- env XDG_RUNTIME_DIR=/run/user/$(id -u web-services) \
  systemctl --user status brioso-phoenix.service

# Local health
curl -sI http://localhost:6006/

# Over Caddy (prod)
curl -sI https://ops.usebrioso.com/

Custom subdomain (Caddy)

Phoenix is reached in prod at https://ops.usebrioso.com via the shared Caddy
instance — see deploy/README.md’s “Custom subdomains (Caddy)” section for
the full DNS/Caddy/ACL setup (shared with Drizzle Gateway, MCP SQLite, and
apps/admin). No path-splitting needed here: the UI, OTLP collector, and
/mcp all share the same backend port (6006), so ops.usebrioso.com
reverse-proxies the whole thing.

Auth

Primary control is loopback bind (PublishPort=127.0.0.1:6006:6006) + the
Tailscale ACL on the shared Caddy IP (autogroup:member — see
deploy/README.md). Additionally,
PHOENIX_ENABLE_AUTH=true + PHOENIX_SECRET (from /etc/brioso/phoenix.env,
mode 0600, owned by web-services) are set as defense-in-depth. Generate the
secret with openssl rand -hex 32.

Project routing by deployment environment

DEPLOYMENT_ENV ("prod" | "beta", default "prod") selects which Phoenix
project an app’s traces land in: the prod Phoenix project is default, the
beta project is beta. Beta servers must have DEPLOYMENT_ENV=beta set
(beta’s env file is /opt/brioso/shared/beta.env); prod needs nothing —
absence is prod. If a beta box is missing the variable, its traces silently
land in the default (prod) project. Filter accordingly when comparing
environments.

PHOENIX_SECRET and PHOENIX_API_KEY both live in Infisical under
/infra/phoenix (prod env, same project as the app secrets) — nothing to
write on the VPS by hand. deploy-vps.sh’s sync_secrets() syncs that
folder into /etc/brioso/phoenix.env on every deploy and restarts
brioso-phoenix (as web-services, via systemctl --user) only if a value
actually changed, since env vars are read once at container start. See
deploy/README.md’s “Custom subdomains (Caddy)” section for the full
mechanics.

The /mcp endpoint itself supports two auth modes on top of that: an
interactive OAuth flow (Phoenix acts as its own authorization server, browser
login on first connect — fine for local dev, first-tab-of-a-session UX) or a
PHOENIX_API_KEY passed as an Authorization: Bearer header for headless/
scripted clients. Create the key from the UI (Settings → API Keys) once
PHOENIX_ENABLE_AUTH is on, then store it in Infisical under
/infra/phoenix — needed for the prod config below since agents there
aren’t sitting behind an interactive browser.

Tailscale ACL setup (one-time, out-of-repo)

Shared with Drizzle Gateway, MCP SQLite, and apps/admin — all four sit
behind the same Caddy instance on one IP:port (100.105.198.36:443), and
Tailscale ACLs are IP:port-scoped, not per-hostname. The ACL is
autogroup:member (this tailnet is just the two engineer accounts); see
deploy/README.md’s “Custom subdomains (Caddy)” section for the full
rationale and the exact ACL JSON. Phoenix’s own PHOENIX_ENABLE_AUTH +
PHOENIX_API_KEY (below) is the actual access-control boundary for /mcp,
same as Drizzle Gateway’s passcode is for its own UI.

Client configuration (agent-first)

Phoenix’s /mcp endpoint speaks Streamable HTTP directly — every client
config below is a remote url, no local process to spawn or version to pin.

Local dev

claude mcp add --transport http phoenix http://localhost:6006/mcp

Or, for Cursor/VS Code-style JSON configs:

{
  "mcpServers": {
    "phoenix": {
      "url": "http://localhost:6006/mcp"
    }
  }
}

First connection triggers a browser OAuth prompt (PHOENIX_ENABLE_AUTH is
prod-only, so this is a no-op locally unless you’ve turned it on there too).

Remote (prod, via Caddy)

claude mcp add --transport http phoenix-prod \
  https://ops.usebrioso.com/mcp \
  --header "Authorization: Bearer <PHOENIX_API_KEY>"

Or as JSON:

{
  "mcpServers": {
    "phoenix-prod": {
      "url": "https://ops.usebrioso.com/mcp",
      "headers": {
        "Authorization": "Bearer <PHOENIX_API_KEY>"
      }
    }
  }
}

The Bearer header skips the interactive OAuth flow, which matters for
non-interactive/scripted agent use. The Tailscale hop happens transparently
inside the client’s own HTTPS request to the url (gated by the
autogroup:member ACL on Caddy’s shared IP, see deploy/README.md) — no
mcp-remote bridge needed, since Phoenix already speaks HTTP natively.

Works from any MCP-capable client (Claude Code, Cursor, Claude Desktop, VS
Code).

Querying a failing e2e trace

1. Get the job’s trace_id from jobs.trace_id (Drizzle Studio or MCP SQLite,
   both under db.usebrioso.com).
2. Call getSpans(project_identifier="default", trace_id=[<id>]) on the
   Phoenix MCP tools — pulls every span of that one trace in a single call.
   listProjectTraces is the complementary tool for browsing traces when you
   don’t have a trace_id up front (e.g. by time range).
3. Read each span’s parent_id/start_time/end_time to reconstruct hop
   order: POST /jobs (web) → chat/invoke_agent (ai-sdk refine) →
   dispatch.runpod → runpod.handler (or the home-worker equivalent,
   worker.process_job). The hop with the error status or longest duration
   is the one to dig into first.

Troubleshooting