Spaces happy path: perf percentiles (phase span map)

Spaces happy path: perf percentiles (phase span map)

Status: PR 1 (feat/bri-337/perf-span-coverage, merged to beta) adds the
span/event coverage. This doc (PR 2) holds the HogQL recipes and the insight
checklist. All 10 insights are created in PostHog (tag
performance/spaces happy path) with short_ids recorded below; they return
empty until PR 1’s span coverage has ~24h of data on the events listed.
Linear: BRI-337.

Phase -> span/event map

Every phase emits a Phoenix span (when the work happens inline) and a PostHog
event ${name}.completed with duration_ms, both via the obs facade
(withSpan / with_span / event). Never raw OTel/PostHog calls. When comparing environments in an insight,
filter PostHog events on deployment.environment (beta vs prod; prod is
the default when DEPLOYMENT_ENV is unset).

Property conventions

All phase spans/events carry:

• runner: "home" or "runpod" (which runner executed the job)
• slug: job slug (maps to Phoenix span session.id)
• trace_id: present in PostHog event props when ctx.traceId is set (via
  obs-js contextProperties); enables Phoenix deep links
  phoenix://projects/brioso/traces/<trace_id>
• track_count: number of tracks in the job
• duration_s: requested audio duration, where meaningful
• duration_ms: always present on ${name}.completed PostHog events
  (emitted by withSpan/with_span on success)

Runner phase spans additionally carry audio.duration_s and track.count as
span attributes.

PostHog insights (PR 2)

Project: https://eu.posthog.com/project/219634 (insight URLs below follow the
/insights/<short_id> pattern used by the existing KPI docs). There is no
in-repo script that creates insights via the PostHog API today; the closest
tooling is posthog-cli (see .agents/notes/2026-08-14-posthog-cli.md), used
for manual UI/CLI work with --host https://eu.posthog.com. Insights are
therefore created in the PostHog UI (HogQL mode) and only their short_ids are
recorded here.

Recipe A: per-phase percentiles

One insight per phase event. HogQL query (replace the event name per phase):

SELECT
  properties.runner AS runner,
  quantiles(0.5, 0.75, 0.9, 0.95)(properties.duration_ms) AS p50_p75_p90_p95_ms
FROM events
WHERE event = 'runner.generate.completed'
  AND timestamp > now() - interval 24 hour
GROUP BY runner
ORDER BY runner

Phase events to cover: space.create.completed, compose.dispatch.completed,
queue.wait.completed, runner.generate.completed,
runner.encode.completed (runpod only), runner.upload.completed (runpod
only), runner.encode_and_upload.completed (home only),
delivery.url_signed.completed.

Extra breakdowns (swap properties.runner for, or add as extra groups):

• properties.track_count: does parallelism hurt encode/upload?
• properties.duration_s: does requested audio length drive generate time?

PostHog UI breakdowns cap at 25 values per breakdown in trends; using HogQL
GROUP BY in a SQL insight avoids the limit entirely, so prefer the SQL
insight type for these.

Recipe B: end-to-end runner time

The runner root span duration is the E2E runner time (generate + encode +
upload inside one process). runpod.handler.completed covers the RunPod path,
worker.process_job.completed the home path; combine both in one insight:

SELECT
  properties.runner AS runner,
  quantiles(0.5, 0.75, 0.9, 0.95)(properties.duration_ms) AS e2e_runner_ms
FROM events
WHERE event IN ('runpod.handler.completed', 'worker.process_job.completed')
  AND timestamp > now() - interval 24 hour
GROUP BY runner
ORDER BY runner

Queue wait is the missing upstream leg (RunPod path only, per
completion.ts:286). Read it alongside the E2E number:

SELECT
  quantiles(0.5, 0.75, 0.9, 0.95)(properties.duration_ms) AS queue_wait_ms
FROM events
WHERE event = 'queue.wait.completed'
  AND timestamp > now() - interval 24 hour

True user-perceived E2E (space create through signed URL) would need a
cross-event sum per trace; the two insights above are the practical proxy:
space.create + compose.dispatch + queue.wait + runner root + delivery.

Recipe C: slow contributors top 10

One insight per slow phase (start with runner.generate.completed and
queue.wait.completed). SQL insight type:

SELECT
  distinct_id,
  properties.slug AS slug,
  properties.trace_id AS trace_id,
  properties.duration_ms AS duration_ms,
  timestamp
FROM events
WHERE event = 'runner.generate.completed'
  AND timestamp > now() - interval 24 hour
ORDER BY duration_ms DESC
LIMIT 10

Deep-link the worst offenders into Phoenix:
phoenix://projects/brioso/traces/<trace_id> (paste the trace_id into the
Phoenix trace search; the URL form is the convention used in the runbooks).

Note: delivery.url_signed.completed has no first-sign marker in PR 1 (every
sign fires, see tracks.ts:254), so for that phase either accept that repeat
downloads pollute the top-10 or filter duration_ms > 1000 to surface only
meaningfully slow signs.

Insight checklist

Folder: “Performance / Spaces happy path” (create it if missing). Create each
insight with the recipe above, name it as suggested, and record the short_id
here during execution.

Reading the bottleneck

Compare each phase’s p95 against its p50: a wide gap means a tail problem
(capacity, cold start, one slow shard), a uniformly high p50 means a constant
cost (model, egress bandwidth).

• queue.wait high -> RunPod cold start / capacity (check whether p95 spikes
  cluster at job bursts; cross-reference runpod.handler cold starts)
• runner.generate high -> model/engine (SA3 wall time; compare
  audio.duration_s vs duration_ms)
• runner.encode high -> ffmpeg/encode workers (scales with track_count)
• runner.upload high -> R2 egress (scales with track_count; on the home
  runner read runner.encode_and_upload minus runner.generate as the
  combined encode+upload proxy)
• space.create / compose.dispatch / delivery.url_signed should be
  milliseconds; if not, the web app or R2 presign is the problem

AC deviations (record on BRI-337)

• “Aggregation dashboard loads in < 2s”: moot. There is no in-repo dashboard
  page; the dashboard IS PostHog, so this AC collapses into PostHog’s own
  insight load performance.
• “Percentiles for 24h window within 5s”: reinterpret as PostHog insight load
  time. Verify once (after the first 24h of PR 1 data) by timing each insight
  in the checklist; if any exceed 5s, note it rather than optimizing.
• Encode/upload split exists only on the RunPod runner (handler.py:457,466).
  The home runner keeps a combined runner.encode_and_upload span
  (worker main.py:387) because encode and upload run concurrently in a shared
  thread pool inside inference_core, so insights 5 and 6 show home data only
  as the combined proxy.
• queue.wait.completed fires only on the RunPod completion path
  (completion.ts:286): the home runner records no claim latency.