Evals Fundamentals

Evals Fundamentals

Team reference for LLM and audio-generation evaluation. Distills the research
that informed the BRI evals upgrade plan so the next
session does not re-derive it. Concrete, not theoretical: every section
ends in “what we do at Brioso.”

1. Eval anatomy

Every eval has four parts. Anthropic frames them in their
develop-tests guide
and the Anthropic eval cookbook:

  1. Input: the prompt + context the system receives.
  2. Output: what the system produced.
  3. Reference / golden: the expected output, or a set of properties
    the output must satisfy.
  4. Score: a function from (output, reference) to a number or label.

Success criteria come before evals. You cannot grade what you have not
specified. Before writing a test, write the success criteria: what does
“this refined prompt is good” actually mean? Behaviors, not vibes. Examples:

  • “Prompt contains no vocal terms” (regex-checkable, easy).
  • “Refined prompt mentions the same three constraints as the user input”
    (judge-checkable, needs anchors).
  • “Generated audio scores >= 3.0 on Audiobox content-enjoyment on the
    FMA-Pop prompt subset” (metric-checkable, needs a numeric floor).

If you cannot write the criteria as a checklist, you cannot grade them.
The Anthropic eval cookbook (evaluate-with-human-review)
recommends writing criteria before touching the model.

At Brioso: every new .eval.ts file in
packages/verticals/*/src/ai-workflow/ opens with a comment block
naming the four parts and the success criteria. Without that block, the
PR is incomplete.

2. Grading hierarchy

Use the cheapest reliable grader. The order, from best to worst when
“best” means “cheapest AND most reliable”:

  1. Code / regex (always first). Schema validity, length bounds,
    forbidden-term absence, JSON shape, cost ceiling.
  2. Human (for ground truth and calibration). Slow, expensive, used
    sparingly.
  3. LLM-as-judge (for what code cannot see). Cheaper than human, but
    has failure modes that code does not.

Code graders fail loud and deterministic. Human graders are slow and
expensive but set the ceiling. LLM judges sit between them and inherit
the LLM’s quirks, so we constrain them tightly.

LLM-as-judge rules

Anthropic’s claude-as-judge guide
and OpenAI’s evals guidance
converge on the same recipe:

  • Rubric with behavioral anchors. Not “is this good?” but
    “does output contain X? does it avoid Y? is Z preserved?”
    Each anchor is a binary or 1-3 score, not a Likert “rate from 1 to 5.”
  • CoT-then-verdict structured output. Two sections in the response:
    reasoning (free-form, separate field) and verdict (the score).
    Mixing them lets a confident-but-wrong rationale contaminate the grade.
    Use JSON schema or tool-use to enforce the split.
  • Pairwise over absolute. “Is A better than B?” is much easier for a
    judge than “rate A from 1 to 5.” Anchor on the original-vs-refined
    pair, not a free-floating score.
  • Cross-family model. The judge must be a different model family than
    the model being graded. If the refine workflow uses Claude, the judge
    is not Claude. Same-family judges share the same blind spots.
  • Temperature 0. Deterministic verdicts. Sampling temperature above 0
    in a judge adds noise without adding signal.
  • TPR / TNR calibration before gating. Before a judge becomes a CI
    gate, hand-label 50 cases (positive + negative). Report
    true-positive rate, true-negative rate, and inter-rater agreement
    (Cohen’s kappa against the human labels) in the PR that introduces
    the judge. A judge with TPR < 0.8 is not a gate; it is a triage signal.

At Brioso: judge rules are encoded in a shared judge template under
packages/ai-workflow/src/evals/judge-template.ts. Every judge imports
from it, including rubric structure, temperature setting, and the
schema for reasoning + verdict. Cross-family rule means a judge is
never the same model as the task. Calibration numbers live in the PR
description, not in a wiki.

3. Nondeterminism

LLM outputs are stochastic. A single shot tells you nothing.

  • N-trial pass-rate. Run each case N times (default 5, per the plan).
    The case passes if passRate >= 0.8. Single-shot assertions hide
    variance and produce flaky CI.
  • Paired deltas, not absolute scores. When comparing model A vs
    model B on the same case set, compute per-case pass-rate deltas
    (A - B) and report the mean + standard error across cases, not the
    raw pass-rate of each model in isolation. The pairing kills
    case-difficulty confounders.
  • MDE before trusting small deltas. Minimum detectable effect at
    alpha=0.05, power=0.8, depends on case count and per-case variance.
    Compute it from the first 30 cases, then add it to every eval report.
    A 2-point pass-rate delta on a 30-case suite with high variance is
    noise, not signal. The Anthropic eval cookbook
    (statistical-analysis)
    walks through the calculation.

At Brioso: the runner reports (meanDelta, stdError, mde) per
comparison. Sub-MDE deltas are reported as “no signal detected,”
not as “no improvement.” A red CI on a sub-MDE delta is a flake,
not a regression.

4. Dataset hygiene

The dataset is the most important artifact in any eval system. It is
also the easiest to quietly corrupt.

  • Provenance tags. Every case carries a provenance field:
    {source: "prod" | "synthetic", prodDate?, ticketRef?, authorId}.
    Production cases get a ticket ref so a regression has a story.
    Synthetic cases get an author so we can audit who invented which
    adversarial input.
  • Production-failure promotion loop. When a user-reported failure
    hits the refine/strategy workflow, the on-call runs semantic dedup
    against the current dataset (CLAP embedding on the prompt, cosine
    threshold 0.92) and adds the case with ticketRef: "BRI-NNN". This
    is the BRI-148-style incident-cited regression rule applied to evals.
  • Anti-overfit (fresh-author rounds, quarterly). Every quarter, a
    different team member authors 5-10 fresh cases from real usage
    without seeing the current dataset. If their cases have a
    pass-rate-statistically-different from the legacy set, the legacy
    set was overfit to its authors.
  • Provenance-weighted scoring. Weight cases by recency: a case from
    the last 30 days of production gets weight 2, older cases weight 1.
    This keeps the eval honest about drift in real usage.
  • Version pinning in CI. The dataset version is a hash in the
    runner config. CI fails if the hash in the runner does not match the
    hash in the versioned JSONL file. A runner silently loading a
    mutated dataset is the worst failure mode in this whole stack.

At Brioso: datasets live as versioned JSONL in
packages/verticals/<vertical>/src/evals/datasets/<workflow>.vN.jsonl.
The runner reads the file path, hashes it, and refuses to start if
the hash drifts mid-run.

5. Audio-generation metrics

Audio eval is harder than text eval because there is no free “did it
contain X” check. We triangulate four signals:

FAD (Frechet Audio Distance)

Use Microsoft fadtk (Python,
published with Gui et al. ICASSP 2024, arXiv:2311.01616).
FAD measures distributional distance between a candidate’s audio
embeddings and a reference set’s embeddings in a frozen embedding
space (VGGish, CLAP, or MERT). Lower is better.

Best practices from fadtk’s own README:

  • Pick a reference set that matches the task. For generative music,
    fadtk recommends FMA-Pop (4839
    tracks, the subset Gui et al. propose). Do not default to MusicCaps
    for open-ended generation.
  • Report the embedding model, reference set, sample count, and
    duration alongside every FAD number. Without all four, the number is
    not interpretable.
  • Run per-song FAD on outliers (fadtk --indiv) to listen to what
    “low” and “high” actually mean in your domain.

At Brioso: FAD runs inside the
packages/classification venv (Python, isolated). The runner calls
fadtk as a subprocess, parses scores.csv, and persists the result
to the Phoenix experiment.

CLAP-to-original-intent (weak proxy)

CLAP cosine similarity between the generated audio and the original
user prompt (not the refined prompt) measures whether the generator
preserved user intent. It is a weak proxy, with measured correlation
to human preference:

This means CLAP is fine as a smoke signal (does the model ignore the
prompt entirely?) but not as a quality gate. A 5-point swing in CLAP
score does not predict a 5-point swing in human preference.

Audiobox Aesthetics (MOS proxy)

Meta Audiobox Aesthetics
(Tjandra et al., 2025) is a no-reference per-item predictor trained on
four orthogonal axes:

  • PQ (Production Quality): clarity, fidelity, absence of artifacts.
  • PC (Production Complexity): sophistication of the production.
  • CE (Content Enjoyment): subjective appeal.
  • CU (Content Usefulness): fitness for downstream use.

Per the paper, the AES predictors show comparable or superior
correlation with human MOS while being more robust across audio types
than DNSMOS / NISQA / UTMOS. We already have
facebook/audiobox-aesthetics on HuggingFace and the
audiobox-aesthetics PyPI package, so this drops into
packages/classification next to fadtk.

Use it as the scalable MOS proxy: every audio-gen experiment run
records the four AES scores per case. Aesthetics is per-item, fast,
and reference-free, so it scales to nightly full-tier runs.

Periodic human pairwise

Before shipping any model swap, run a small human pairwise study:

= 5 ratings per (case, pair) cell, counterbalanced (every rater
hears A and B in both orders). Blind to model identity. Record:

  • Per-case winner.
  • Bradley-Terry score per model across the case set.
  • Win rate with 95% Wilson interval.

This is the ground-truth anchor for FAD + CLAP + AES triangulation.
If the three automated metrics agree with human pairwise, ship it.
If they disagree, debug the metrics, not the model.

At Brioso: the audio testbed pins prompt set, seeds, and SA3
variant. Candidate-vs-incumbent reports always show all four metrics
(FAD, CLAP-to-intent, AES-CE, AES-PQ) plus the human pairwise result
when one was run. A model change without at least one human pairwise
study goes to the bottom of the queue.

6. Phoenix architecture decision

Synthesis of the “Phoenix vs vitest” section of the upgrade plan.
We keep both, with a hard split.

What Phoenix owns (system of record)

Phoenix is the system of record for
eval data. It holds:

  • Versioned datasets (datasets concepts,
    how-to).
    Every example carries its provenance tags.
  • Experiments (how-to-experiments):
    one experiment = one run of a task + evaluators over a pinned dataset
    version. Outputs, scores, traces all persisted together.
  • UI side-by-side: prompt variants, model candidates, and
    strategy-change experiments compare on the same dataset version with
    paired deltas visible per case.
  • Annotations (annotating in the UI,
    cookbook).
    Human-in-the-loop review of judge outputs and trace spans.
  • Online evals on spans (annotations-and-evaluations tutorial):
    sample 2-5% of live production traces, run the same rubric, promote
    failures to the dataset.

The TS client reference (phoenix-client annotations)
lives in the obs facade only. No app code imports
@arizeai/phoenix-client directly, per the
scripts/check-obs-imports.ts rule.

What vitest/pytest owns (thin CI gate)

The test file’s job becomes:

  1. Load the pinned Phoenix dataset version.
  2. Run the task under test.
  3. Log the experiment (task output + evaluator scores + traces).
  4. Assert locally: passRate >= 0.8 and no regression vs the last
    green baseline.

Phoenix’s pytest integration
(@pytest.mark.phoenix, suite -> dataset, case -> example, run ->
experiment, exit code = gate) does this natively. The
Arize blog
expands the pattern. TS uses the
vitest integration
at >=6.11.1. Set PHOENIX_TEST_TRACKING=0 to run the gate fully
offline (no Phoenix server round-trip), useful for air-gapped CI and
local debugging.

So the rule is: Phoenix = datasets, experiments, scores, judges,
traces. vitest/pytest = gating. The test runner is a thin shell
around the Phoenix experiment.

PostHog boundary

PostHog stays product analytics only: funnels, retention, feature
flags, session replay. It has no eval role. Eval data does not flow
through PostHog, and eval dashboards are not PostHog dashboards.
Core AI tooling consolidates on Phoenix.

At Brioso: packages/obs-js and packages/obs-py are the only
importers of @arizeai/phoenix-client and arize-phoenix-client.
All other code routes through the obs facade.

Canonical references, grouped by topic. The
Phoenix evals skill (Arize-published,
v1.0.0) is the in-repo manual; it covers setup, evaluators,
experiments, and CI integration step by step.

8. Promotion loop

When a user reports a failure in a refine/strategy/analyze workflow, the
on-call engineer runs the production case through the promotion loop to add it
to the golden dataset. This is the incident-cited regression rule applied to
evals (BRI-148-style). The goal is to make every real failure a permanent
test that cannot silently regress again.

Step 1: Flag

A user or on-call flags a production failure. The case is captured as a
structured payload:

  • brief or captureDoc: the input that triggered the failure.
  • observedFailure: what the model did wrong (free text, used in notes).
  • spanLink: optional Phoenix span URL for the trace.

No PII is required in the payload. The promotion tool runs a redaction pass
before writing to the dataset.

Step 2: Export

The engineer exports the failure as a JSON payload (file or stdin) and notes
the Linear ticket number (e.g. BRI-320). The ticket number is the provenance
anchor: it ties the eval case back to the incident so future readers can find
the context.

Step 3: Redact

bun eval-promote.ts runs a PII redaction pass over the brief before
writing. Redacted types: email addresses, phone numbers, bearer tokens/JWTs,
long hex blobs (40+ chars), long base64 data URIs (image/audio payloads).
A redaction report is printed in the output so the engineer can audit what
was changed.

Step 4: Promote

bun eval-promote.ts --ticket BRI-320 --suite refine < failure.json

eval-promote.ts does the following:

  1. Parses the JSON payload and validates it has a content field.
  2. Derives a deterministic caseId from the ticket number plus a slug
    derived from the brief. The format is {ticketSlug}-{briefSlug}, e.g.
    refine-bri-320-refuses-short-brief. Determinism means re-running the
    same promote is idempotent: the second run finds the duplicate and
    refuses to append.
  3. Loads existing caseIds from the target JSONL and refuses to overwrite.
    Append-only, never destructive.
  4. Validates the generated row through the matching schema parser
    (parseRefineCase, strategyCaseSchema, or analyzeCaseSchema).
  5. Sets provenance.kind = "prod", provenance.ticket = "BRI-NNN",
    provenance.date = YYYY-MM-DD.
  6. Appends the row to the correct JSONL file (refine.jsonl,
    strategy.jsonl, or analyze.jsonl) via Bun.write with append: true.

Step 5: PR

The row is appended as a single new line in the JSONL. A PR is required to
merge it into the golden dataset. The PR must include the Linear ticket so
reviewers can see the incident context. PR review checks:

  • The brief/captureDoc is accurate to the production failure.
  • Tags, budgets, and other fields are correct.
  • PII redaction was sufficient.
  • The notes field captures the failure mode in plain language.

What we do at Brioso: the promotion loop is gated behind the on-call
runbook. The eval-promote.ts script is the single entry point; do not
hand-edit JSONL files. caseId naming follows the refine-bri-NNN-slug
convention so tickets and cases are cross-referenceable.

9. Judge calibration cadence

Before a judge can gate a merge, two conditions must be met: calibration
(accuracy vs. human labels) and cadence (regular refresh to prevent drift).

Calibration gate

A judge is not a CI gate until it has been calibrated against 50 hand-labeled
cases (positive and negative). Report in the PR that introduces the judge:

  • TPR / TNR: true-positive rate, true-negative rate against human labels.
    TPR < 0.8 disqualifies the judge as a gate; it becomes a triage signal.
  • Inter-rater agreement: Cohen’s kappa against human labels.
    Agreement >= 0.6 is acceptable; >= 0.75 is strong.
  • Two-rater agreement (online evals): before judge verdicts gate merges
    in the online sampling flow, two raters annotate the same sample independently.
    Agreement must be >= 80-85% before the judge’s verdicts are trusted as
    authoritative. Below that threshold, judge verdicts are advisory only.

The rubric stays binary or decomposed checks, never a Likert scale:
“does output contain X?” (pass/fail), “does it avoid Y?” (pass/fail). Each
check is independently readable and auditable.

Quarterly refresh

The judge rubric, prompt, and any behavioral anchors must be refreshed at
least once per quarter. The trigger is calendar, not degradation signals:
judges degrade slowly and quietly. A quarterly refresh keeps the rubric
anchored to current product behavior rather than the behavior that existed
when the judge was written.

Refresh steps:

  1. A different engineer (not the original judge author) hand-labels 20 cases
    from the most recent quarter of production traces.
  2. Run the judge against those 20 cases.
  3. Compare TPR/TNR against the baseline from the original calibration run.
    Any regression > 5 points is a signal to rewrite anchors.
  4. If anchors are rewritten, re-calibrate against a fresh 50-case set before
    re-enabling the gate.

What we do at Brioso: judge calibration numbers live in the PR description
of the PR that introduced the judge. The quarterly refresh is tracked as a
recurring Linear task. Phoenix annotations are the source of truth for both
calibration cases and the human-in-the-loop review that drives online eval
sampling.

Previous Dogfood Runbook — apps/web Next Feature flags