BRI-250 benchmark — latency, GPU/CPU energy, idle states (2026-08-08)

BRI-250 benchmark — latency, GPU/CPU energy, idle states (2026-08-08)

Supersedes bri-250-preliminary-gpu-benchmark-2026-08-07.md for latency and
GPU energy numbers (this run uses 10Hz GPU power sampling vs. the
preliminary run’s 1Hz, so it resolves short ~2.2s jobs far better — 2-3
samples per job at 1Hz badly undersampled the power ramp). Adds real CPU
package energy via Intel RAPL (exact counter delta, no sampling error) and
two of the four required operating states.

Still not done, and this doc does not claim otherwise: State D
(sustained/max-throughput, batch_size > 1), the 3-7 day continuous run, and
a real wall-power-meter reading. Everything “whole-PC” below is a
software-measured DC total with an explicit, clearly-flagged margin
applied — not a wall measurement. See “Conscious gap” below for exactly
what that margin does and doesn’t cover, and why a real meter still
matters before a final go/no-go call.

Method

apps/home-gpu-server/scripts/benchmark_phase2.py, run directly on the box
(not over SSH, so latency isn’t skewed by network round-trip) against the
resident home-gpu-server service. 5 untimed warm-up calls, then 100 timed
POST /generate calls at duration=180s, steps=8 (the BRI-249/250
reference case), sequential (max_concurrency=1).

Per call: CPU package energy via /sys/class/powercap/intel-rapl:0/energy_uj
delta (exact, not sampled). GPU energy via nvidia-smi --query-gpu=power.draw -lms=100 (10Hz) sampled continuously in the background and
trapezoidal-integrated over each call’s wall-clock window — this GPU/driver
doesn’t expose a cumulative energy counter the way RAPL does, so sampling is
the only option there.

Idle states A and B measured separately with
apps/home-gpu-server/scripts/measure_idle_state.py: 30s of continuous
10Hz GPU power sampling + RAPL energy delta, with zero in-flight requests.

Results: latency (100 runs, 180s/8-step, warm)

Extremely tight distribution (P99/min ratio ~1.004) — no cold-start,
throttling, or GC-style jitter across 100 back-to-back generations. Confirms
the preliminary run’s 2.18s figure was not a one-off: ~28x faster than
Runpod’s ~60s/job baseline.

Results: energy per song (100 runs, DC-side, software-measured)

Combined DC energy/song: 0.1487 Wh avg (GPU + CPU package, no PSU loss
or board overhead applied yet).

Note this revises the preliminary run’s GPU-only 0.085 Wh/song estimate
upward to ~0.141 Wh/song — the earlier number came from 1Hz sampling
against ~2.2s jobs (2-3 samples), which likely missed most of the power
ramp. This run’s 10Hz sampling (~21-22 samples/job) is materially more
trustworthy.

Results: idle states A and B

Directly answers epic question #5 (“does keeping the model resident
materially increase idle draw?”): no — GPU idle draw is nearly
identical whether the model is loaded or not (16.3W vs 17.0W), since a
resident-but-inactive model just sits in VRAM without drawing meaningful
extra power. State B’s lower CPU reading than State A is more likely
measurement noise from a single 30s window right after a service
stop/restart than a real effect — the eventual 3-7 day run will resolve
this cleanly since it removes the boot-adjacent noise entirely.

Conscious gap: estimating whole-PC from DC-side numbers

Two things happen between “DC power the OS can measure” and “watts the
wall meter would show,” and neither is visible to software running on the
box:

1. PSU AC→DC conversion loss. Assumed 80-90% efficiency (a defensible
   range for a decent-but-unrated/mid-tier PSU; a certified 80+ Gold unit
   would sit at the high end, an older/unrated unit at the low end — the
   real unit’s rating isn’t confirmed). This box’s actual PSU has not been
   inspected for its efficiency certification. Still the single largest
   open unknown in this whole section — nothing below narrows it.
2. Motherboard/chipset/VRM-beyond-CPU-rail/fans/storage/NIC (“board
   tax”). RAPL’s package domain covers the CPU die and DRAM controller,
   but nothing else on the board. Originally a blind 15-35W guess; refined
   below into a bottom-up estimate grounded in the box’s actual identified
   hardware.

Board-tax breakdown (bottom-up, not a blind guess)

Identified via SSH (/sys/class/dmi/id/board_name, lsblk): ASUS
MAXIMUS VII HERO (Z97 chipset), storage is 2x SATA SSD — Samsung 840 +
Kingston SA400S3, no spinning HDD. No fan RPM telemetry available without
force-loading a sensor kernel module (nct6775), which wasn’t done — fan
count/speed below is still an assumption, not measured.

Storage turned out to be genuinely negligible — SATA SSDs draw well under
1W idle, far less than the original guess assumed. The remaining
uncertainty is concentrated in fan count/RPM and motherboard VRM/controller
overhead, neither of which can be pinned down further without either
forcing the sensor driver or a real wall meter.

Both ranges are explicitly conservative estimates, not measurements. A real
wall meter replaces both with one number and removes this whole section —
that’s still the right next step before a final go/no-go, this is a
stopgap so Phase 2 analysis isn’t fully blocked on procurement.

Marginal cost/song, GPU+CPU only (board tax excluded — it’s fixed
overhead, not incremental to one more generation; only PSU loss applies):

Marginal cost/song, all components (adds board tax’s contribution
during the ~2.18s active generation window only — not the idle-hours
amortization below, just what the whole workstation draws while that one
song is actually being generated):

Board tax barely moves this number — a song only takes ~2.2s, so even the
high-end 26.6W board tax only contributes ~0.016 Wh to a ~0.15 Wh song.
This is why “worst case” here (41,496 songs/€1) isn’t far from the
GPU+CPU-only worst case (46,083 songs/€1) — the real cost driver for
“cost per song” is compute, not board overhead. Board tax matters far more
for the idle/24-7 number below, where it’s paid every second regardless
of whether a song is generating.

Monthly idle/keep-warm cost (refined 13.9-26.6W board tax + PSU loss,
using the realistic-run weighted idle baseline of 25.1W DC):

Working range: €3.65-5.44/month to keep the box up and model-resident
24/7 — narrower than the original €3.5-6.5 blind-guess range.

Runpod comparison

Fully-loaded cost/song at volume (refined board-tax range)

Crossover volume vs. Runpod’s €0.025/song: ~146-218 songs/month
(refined from the original ~140-260 range) — even under the conservative
end of the bottom-up assumptions. Below that volume the home GPU is still
cheaper than Runpod as long as monthly usage clears roughly 146-218 songs
— a bar the epic’s own example (~300 generations/user/month for a single
user) already clears on its own.

This does not include Model B (full TCO with hardware
depreciation/maintenance) — only Model A (marginal cash cost, existing
hardware as sunk cost), per the epic’s own framing.

Realistic mixed-traffic run (24.4 min, 393 jobs)

The clean 100-run sweep above is a controlled, uniform-load benchmark. This
run instead simulates bursty real traffic against a single continuous
power-sampling session (GPU 10Hz + CPU RAPL 1Hz running the whole time, so
idle gaps between activity are captured as real idle power, not skipped):
a 5-job concurrent burst, 3min idle, 1 single job, 1min idle, another
5-job burst, 5min idle, then 15min of back-to-back sequential load.
Script: apps/home-gpu-server/scripts/realistic_workload_bench.py.

Queueing: both 5-job bursts hit max_queue_depth: 4 (1 processing + 4
queued) — the server absorbs simultaneous-request spikes by queueing
rather than erroring, as designed (max_concurrency=1).

Idle cross-validation: three independent idle windows averaged 25.1W
combined DC (weighted by duration) — consistent with the controlled State
A/B measurements above (21-27W) despite following real burst activity, not
a clean stop/restart. Idle draw doesn’t drift with traffic history.

Margin-adjusted economics from this run (using the refined bottom-up
13.9-26.6W board tax — see “Board-tax breakdown” above): marginal cost
€0.0000202-0.0000227/song (GPU+CPU only), monthly idle cost €3.65-5.44,
crossover vs. Runpod’s €0.025/song at ~146-218 songs/month — this
run’s idle baseline (25.12W) is close enough to the clean benchmark’s
(25.1W) that, once both use the same refined board-tax range, the two
crossover estimates converge almost exactly. Two differently-shaped tests
landing on the same crossover band is a meaningfully stronger signal than
either alone.

What’s still open before a final go/no-go

1. Real wall-power-meter reading — replaces the “conscious gap” section
   above with one measured number. Board tax is now a refined bottom-up
   estimate (13.9-26.6W) rather than a blind guess, but PSU AC→DC
   conversion efficiency (80-90% assumed) is now the single largest
   remaining unknown — nothing short of a real meter (or inspecting the
   PSU’s own efficiency certification label) narrows that further.
2. State D — sustained/max-throughput, batch_size > 1. The current API
   is single-request (max_concurrency=1); testing this needs either a
   server-side batching change or running concurrent sa3_worker processes
   outside the resident-model service.
3. 3-7 day continuous run — idle draw is now cross-validated across
   five separate windows (two 30-60s controlled snapshots plus three
   1-5min windows from the realistic-traffic run, all landing in the same
   21-32W band), which is stronger than a single snapshot but still not
   the sustained real-world average the epic asks for — also the only way
   to catch thermal drift, memory leaks, or driver issues a short
   benchmark can’t surface.
4. Full TCO (Model B: GPU/PC depreciation, maintenance, optional UPS).

The signal so far is strong enough that none of the above looks likely to
flip the go/no-go call — electricity cost clears Runpod’s bar by roughly
two orders of magnitude even under conservative assumptions — but per the
epic’s own instruction not to skip Phase 2’s rigor, these remain open
before formally closing BRI-250.