BRI-250 preliminary GPU-only benchmark (2026-08-07)
BRI-250 preliminary GPU-only benchmark (2026-08-07)
Quick, informal sanity check run on the home GPU box (gpu-worker, RTX 4070
Ti Super) right after BRI-249’s Phase 1 verification passed. This is not
the rigorous Phase 2 benchmark BRI-250 asks for — it’s GPU-telemetry-only,
one machine state, one 10-minute window, no wall power meter. Keep as a
reference/sanity baseline; BRI-250’s actual DoD still needs the full
multi-state, multi-day, whole-PC measurement below.
Method
Sequential POST /generate calls (stable-audio-3-medium, 8 steps, no
sleep between requests — server’s max_concurrency=1 serializes them
naturally) cycling through durations [15, 30, 60, 90, 120, 150, 180]s, for
a fixed 600s wall-clock window. nvidia-smi --query-gpu=... -l 1 sampled
power/utilization/VRAM/temp once a second in parallel. Energy integrated via
trapezoidal rule over the power samples.
Results
- 347 jobs completed, 0 failures, over a 599.1s window.
- GPU power: avg 177.7W, max 262.4W (min 16.7W between requests).
- GPU utilization: avg 60.2%, max 100%.
- VRAM: constant 9,658 MiB (model resident throughout — no reload spikes).
- GPU temp: avg 60.8°C, max 67°C — no throttling signs at ~60%+ sustained
utilization. - Total GPU energy over the window: 29.588 Wh → €0.00346 (at
€0.116952/kWh). - GPU energy per song: 0.085 Wh → €0.00001/song (GPU electricity only,
excludes idle/24-7 keep-warm cost).
Inference latency by requested duration (8 steps fixed; latency scales with
duration, not step count — decoding/serialization dominates at longer
lengths):
| Duration | Avg inference time |
|---|---|
| 15s | 0.73s |
| 30s | 0.79s |
| 60s | 0.98s |
| 90s | 1.12s |
| 120s | 1.44s |
| 150s | 1.80s |
| 180s | 2.18s |
180s/8-step generation is the DoD reference case (BRI-249 #4): 2.18s warm
vs. Runpod’s ~60s baseline, ~28x faster.
What this does NOT answer yet
Per BRI-250’s actual scope, still needed before a go/no-go decision:
- Whole-PC wall power (HIGH PRIORITY per the epic) — GPU telemetry
misses CPU, motherboard, RAM, SSD, fans, PSU losses. Needs an external
smart plug / wall meter. - The 4 required operating states — idle/no-model, idle/model-loaded
(warm), active inference, sustained max throughput — each with whole-PC
W, not just GPU W. - Multi-day (3-7 day) continuous run to capture real 24/7 availability
cost, not a 10-minute burst. - Marginal vs. fully-loaded cost/song, and the two TCO models (marginal
cash cost vs. full economic cost with depreciation). - Batch-size > 1 capacity benchmark.
- The actual Runpod comparison table (this doc only has the home-GPU side
filled in).
This preliminary run does suggest the GPU-electricity component alone is
not going to be the bottleneck in the economics — the open question is
almost entirely about idle/24-7 draw, which whole-PC measurement will
answer.