CI runner policy

CI runner policy

Goal: GitHub Actions never costs money. Ladder, in order:

1. Self-hosted tailnet runners (RUNNER_TARGET=self-hosted-tailnet): the
   default. The OVH Garage VPS adds one slot to the application VPS fleet of four.
2. GitHub Free included minutes (RUNNER_TARGET=ubuntu-latest): a manual
   fallback with 2,000 Linux min/mo on the briosoco Free-plan org.
3. Hard stop: an org Budget with prevent_further_usage: true on the
   actions SKU. Even if the watchdog fails, spend stops at zero.

The blacksmith-4vcpu-ubuntu-2204 runner config stays deleted;
RUNNER_TARGET must never return to that value.

Watchdog

.github/workflows/runner-budget.yml runs every 6h
(scripts/ci-minutes-remaining.ts):

• remaining < LOW_BUFFER (100 min) AND a self-hosted-tailnet runner is online → PATCH
  RUNNER_TARGET to self-hosted-tailnet after a manual hosted fallback
• while on self-hosted-tailnet → keep it there across billing cycles
• billing API unreadable → the workflow FAILS LOUD (never assumes minutes
  remain). RUNNER_TARGET is left untouched.

The thresholds are minutes, not a boolean: LOW_BUFFER = LAST_FULL_PIPELINE_MINUTES
(20, measured 2026-10-10) x PIPELINES_PER_CHECK_WINDOW (5), so hosted stays
in use only while at least 5 full pipelines of minutes remain. Re-measure and
adjust both constants in scripts/ci-minutes-remaining.ts when CI cost or
push rate changes.

Manual check (dry-run, never PATCHes): mise run ci:minutes.

Secret/variable requirements:

• MINUTES_PAT: fine-grained PAT with org Administration: read (billing)
  and Variables: RW (the RUNNER_TARGET PATCH; GITHUB_TOKEN cannot do
  either).
• RUNNER_TARGET: repo variable (not secret), value ubuntu-latest or
  self-hosted-tailnet. All workflows read
  ${{ vars.RUNNER_TARGET || 'ubuntu-latest' }}.

One-time hard-stop setup (run once, verify after billing-platform changes):

gh api -X POST /orgs/briosoco/settings/billing/budgets \
  -f budget_product_sku=actions -F prevent_further_usage=true ...
gh api /orgs/briosoco/settings/billing/budgets   # verify

If no card is on file, GitHub Free also blocks Actions at 0 min by itself;
the budget is defense in depth for when that changes.

Application VPS self-hosted runners (historical)

Superseded by brioso-vps runners:
the ovhcloud-ghrunner-* fleet is decommissioned and CI runs on the Garage
VPS. The recipe below is kept for the sizing rationale and security model.

See the infrastructure map for host placement, accounts,
and the dated registration snapshot. The application VPS (ovhcloud) runs
24/7 in the tailnet with bun/uv/mise and the turbo cache on localhost.
The following runner model and setup recipe apply to that host. Runner facts:

• Outbound-only: the runner long-polls github.com; no inbound ports needed.
• The runner container runs as root with sudo available; no deploy keys and
  no secrets in env. Private-repo self-hosted runners execute arbitrary
  workflow code, so the container boundary IS the security boundary (labels
  are not).
• 4 registrations on the 6-core, 11.7 GB box (label self-hosted-tailnet
  on all) so a push’s ~12 jobs don’t queue behind 2 slots. Not 5: with the
  VPS’s other services at about 2.1 GB RSS, a fifth unconditional slot’s
  worst case exceeds the available ~8.9 GB. Add a 5th only if the
  CI_MEASURE=1 numbers show light jobs under ~1.2 GB and
  bun scripts/ci-queue-wait.ts p90 is still above 2 min.
• Fresh container per job (BRI-449): rootless Quadlet units
  deploy/ci-runner/ghrunner@.container (template, instances 1 to 4) run
  ghcr.io/briosoco/brioso-ci-runner (built and published by
  .github/workflows/ci-runner-image.yml from deploy/ci-runner/Containerfile:
  ubuntu:24.04 (same glibc as ubuntu-latest) + the official actions/runner tarball + mise-baked
  bun/node/uv/python, nothing else — no ffmpeg, docker, pwsh, awscli or
  gcc) with EPHEMERAL=true, --rm and Restart=always. The runner
  exits after one job, the container is destroyed and systemd recreates
  it with a new registration.
• The PAT comes from a podman secret (gh-runner-pat), mounted at
  /run/secrets/gh-runner-pat; the image entrypoint mints registration
  tokens from it — the PAT never sits in the unit or on disk in the
  clear.
• Shared bun cache: the ONLY volume is ci-bun-cache, mounted at
  /root/.bun/install/cache and shared by every job on every runner. One
  volume, not per-job volumes: per-job caches would re-download ~500 MB
  of dependencies every run, and the poisoning risk of a shared cache is
  accepted because same-repo PRs are opened only by the owner —
  poisoning would need a compromised dependency or account, and those
  already reach the build directly.
• Slice limits (deploy/ci-runner/user-997.slice (was user-995.slice)):
  MemoryHigh=6.5G (throttles, never kills), MemoryMax=8G,
  MemorySwapMax=1G, CPUWeight=40, IOWeight=40 on the ghrunner
  user’s logind slice, covering all containers. Each container also has a
  hard --memory=5g so a runaway job is OOM-killed alone.
• Heavy lock: the vite build and the boot smoke (~4 GB each) run under
  flock -w 900 /root/.bun/install/cache/.heavy-build on self-hosted
  runners, so only one is in flight fleet-wide (lock file on the shared
  volume, never a GitHub concurrency group, which cancels queued deploy
  builds).
• Per-slot memory env: the entrypoint writes the runner’s .env from
  CI_SLOTS and CI_SLICE_MAX_MB (set in the Quadlet template): GOMEMLIMIT,
  GOMAXPROCS, TURBO_CONCURRENCY, RAYON_NUM_THREADS,
  VITEST_MAX_WORKERS and a NODE_OPTIONS heap cap, so tsc, tsgolint,
  turbo, oxlint and vitest size themselves to slice-max / slots. Tune the
  values from the max-RSS table that mise run validate prints per step
  (TS and Python, via /usr/bin/time -v; CI_MEASURE=0 opts out).
  The tool is OS-level, so it works on any process, not just these gates:
  /usr/bin/time -v <cmd> on Linux (max RSS in KB, Percent of CPU),
  /usr/bin/time -l <cmd> on macOS (bytes). For a timeline use psrecord,
  for hardware counters perf stat.
  Hosted runners never see the .env.
• Queue watcher: ci-queue-watch.{sh,service,timer} (every 2 min,
  outside GitHub) emits a JSON status line, reaps stale offline
  registrations and pings Healthchecks.io ci-runner-queue (success, or
  /fail when the queue is non-empty with no runner and no healthy
  orchestrator, or a run has been queued over 6 h). It reads HC_PING_KEY
  from ~ghrunner/.config/ci-queue-watch.env. Details in the brioso-vps section below.
• Two timers (deploy/ci-runner/): ci-runner-image-pull.timer pulls
  the published :latest daily (the workflow rebuilds it weekly to keep
  the actions/runner version inside GitHub’s support window; a pull only
  retags, so running jobs are never interrupted); ci-bun-cache-wipe.timer wipes the cache
  volume monthly (safe while bun runs; replaces the old nightly prune).
• Tailscale ACL: tag the node tag:ci, egress to github.com + turbo cache
  only.

Setup recipe (Quadlet form — the @ template needs podman >= 5.2; the
Secret= unit key needs >= 4.8). Cutover: start instances 1 and 2, then
podman stop (not rm) the old myoung34 pair, add 3 and 4 while watching
free -m and the slice’s memory.peak; remove the old containers after a
green week:

# on the VPS, as the ghrunner user (uid 995)
podman pull ghcr.io/briosoco/brioso-ci-runner:latest
podman secret create gh-runner-pat -          # PAT on stdin, never on disk
cp deploy/ci-runner/ghrunner@.container \
  ~/.config/containers/systemd/
cp deploy/ci-runner/*.{service,timer} ~/.config/systemd/user/
install -D deploy/ci-runner/ci-queue-watch.sh ~/.local/lib/ci-runner/ci-queue-watch.sh
systemctl --user daemon-reload
systemctl --user start ghrunner@{1,2,3,4}.service
systemctl --user enable --now \
  ci-runner-image-pull.timer ci-bun-cache-wipe.timer ci-queue-watch.timer
# as root: install user-997.slice as the drop-in, then systemctl daemon-reload

The retired recipe — docker run --restart=unless-stopped myoung34/github-runner — combined EPHEMERAL=true with a
same-container restart, which was the documented root cause of both the
~16 GB writable-layer growth and the tamper gap: state (mise/bun
binaries) persisted across jobs, so a root PR job could poison a later
secret-bearing build. --rm + Restart=always is the fix.

Watchdog behavior if the runner is down: it will NOT flip
RUNNER_TARGET (it checks GET /repos/.../actions/runners for an online
runner carrying the label first). If minutes are exhausted AND no runner is
online, jobs fail rather than bill — acceptable, that is what the budget
hard stop would do anyway.

brioso-vps runners (scale to zero, BRI-433)

brioso-vps-ovh (the Garage VPS, 8 vCPUs, 23 GiB RAM) is the CI host. The four
ovhcloud-ghrunner-* runners and the old single slot are gone; nothing
queues on them. The ghrunner user is UID 997 here.

Model. Nothing runs when idle. The orchestrator
(deploy/ci-runner/ci-orchestrator) starts one fresh ephemeral container per
queued self-hosted-tailnet job, at most 4 at once, and each container is
removed when its job ends. A workflow_job webhook (HMAC verified by
adnanh/webhook, only /hooks/github exposed through the Cloudflare tunnel)
cuts latency; the 60 s ci-reconcile.timer polls GitHub and is the source of
truth, so a lost delivery, a dead tunnel or a reboot only delays a job by up
to a minute. Measured idle cost of four always-on runners was about 100 MB
RAM and 0.2 core, so the orchestrator is justified by clean per-job state and
no accumulated cache or layers, not by idle savings. K3s with ARC was
rejected: its server alone costs more than the idle fleet, dind needs
privileged pods, and it adds a second orchestrator.

Off switch and fallback. ci-runners.target is the always-on fleet
(ghrunner@1..4), installed but disabled. Disable ci-reconcile.timer and
ci-webhook.service, enable the target, and the orchestrator stays idle
while it is active. Stopping the target drains every runner; never mask.
Crash loops stop after 5 failures in 15 min and are reported by the watcher.

Job containers get --device /dev/fuse (when the host has it) so the runner image self-build can use buildah on fuse-overlayfs instead of vfs, which needed about 3x the image size on the shared /srv/ci (BRI-449 follow-up, .agents/plans/2026-10-17-ci-runner-image-disk.md).

Guard. Before each start: /srv/ci mounted, over 3 GiB free on root,
over 5 GiB free and 200000 inodes on /srv/ci, over 2 GiB MemAvailable. A
failing guard leaves the job queued and the watcher alerts. user-997.slice
bounds the fleet (MemoryHigh=13G, MemoryMax=16G); each container has a
5 GiB ceiling (a limit, not a reservation) and RuntimeMaxSec (2 h) caps a
hung job.

Credentials. The long lived fine-grained PAT (repo Administration RW,
Actions read) lives at /home/ghrunner/.config/brioso/github-runner-pat,
mode 0600. The wrapper mints a short lived registration token per container;
the PAT is never mounted into a job (workflow code runs as root in the
container and could read it). The webhook secret is in
~ghrunner/.config/ci-orchestrator.env, mode 0600.

Visibility. ci-fleet-status prints units, containers, registrations,
queue, disk and memory in one table. ci-queue-watch.sh writes a JSON line
per run to journald and /srv/ci/state/ci-fleet.jsonl, and alerts on a
non-empty queue with no runner and a stale orchestrator, or any run queued
over 6 h. runner-budget treats a healthy orchestrator (Healthchecks badge
CI_ORCHESTRATOR_BADGE_URL) as runner capacity, since zero runners online is
normal. See deploy/ci-runner/README.md for install and recovery.

Monitor placement (BRI-401)

Monitors (runner-budget, vps-health, pyinfra-drift) do NOT ride
vars.RUNNER_TARGET — a monitor that shares the fleet’s runner dies with
the very outage it exists to report (zero minutes → hosted job never
queues; dead VPS runner → no run at all). All three are pinned to
self-hosted-tailnet, on schedule/workflow_dispatch triggers only —
never pull_request, because a PR-triggered run on the box holding the
passwordless-sudo pyinfra key would be an exfiltration path. runner-budget
is self-hosted outright (it needs only the billing API, so it runs even at
zero hosted minutes); vps-health/pyinfra-drift reach secrets only via
the GitHub environment: (deployment branch policies gate the refs that
can resolve VPS_PYINFRA_SSH_KEY etc.).

Dead-man heartbeat: each monitor ends with
curl -fsS -m 10 "https://hc-ping.com/$HC_PING_KEY/$JOB" on success only
(HC_PING_KEY is the bare Healthchecks.io ping key; the same key feeds
ci-queue-watch on the VPS, so there is no URL secret) — no ping, and a
Healthchecks.io check (outside GitHub) fires the alert. JOB is
runner-budget / vps-health / drift-beta / drift-prod. Steps are
skipped while the HC_PING_KEY secret is unset; the owner creates the
checks and stores the secret.

The tailscale step in vps-health/pyinfra-drift is
if: runner.environment == 'github-hosted' (the self-hosted box is already
on the tailnet), followed by a tailscale status diagnostic on hosted
runs — the “Connect to tailnet” failures’ root cause (OAuth client,
tag:ci ACL, or the v4 ping input) is still open; owner must check the
TS OAuth client.

Trigger volume (reduced 2026-09, BRI-356)

• Deploy beta: dispatch-only (legacy push path removed during deploy-v2).
• Security Gates: PR-only (push:main duplicate dropped).
• CI validate: paths-ignore for **/*.md, docs/**, .agents/**,
  **/*.svg, videos/**.
• Slow tests: nightly cron disabled (commented in test-slow.yml with a
  re-enable note); PR paths-filtered trigger remains.
• pyinfra drift: daily at 06:17 UTC (was hourly); post-deploy drift still
  runs per deploy in pyinfra-deploy-beta.yml.

To re-enable a disabled schedule, uncomment the cron in the workflow file;
to check what a change will cost, gh run list before/after a week.

macOS/Linux parity

Root pyproject.toml declares [tool.uv] environments for both
darwin-arm64 and linux-x86_64, making the lockfile universal.
scripts/validate.py runs uv lock --check on every invocation: it
resolves the Linux dependency graph on the laptop in ~1s with no wheel
downloads, so a platform marker that breaks Linux resolution fails locally
instead of only surfacing after a push to CI.

The residual platform-only set that parity can’t (and shouldn’t) close —
these are real, intentional differences, not gaps:

uv tree on macOS vs. the VPS for the same commit should only ever diverge
on this list; anything new appearing in that diff is a regression.

Agent escalation order for platform-sensitive changes

Never push to observe behavior — push only to land batched work. For
anything platform-sensitive (Python deps, torch, native modules, Nitro
output):

1. mise run validate:quick locally, mise run dev:preview for build/SSR
   shapes. uv lock --check (now part of validate:quick:py) already
   catches the most common “works locally, fails in CI” class: dependency
   resolution divergence.
2. Still platform-sensitive after (1)? Run it on the VPS over the tailnet
   (ssh ovhcloud, same mise run <task>) — same OS/arch/uv resolution,
   warm caches, seconds instead of a push-build-deploy round trip.
3. VPS-only runtime bugs: mise run deploy:beta -- <sha> on the box, then
   mise run ops:health.
4. Push to land work, batched — never to observe it.

Agent conventions

Batch pushes and use [skip ci] on docs-only intermediate commits; see
AGENTS.md.