Diag Access Runbook

Diag Access Runbook

Locked-down diag Linux user on the VPS for teammates’ AI agents to run
diagnostics over Tailscale SSH: read journald, check unit health, and restart
exactly the two web slots. Replaces the need for any per-agent service or
token: ssh diag@<vps> "journalctl ..." is the whole interface.

Scope for the diag user: read all journald units (via the
systemd-journal group), systemctl status / is-active on
brioso-web*, and sudo systemctl restart on brioso-web-a and
brioso-web-b only. No root, no shell as root, no file reads outside
world-readable paths, no writes anywhere.

What diag can do

Everything else is denied by normal file permissions (/home/* is 700) and
by sudoers: sudo -n systemctl restart caddy, sudo -n /bin/bash, any other
sudo invocation fails.

Provisioning (pyinfra)

The user, group membership, permissions, and sudoers are provisioned by the
v2 diag_user pyinfra subsystem (deploy/pyinfra/v2/state/diag_user.py,
all envs). The sudoers drop-in /etc/sudoers.d/diag is rendered from
deploy/sudoers/diag (kept in exact parity by
v2/tests/test_v2_sudoers_parity.py):

diag ALL=(ALL) NOPASSWD: /usr/bin/systemctl status brioso-web*
diag ALL=(ALL) NOPASSWD: /usr/bin/systemctl is-active brioso-web*
diag ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart brioso-web-a.service
diag ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart brioso-web-b.service

Validate after deploy with sudo visudo -c. (Both files exist on the branch
and are being implemented in parallel; this runbook covers the access model
either way.)

last-result.json readability (lockstep rule)

The deploy result file <deploy_dir>/last-result.json is rewritten by the
root-run activate binary on every deploy, so its permissions must be part
of the desired state, not a one-off chmod (which regresses at the next
deploy). The lockstep rule across three files:

• deploy/activate/src/commands.ts (writeResult): writes the file mode
  0644.
• deploy/pyinfra/v2/state/diag_user.py: keeps both deploy dirs
• This runbook: documents the capability (table above).

Group path traversal (diag in the dir’s group, dir g+x) makes the
root-owned file readable without sudo — plus a traverse-only (--x) ACL
for diag on /opt/brioso itself, which is web-services:brioso 0750 and
would otherwise block diag from entering the tree at all (same module
pins it; --x grants no listing, so shared/ env files stay unreachable).
If any of these moves, move all.

Blue-green restart caveat

Restarts are not free: which slot is active matters.

• Restarting the inactive slot is always safe: Caddy proxies to the other
  slot, and deploys use this exact mechanism.
• Restarting the active slot briefly drops traffic (seconds, user-visible)
  until the unit comes back.

Check which slot is active before restarting: read /opt/brioso/deploy/state.env
on the VPS (ACTIVE_SLOT / the active slot’s port is recorded there). When in
doubt, don’t restart; report instead.

Common agent one-liners

journalctl -u brioso-web-a -n 200 --no-pager   # recent logs for a slot
systemctl --failed                             # any failed units
df -h                                          # disk
systemctl status brioso-web-a                  # unit health
sudo -n systemctl restart brioso-web-a         # restart (see caveat above)

Manual Tailscale steps (one-time, out-of-repo)

These are done in the Tailscale admin console, not by pyinfra. Export the
live policy first and edit on top of it — never paste
.agents/docs/tailscale-acls.jsonl over live (it would drop CI’s SSH rule and
break every deploy).

1. Network grant on :22: Tailscale permits an SSH connection only when
   the policy has both a network grant and an SSH rule. Add to grants:

   {
     "src": ["autogroup:member"],
     "dst": ["tag:server"],
     "ip":  ["tcp:22"]
   }
   

   Members otherwise only have tcp:443 to tag:server, so without this they
   are blocked at the network layer before the ssh section is consulted —
   which is why the VPS looks “online and reachable” (ICMP/443 work) while SSH
   does not.
2. SSH ACL rule: add to the policy’s ssh section:

   {
     "action": "accept",
     "src":    ["autogroup:member"],
     "dst":    ["tag:server"],
     "users":  ["diag"]
   }
   

   accept, not check: teammates point non-interactive AI agents at this and
   check stalls the first connect on a browser re-auth URL. Attribution still
   comes from tailnet identity in the connection logs, and the account is
   fenced by sudoers + file permissions. The existing baseline check rule
   grants members SSH to autogroup:self (their own devices) as any nonroot
   user; it does not cover tag:server, so the two rules don’t conflict.
3. Session recording for diag (follow-up, not yet done): record is not
   a valid ssh action — the only actions are accept and check. Recording
   is a recorder field (plus optional enforceRecorder) on an
   accept/check rule and needs a tsrecorder node provisioned and
   tagged in the tailnet. Tracked as BRI-376; do not add a record rule.
4. Verify the final policy with the ACL editor’s rules preview:
   • members are allowed to tag:server as diag and as no other user;
   • tag:ci -> deploy (the CI deploy rule) is still allowed.

After saving, overwrite .agents/docs/tailscale-acls.jsonl with the live
post-change export (not a hand-edit) and keep doing that after every console
change — the file is a read-only mirror, not the source of truth.

Audit

• Tailscale admin console: connection logs tie every SSH session to the
  tailnet identity that opened it, before mapping to the shared diag Linux
  user.
• journald sudo logs: journalctl -t sudo on the VPS shows every sudo
  command diag ran (command line, tty, exit status).
• Session recording is NOT active (no tsrecorder node yet — see Manual
  Tailscale steps #3). Until it is, sudo journald logs plus Tailscale
  connection logs are the only attribution for the shared diag account.
• Shared-user audit story: multiple teammates share diag, but attribution
  comes from Tailscale identity at connect time; sudo logs add command-level
  confirmation.

Revocation (per teammate)

There are no per-user credentials on the VPS to rotate. To revoke a teammate:
remove them from the tailnet. If finer grain is wanted later, switch the ACL
rule’s src from autogroup:member to a dedicated group:diag and manage
membership in the admin console.

Verification

After provisioning and the manual Tailscale steps:

1. Positive (as a teammate, from their tailnet machine, non-interactive):
   ssh -o BatchMode=yes diag@<vps> 'journalctl -u brioso-web-a -n 5 --no-pager'
   succeeds; ssh -o BatchMode=yes diag@<vps> 'sudo -n systemctl status brioso-web-a'
   succeeds with no browser prompt and no password prompt.
2. Negative: ssh root@<vps> and ssh deploy@<vps> as a member are
   denied; sudo -n systemctl restart caddy as diag fails with “not allowed”;
   cat /home/<other-user>/.bashrc gets permission denied;
   sudo -n /bin/bash is refused. (Verified 2026-08-26 on prod via
   sudo -u diag — all pass.)
3. CI not broken (the real regression risk): re-run deploy.yml (or push
   a trivial commit to beta) in the same session as the policy change and
   confirm the Deploy over Tailscale SSH step still connects as deploy@.
4. Recording: not active — no tsrecorder node yet (Manual Tailscale steps
   #3). BRI-376 tracks provisioning it.
5. LogMCP no-regression — LogMCP was removed repo-side on
   2026-08-26 (WS4 of the deploy v2 restructure); the VPS-side decommission
   steps are in
   .agents/docs/windmill-logmcp-decommission-runbook.md.
   Once executed, systemctl status brioso-logmcp should report the unit as
   not found/inactive.
6. mise run validate green for the repo change (pyinfra + docs).

LogMCP removal

LogMCP was decommissioned 2026-08-26 (repo-side removal in the WS4 branch;
host-side steps pending in
windmill-logmcp-decommission-runbook.md).
It was loopback-only in practice — the logs.usebrioso.com Caddy block was
never provisioned. Diag SSH is now the sole logs/journald surface and the
restart surface lives in diag’s sudoers (the LogMCP clitool plugin was never
provisioned with a write token).

Teammate handoff

No SSH key is needed. With Tailscale SSH enabled, tailscaled terminates
the connection itself and authenticates the tailnet identity over WireGuard
node keys (SSH auth type none) — there is no authorized_keys file for
diag, and the pyinfra module deliberately never creates one
(deploy/pyinfra/v2/state/diag_user.py). Any key your local ssh offers is
ignored. Nothing to generate, nothing to share.

Prerequisites:

1. Be a member of the tailnet (autogroup:member) with Tailscale running and
   logged in on the machine you connect from.
2. ssh diag@<vps-tailnet-host>.

No tokens to store, no MCP client config, no per-user credentials. Access
follows tailnet membership. The ssh rule uses accept, so there is no
device-approval or browser prompt on first connect.

Two gotchas that look like auth failures but are not:

• REMOTE HOST IDENTIFICATION HAS CHANGED — Tailscale SSH presents its
  own host key, distinct from the VPS’s sshd host key. If you ever SSHed to
  that host directly, drop the stale entry:
  ssh-keygen -R <vps-tailnet-host>.
• Permission denied (publickey) — the connection reached the real sshd,
  not Tailscale SSH, i.e. the policy change has not taken effect for you yet.
  It is not a missing-key problem; do not “fix” it by asking for a key.