2026-07-17 — VPS state management: migration vs reconciliation research

Superseded by state-mgmt-policy.md (most findings incorporated).

2026-07-17 — VPS state management: migration vs reconciliation research

Status: researched, not implemented. Filed for later. Owner: TBD.

Problem statement (verbatim from user)

“it seems to me that with deployment we’re putting a lot of effort on setup and then cleanup of unused/dead services. Can’t we just make it in a db-migration-like basis (ie on deployment we do a series of checks which verify the server is in a ‘healthy’ state and we can automate some scripts for cleanup)?”

User’s pain, concretely observed during 2026-07-15 work:

• Manual systemctl edit drop-ins accumulating on the VPS (drizzle-gateway had 5 bugs patched this way).
• Dead config in unit files (e.g. DB_FILE= nobody reads).
• Old units leaving behind state after decommission (FastAPI stack: brioso-api, brioso-refine).
• 5+ orphan nono profiles at /home/web-services/.config/nono/profiles/ from the FastAPI cutover, repo only has 2.
• Repo unit files drifting from live canonical (otelite drift; would have broken the next setup-observability.sh).
• setup-observability.sh only manages 2 of 4 units. brioso-web and brioso-litestream have no automated installer.
• deploy-vps.sh only invokes the reconciler on a fallback branch (if [[ ! -x drizzle-gateway ]]).

TL;DR (both research lanes agree)

This is not a migration problem. It is a reconciliation problem. The repo unit files are already the desired state; the gap is enforcing them and reaping orphans. Build a shell-only declarative reconciler wired into every deploy, plus a read-only drift gate. ~150 lines of shell, zero new dependencies. Skip migrations, Ansible, NixOS, Packer — all overkill for 1 VPS / 4 units / 1 developer.

The half-built reconciler (setup-observability.sh) already exists and is correct in spirit; the work is to generalize it, make it unconditional, add a manifest, add a cleanup loop, and pair it with a verify gate.

Why migrations are the wrong model (oracle)

Numbered migrations (0001_initial.sh, 0002_...) earn their keep when transitions are irreversible (DB schema with live data). Systemd units are already declarative — the file at deploy/systemd/brioso-otelite.service is the desired end state. A migration system would re-derive that state through N stacked steps, and you’d still need a verify step on top.

Additional migration failure modes at this scale:

• Migrations don’t compose (migration 12’s effect silently undone by migration 28).
• Decommissions need a paired 0028_remove_stale_api.sh — easy to forget, easy to get wrong, no enforcement.
• No diff/plan output — run it and hope.
• No standard library; each ensure_* is hand-rolled.
• Real-world adoption: rare. Mostly personal infra dotfiles repos. No widely-adopted framework for shell-based server-state migrations.

Approaches considered and rejected

The recommendation

One shell script that owns VPS state, called unconditionally on every deploy. Plus a read-only verify step that fails CI on drift.

reconcile-vps.sh (the apply step)

Reads a manifest (array literal at top — no YAML). Three responsibilities:

1. Install + enable every unit in the manifest — copy from deploy/systemd/ to /etc/systemd/system/, including tracked drop-ins under deploy/systemd/$unit.service.d/. Run the existing idempotent installers (install-drizzle-gateway.sh, install-otelite.sh). Set tailscale serve rules.
2. Cleanup — disable + remove any brioso-*.service in /etc/systemd/system/ NOT in the manifest. This is the automatic dead-service removal: drop a unit from the repo, next deploy reaps it on the VPS.
3. systemctl revert to flush untracked drop-ins before copying — the canonical files in the repo are the source of truth, drop-ins are an escape hatch that should not survive a deploy.

Folds in (or imports) setup-observability.sh. Ponytail: fold it, one fewer file to remember.

verify-state.sh (the gate)

Read-only. For each manifest unit, diff /etc/systemd/system/$u.service against deploy/systemd/$u.service. Same for profiles. Exit non-zero on any mismatch. Called from .github/workflows/deploy.yml before deploy-vps.sh over SSH. On failure, the deploy aborts with a clear “drift detected” message instead of silently clobbering.

Wire-in

In deploy-vps.sh, replace the conditional block at lines 86-89 (if [[ ! -x /usr/local/bin/drizzle-gateway ]]; then ... setup-observability.sh ... fi) with an unconditional reconcile-vps.sh call before the build. Drop the if — it’s a smell that the reconciler wasn’t trusted.

Why this is the lazy answer

The repo unit files have been the source of truth all along. The only missing pieces:

1. Actually copying them on every deploy (today: only 2 of 4 units, conditional).
2. Checking they match before deploy (today: nothing).
3. Enumerating which units are expected so orphans can be detected (today: implicit, scattered).

That’s ~150 lines of shell. No new deps. No framework. The cleanup loop is ~10 lines and is the entire answer to “how does dead state get removed automatically.”

Suggested commit breakdown (when we tackle this)

Optional: goss as the verify step instead of hand-rolled diff

The librarian recommended goss for the verify side. It’s a 5MB Go binary, YAML specs, auto-generates from live system via goss autoadd, exits non-0 on drift. If hand-rolled diff in verify-state.sh feels too brittle, goss is the upgrade path.

Tradeoff:

• Hand-rolled diff — zero deps, ~30 lines, but limited to “files match.”
• goss — adds one binary, but can also check ports listening, users exist, file modes, command outputs. Better long-term audit surface.

Ponytail call: start with hand-rolled diff, switch to goss if/when verify needs grow past file comparison.

Drift classes the reconciler solves vs. doesn’t solve

Risk flags from the architectural review (ora-1)

Things that exist today and need attention when we pick this up:

1. 5+ orphan nono profiles at /home/web-services/.config/nono/profiles/ (brioso-api, brioso-cloudflared, brioso-litestream, brioso-refine, plus the otelite one we added). Repo has 2. Decide: reap or accept as harmless.
2. brioso-litestream.service.d/logging.conf is a tracked drop-in. The reconciler must handle *.service.d/ directories, not just bare .service files — otherwise the logging config gets lost. Easy to miss in commit 1.
3. No reconciler for secrets (web.env, litestream.env, drizzle-gateway/secrets.env). A fresh-VPS bootstrap will fail opaquely. Worth a separate verify-secrets.sh backlog item that checks existence + mode without reading contents.
4. brioso-drizzle-gateway.service WorkingDirectory=/opt/brioso is conceptually wrong (it’s the deploy symlink farm). Harmless today because STORE_PATH redirects writes, but worth fixing while we’re in there.
5. tailscale serve reset is destructive and the reconciler makes it more so. Every tailscale serve rule must live in reconcile-vps.sh. Document with a ponytail: comment.
6. Confirm brioso-web.service StartLimit* location. The 2026-07-15 doc claimed lines 17-18 were under [Service] (silently ignored). Current HEAD shows them under [Unit] — appears fixed during commit 07d176e. 30-second confirm during commit 3.

Reference

• Source research:
  • lib-2 (librarian) — industry patterns: goss, testinfra, InSpec, Ansible, NixOS, Packer, migration-style shell.
  • ora-1 (oracle) — architectural review of current deploy/ setup, opinionated recommendation.
• Related session work:
  • .agents/docs/2026-07-15-drizzle-gateway-otelite-tailscale-learnings.md — the bug debrief that surfaced this pain.
  • .agents/plans/2026-07-15-fix-observability-unit-and-deploy-gaps.md — the fix plan that landed 6 commits but did not address the underlying reconciliation gap.
• Tools referenced:
  • goss: https://github.com/goss-org/goss · https://goss.readthedocs.io
  • Ansible systemd_service module: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/systemd_service_module.html
  • testinfra: https://testinfra.readthedocs.io
  • Chef InSpec: https://docs.chef.io/inspec/

Status

Research complete. Implementation deferred per user direction (“we’ll tackle it later”). When picking up:

1. Re-read this doc.
2. Re-verify the risk flags (some may have been addressed by other work).
3. Write the plan in .agents/plans/<date>-vps-state-reconciler.md.
4. File a GH issue in briosoco/brioso and add to project 1 (likely Phase 7: VPS systemd deploy + CI, or a new Phase 8 sub-item).
5. Implement commits 1-3 above.