2026-07-15 — Drizzle Gateway + otelite + Tailscale HTTPS: session learnings

Superseded by deploy v2 (BRI-338) + windmill-logmcp-decommission-runbook.md.

2026-07-15 — Drizzle Gateway + otelite + Tailscale HTTPS: session learnings

Session date: 2026-07-15. Worked directly on the VPS (ovhcloud.tail876c20.ts.net).
Source debrief lives at /home/debian/Documents/prjcts/docs/drizzle-gateway-otelite-tailscale.md
on the VPS and is preserved verbatim upstream of this document. This file is the
in-repo summary, extended with two related bugs the original debrief missed and
with the deploy-automation gap quantified.

Outcome

Both observability backends are now reachable over the tailnet with HTTPS:

• https://ovhcloud.tail876c20.ts.net:8000 → Drizzle Gateway (127.0.0.1:4983)
• https://ovhcloud.tail876c20.ts.net:8001 → otelite web UI (127.0.0.1:16686)

All fixes were applied on the VPS as systemctl edit drop-ins, not by hand-patching
/opt/brioso/current or /opt/brioso/repo, per the project’s deploy convention.
They need to land in the actual unit files and deploy scripts in git so the next
redeploy does not reintroduce them.

Bugs in deploy/systemd/brioso-drizzle-gateway.service

All five are present in the file as committed (verified 2026-07-15 against HEAD):

Bug 1 — User=brioso / Group=brioso does not exist

Only a brioso group (gid 1003) exists on the VPS; web-services is its only
member. The unit runs as a non-existent user and systemd refuses to start it.

Every other unit (brioso-web, brioso-litestream, brioso-otelite) runs as
User=web-services / Group=web-services (or Group=brioso in the web case, both
exist). Fix: use User=web-services / Group=web-services.

Bug 2 — ReadWritePaths= uses comma separator

Systemd’s ReadWritePaths= is space-separated. The unit sets
ReadWritePaths=/opt/brioso/shared/data,/etc/drizzle-gateway, and systemd parses
that whole comma-joined string as one literal path. It does not exist, so the unit
fails with status=226/NAMESPACE (“Failed to set up mount namespacing: …
No such file or directory”).

Fix: change to ReadWritePaths=/opt/brioso/shared/data /etc/drizzle-gateway
(space between the two paths).

Bug 3 — StartLimitIntervalSec=60 and StartLimitBurst=5 under [Service]

Both keys are [Unit]-section directives in current systemd. Placed under
[Service] they are silently ignored (Unknown key 'StartLimitIntervalSec' in section [Service], ignoring), so the crash-loop rate limit configured alongside
StartLimitBurst=5 never actually applies.

Fix: move both lines under [Unit].

Bug 4 — Environment=DB_FILE=... is dead config

strings-d the drizzle-gateway binary: it only reads MASTERPASS and STORE_PATH
from the environment. There is no DB_FILE, DATABASE_URL, or similar env knob.
Drizzle Gateway is configured through its own web UI on first login (dialect:
SQLite, type: file, path /opt/brioso/shared/data/local.db — matches the web
app’s DATABASE_URL in /opt/brioso/shared/web.env).

Fix: remove the Environment=DB_FILE=... line. It is dead config that misleads the
next reader into thinking the DB is pre-wired.

Bug 5 — Missing STORE_PATH env var

Drizzle Gateway persists its connection config to ./drizzle-gateway-volume/store.json
by default (relative to WorkingDirectory, which is /opt/brioso).

Two compounding problems:

1. /opt/brioso is read-only under ProtectSystem=strict and not covered by
   ReadWritePaths, so every startup fails with
   Failed to load data: EROFS: read-only file system, mkdir './drizzle-gateway-volume'.
2. The WorkingDirectory=/opt/brioso is also conceptually wrong — that path is the
   deploy symlink farm and never the right place to write app state.

Fix: add Environment=STORE_PATH=/opt/brioso/shared/data/drizzle-gateway-volume
and ensure that directory exists, is owned by web-services:web-services, and
is covered by ReadWritePaths (it already is once Bug 2 is fixed — both paths
are under /opt/brioso/shared/data).

Bug 6 — deploy/systemd/brioso-otelite.service runs as a non-existent user

User=otelite / Group=otelite — otelite is not a user on the VPS (verified
2026-07-15 via id otelite → “no such user”). install-otelite.sh:72-74
does create the user conditionally, but the deploy path on this VPS apparently
never ran that branch.

Same class of bug as Bug 1 (drizzle-gateway’s User=brioso/Group=brioso).

Fix: use User=web-services / Group=web-services. Same dependency on the
web-services group already existing on the VPS — verified.

Adjacent hazard — repo brioso-otelite.service and the live unit have drifted

Confirmed live on the VPS via /etc/systemd/system/brioso-otelite.service: the
unit the operator actually built and is running:

• Wraps everything in nono run
• Uses User=web-services / Group=web-services
• Uses positional flags --addr / --storage-path (no --web-port)
• Has no StartLimit* lines at all

The repo canonical file (deploy/systemd/brioso-otelite.service) at the time
of writing uses User=otelite, the named --web-port 16686 flag, and the
StartLimit* rate-limit lines under the wrong section. If
setup-observability.sh:51 re-copies the unit from $REPO_DIR on a future
run, the working live config is replaced by a broken one.

This means: the next time setup-observability.sh is run, otelite will
break independently of Bug 6 getting fixed. The two paths have to be
reconciled before the next redeploy.

Recommended approach (plan commit 1 below): reconcile the live unit’s
nono run / positional-flag style into a unified canonical, commit, then
redeploy — not treat the repo unit as the source of truth.

Same [Service] vs [Unit] bug in the other two units (not flagged in VPS doc)

grep -n 'StartLimit' deploy/systemd/*.service shows the same directive-in-wrong-section
mistake in both:

• deploy/systemd/brioso-otelite.service:12-13 — StartLimitIntervalSec=60 and
  StartLimitBurst=5 under [Service].
• deploy/systemd/brioso-web.service:17-18 — same two lines under [Service].

Both services happen to run, but the rate limit is silently inactive. Fold them into
the same unit-file fix when Bug 3 is addressed, otherwise they drift out of sync again.

otelite: misconfiguration, not a unit bug

otelite was swept into the 2026-07-13 decommission of the old FastAPI stack
(systemctl disable --now brioso-api brioso-refine brioso-otelite) and never
re-enabled. Its dashboard had been rebound to 127.0.0.1:3100 during that cutover
(to avoid colliding with brioso-web on :3000). Per the original design,
apps/web/src/server/otelite.ts hardcodes OTELITE_BASE_URL = "http://127.0.0.1:16686",
so the dashboard must stay on :16686.

Repointed ExecStart’s listen args back to :16686 and re-enabled the unit.

Worth a repo-side audit: is there any script or workflow step that re-runs
systemctl disable brioso-otelite (or similar) during a decommission-style
cleanup, that could silently re-disable it again on a future redeploy? Nothing in
the current deploy/ tree does (verified with grep -r "systemctl disable" deploy/), but worth keeping in mind when adding any new cleanup task.

tailscale serve mode

:8000 and :8001 had been set up in tls-terminated-tcp mode pointing at
themselves (127.0.0.1:8000, 127.0.0.1:8001 — nothing ever listened there).
That state was left behind from an earlier pass by another agent session and
surfaced as the NS_ERROR_NET_EMPTY_RESPONSE the user originally reported (the
cert was for the hostname but no upstream was behind the proxy).

Reset and redone as HTTP reverse proxies:

tailscale serve --bg --https=8000 http://127.0.0.1:4983    # Drizzle Studio
tailscale serve --bg --https=8001 http://127.0.0.1:16686   # otelite

This is live tailscaled state on the box, not config tracked in the repo, but the
setup-observability.sh script (which already calls tailscale serve reset
followed by the two serve rules) is the idempotent re-applier. Important: if a
future deploy adds a tailscale serve rule but does not go through
setup-observability.sh, it must not assume any particular existing config —
always reset.

Deploy automation gap (still open)

Verified that the following are not wired in:

• deploy/scripts/install-drizzle-gateway.sh is called from
  setup-observability.sh:39, but neither that script nor deploy-vps.sh is
  itself called from any GitHub Actions workflow (verified by inspection of
  .github/workflows/). Means a fresh VPS never gets Drizzle Gateway unless an
  operator SSHes in and runs setup-observability.sh by hand. The
  install-drizzle-gateway.sh script is already idempotent.
• VITE_OTELITE_URL / VITE_DRIZZLE_GATEWAY_URL are never exported before
  deploy-vps.sh:102 runs bun run build, so the built bundle’s fallback URLs
  are localhost:16686 and localhost:4983 (confirmed by grep on
  .output/public/assets/*.js). A service restart will not fix this — it is
  baked in at build time. Needs the two env vars set to
  https://ovhcloud.tail876c20.ts.net:8001 and
  https://ovhcloud.tail876c20.ts.net:8000 respectively, then a rebuild + redeploy.
• All five unit bugs above plus this automation gap should ideally be fixed
  together so a fresh deploy-vps.sh run produces a working
  brioso-drizzle-gateway unit without any manual systemctl edit drop-in.

Tailscale ACL — not a repo issue, recorded for context

ACL grant used on the tailnet:

{
  "src": ["autogroup:member", "carloshpiedra@gmail.com"],
  "dst": ["tag:server"],
  "ip":  ["tcp:8000", "tcp:8001", "tcp:4318", "tcp:3000"]
}

Note that this is the grants schema — no action key is needed there; only
the legacy acls array requires a sibling "action": "accept". Structurally
fine as written.

One teammate (carloshpiedra@gmail.com) initially could not reach :8000/:8001
despite being in src. Diagnosis, confirmed via tailscale status --json on the VPS:

• OS-level firewall was not the cause — ufw/iptables allow all inbound on
  the tailscale0 interface unconditionally.
• His devices showed up only as anonymized shared nodes
  (Hostname: device-of-shared-to-user, blank DNSName) — unlike fully-joined
  members who get real *.tail876c20.ts.net MagicDNS names. MagicDNS could not
  resolve ovhcloud.tail876c20.ts.net on his end even though the ACL src rule
  matching his email would apply once traffic actually reached the node.
• He had been given access via both “share this device” and a tailnet member
  invite, but only the device-share had taken effect; the member invite was
  likely not accepted yet.
• Workaround used: he added a /etc/hosts entry on his own machine mapping
  the VPS Tailscale IP (100.105.198.36) to ovhcloud.tail876c20.ts.net. Works
  per-device but is brittle to IP changes and won’t help him reach other tailnet
  services. Getting him fully accepted as a member is the durable fix and is an
  operational/Tailscale-admin task, not a repo fix.

What changed on the VPS (for systemctl edit diff context)

Patches were applied as systemctl edit brioso-drizzle-gateway / brioso-otelite
drop-ins, leaving /etc/systemd/system/<unit>.service unchanged from git’s
view but layering overrides on top. Canonical fix lands in
deploy/systemd/brioso-drizzle-gateway.service (and the other two units) so the
next setup-observability.sh run produces the right unit without the drop-in.