Observability — metrics tier

Two layers of monitoring run today:

1. DigitalOcean droplet alerts + uptime check (see alerts.md). Tells us when the droplet is unhappy at the OS level (CPU, mem, disk, load, down, SSL expiry, latency from outside).
2. Grafana Cloud free tier — Prometheus + dashboards (this doc). Tells us what shithubd is doing internally: request latency, DB pool saturation, panics, job-queue depth, archive failures.

Layer 1 fires for "is the box alive?" Layer 2 fires for "is the app healthy?" Different signal, different audience, both needed.

What's wired

The monitoring-client ansible role runs node_exporter (host metrics on 127.0.0.1:9100) and Grafana Alloy (the modern Grafana Agent rebrand). Alloy scrapes:

• 127.0.0.1:9100/metrics — host metrics from node_exporter
• 127.0.0.1:8080/metrics — shithubd's internal metrics

Alloy remote_writes both into Grafana Cloud's Prometheus endpoint (Mimir under the hood). No inbound port on the droplet — push-only. No Prometheus or Grafana running locally.

Notable shithubd metrics

Operator setup (one-time)

Free-tier Grafana Cloud gives you ~10k active series — way more than one shithubd droplet emits. Sign-up flow:

1. Go to https://grafana.com/auth/sign-up/create-user. As of 2026 the default flow drops you into a 14-day "Unlimited usage trial" — that's fine; without a card on file it auto-reverts to the Forever Free tier (~10k active series) when the trial ends.
2. Pick a Stack name (e.g. shithub). The portal creates the stack at <name>.grafana.net and lands you in the in-app "Get started" guide.
3. Get the Prometheus details. The in-app guide has drifted; the reliable path is via the org-level Cloud portal:
   • In the right column of the Get started page, Organization details → click Cloud portal, OR navigate directly to https://grafana.com/orgs/<your-org>.
   • Click Details on the Prometheus tile.
   • The page shows the Remote Write Endpoint (full URL), Username / Instance ID (numeric), and a Generate now link for the Password / API Token (a glc_… Access Policy token with metrics:write). Copy the token immediately — it's shown once.
4. Add to inventory/production. Use the exact Remote Write Endpoint shown on the page — the host (prod-NN, region) differs per tenant:

   grafana_cloud_prom_url=https://prometheus-prod-XX-prod-REGION.grafana.net/api/prom/push
   grafana_cloud_prom_user=1234567
   grafana_cloud_prom_token=glc_eyJrIjoi...   # the token from step 3
   

5. ansible-playbook -i inventory/production deploy/ansible/site.yml -t monitoring (or just rerun site.yml — the role is idempotent).

If you don't have ansible installed (manual fallback)

The monitoring-client role's actions are simple enough to apply by hand over SSH. Use this if you're trying to bolt monitoring onto an existing droplet and don't want to risk the full play.

ssh root@shithub.sh '
  # node_exporter
  apt-get install -y prometheus-node-exporter
  systemctl enable --now prometheus-node-exporter

  # Grafana apt repo + alloy
  mkdir -p /etc/apt/keyrings
  curl -fsS https://apt.grafana.com/gpg.key -o /etc/apt/keyrings/grafana.gpg.key
  chmod 0644 /etc/apt/keyrings/grafana.gpg.key
  echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg.key] https://apt.grafana.com stable main" \
    > /etc/apt/sources.list.d/grafana.list
  apt-get update && apt-get install -y alloy

  # Credentials env (mode 0640, root:alloy)
  mkdir -p /etc/alloy && chown root:alloy /etc/alloy && chmod 0750 /etc/alloy
  install -o root -g alloy -m 0640 /dev/stdin /etc/alloy/credentials.env <<EOF
GRAFANA_CLOUD_PROM_URL=<your URL>
GRAFANA_CLOUD_PROM_USER=<your numeric tenant id>
GRAFANA_CLOUD_PROM_TOKEN=<your glc_… token>
EOF

  # systemd drop-in to source the credentials
  mkdir -p /etc/systemd/system/alloy.service.d
  cat > /etc/systemd/system/alloy.service.d/shithub.conf <<EOF
[Service]
EnvironmentFile=/etc/alloy/credentials.env
EOF
  systemctl daemon-reload
'

Then copy the rendered alloy-config.river.j2 (substitute the {{ ansible_hostname }} template var with the real hostname) to /etc/alloy/config.alloy (mode 0644 root:alloy) and:

ssh root@shithub.sh 'systemctl enable --now alloy'

The token is the only secret; do NOT pipe it through your shell history. The install … /dev/stdin … <<EOF form above keeps it out of ps/argv, but you'll still want to clear shell history after.

Within a minute, metrics start landing. From the Cloud portal:

• Explore tab → datasource = your Prometheus → query up{job=~"node|shithubd"} — both should show 1.
• Sample queries to confirm shithubd telemetry:
  • sum(rate(shithub_http_requests_total[5m])) — overall QPS
  • histogram_quantile(0.95, sum by (le, route) (rate(shithub_http_request_duration_seconds_bucket[5m]))) — p95 latency by route
  • shithub_db_pool_acquired / shithub_db_pool_total — pool utilization
  • rate(shithub_panics_total[5m]) > 0 — recovered panics

Building a starter dashboard

Cloud → Dashboards → New → import. Use the panels below as a spine. (We're not committing the dashboard JSON to the repo yet because Grafana's UUIDs are stack-specific; doc the queries and let the operator import once.)

Alerting

Three places to set alert rules. Pick one:

• Grafana Cloud → Alerts → Alert rules. UI-first; fires to email/Slack/PagerDuty via Cloud's contact points.
• Alertmanager via remote_write. Out of scope for the free tier without their managed contact points.
• Skip Grafana alerts entirely; use the DO alerts (alerts.md) for operational pages and read shithubd metrics manually.

Suggested first three alerts (Cloud UI):

When metrics stop landing

ssh root@shithub.sh '
  systemctl status alloy --no-pager
  journalctl -u alloy -n 30 --no-pager | tail -20
  curl -fsS http://127.0.0.1:8080/metrics | head -3
  curl -fsS http://127.0.0.1:9100/metrics | head -3
'

Common failures:

• 401 Unauthorized in alloy logs — token expired or copy-pasted wrong. Mint a fresh one in the Cloud portal, update inventory, re-run the role.
• node_exporter not listening on 9100 — apt reinstall prometheus-node-exporter, check journal.
• shithubd not exposing /metrics — SHITHUB_METRICS__ENABLED=true in web.env (the default), restart shithubd-web.
• Free-tier limit hit — Cloud portal shows "Active series" at cap. Drop high-cardinality labels via metric_relabel_configs in alloy-config.river.j2.
• up{job="shithubd"} = 0 while up{job="node"} = 1 — alloy can reach shithubd (scrape_samples_scraped > 0) but the parser fails with expected a valid start token, got "\x1f". Root cause: alloy advertised Accept-Encoding: gzip, shithubd correctly returned gzipped bytes with Content-Encoding: gzip, but alloy's Prometheus scraper parsed the raw bytes pre-gunzip. Fix is already in the template (enable_compression = false on the shithubd scrape job). If you see this on a hand-rolled config, add that attribute and systemctl restart alloy. Verify via curl http://127.0.0.1:12345/api/v0/web/components/prometheus.scrape.shithubd → look for health: up.

Why Grafana Cloud over self-hosted Prometheus

• Self-hosted needs Prom + Grafana on the droplet → ~500 MB RAM, storage for time-series, an open inbound port (or VPN), TLS for the Grafana UI, an OAuth proxy for auth, and we maintain the upgrade path.
• Free Grafana Cloud is enough for our scale, gives us managed backup of metrics + dashboards + alerts, and the data leaves the droplet (so a droplet outage doesn't kill its own observability).
• Trade-off: cardinality cap. Once we approach 10k active series, evaluate either paid Cloud (~$8/mo for 10× the cap) or self-host.