Alerts

DigitalOcean's built-in monitoring is the cheap-and-good alerting tier. Two layers, both managed via doctl:

• Resource alerts scoped to the shithub-app droplet, fired when CPU / memory / disk / load1 cross a threshold for a sustained window.
• Uptime checks that probe the public URL from two regions and fire on outages, SSL-cert decay, or latency regressions.

Everything is reproducible via deploy/cutover/provision-do-alerts.sh — re-running is a no-op. The script is the source of truth for what alerts SHOULD exist; the DO account is the source of truth for what currently fires.

What's wired

Droplet resource alerts

Tagged on droplet shithub-app (id resolves at runtime from the droplet name in provision-do-alerts.sh).

Window is 10m so a single transient spike (build, AIDE re-baseline, WAL burst) doesn't page. All four email wolffemf@dukes.jmu.edu (the account-owner email; DO refuses unverified destinations).

Uptime check shithub.sh

https://shithub.sh probed every minute from us_east and eu_west. Three alerts hang off it:

down_global deliberately requires both regions to agree before paging — single-region noise from DO's own infrastructure is more common than real outages and was the leading cause of false positives on a previous setup.

ssl_expiry < 14 days gives a buffer to investigate Caddy's auto-renew if it doesn't fire on its own. Caddy's renewal window is normally 30 days before expiry, so by the time this alert fires we're already off the happy path.

When an alert fires

1. Read the email. It names the alert and the value that crossed.
2. Check the droplet first:

   ssh root@shithub.sh 'top -bn1 | head -20; df -h /; free -h; uptime'
   

3. Common patterns:
   • CPU sustained > 80% — usually aideinit or aide --check mid-run (12 min daily, 7 min for re-baseline). Cross-check against journalctl -t shithub-aide -n 50. If unrelated: top -c -o %CPU and identify the process.
   • Memory > 90% — Postgres + shithubd combined exceed budget. systemctl status shithubd-web for a dump of resident set; sudo -u postgres psql -c "SELECT * FROM pg_stat_activity" for long-running queries.
   • Disk > 80% — first check du -sh /var/log/* /data/* /root/* /var/lib/postgresql/* | sort -h. Most likely Postgres logs accumulating; check log_rotation_age.
   • Load1 > 4 for a 4-vCPU droplet means we're saturated. If it's also a CPU alert, same diagnosis. If load is high but CPU isn't, look at I/O wait (top → 1 to expand cores → wa).
   • down_global — curl -v https://shithub.sh/ from your laptop. Caddy log: ssh root@shithub.sh 'tail -100 /var/log/caddy/access.log | jq .'. shithubd journal: journalctl -u shithubd-web -n 100.
   • ssl_expiry — ssh root@shithub.sh 'systemctl status caddy; journalctl -u caddy -n 200 | grep -iE "renew|cert"'. Most likely cause: rate-limited by Let's Encrypt or DNS misconfiguration after a CNAME change.
   • latency p95 > 2s — could be slow-query regressions, a wedged worker, or Caddy reverse-proxy timeouts. Start with journalctl -u shithubd-web -n 200 | grep duration | sort -k1 -nr | head.

Editing thresholds

Change in deploy/cutover/provision-do-alerts.sh, then either:

1. Delete the existing alert via doctl monitoring alert delete <uuid> and re-run the provision script (it'll create the new one), OR
2. Edit in place: doctl monitoring alert update <uuid> --value <new>.

For uptime alerts: doctl monitoring uptime alert update <check-id> <alert-id> .... (CLI bug: uptime check update strips --enabled; if you mutate uptime properties, delete + recreate via the script.)

What's NOT here

• shithubd-internal metrics (request latency p95, DB pool saturation, job queue depth, throttle rejections, archive failures from Postgres). Those need a Prometheus scrape of 127.0.0.1:8080/metrics plus a Grafana Cloud free-tier — see task #263 / runbooks/observability.md (TBD).
• Slack delivery. Today everything emails. The DO API supports Slack webhooks; add --slack-channels + --slack-urls to the provision script when there's a destination.
• Pagerduty. Same — DO supports webhook destinations.